[jacorb-developer] Client connection threads are never released

[Stout, Eric A.]

We have a mixed-platform, mixed-language, mixed-ORB system of about 1500 processes.  A few Java JacORB processes are clients to all, or virtually all, of the processes in the system.  I have set jacorb.connection.client.max_receptor_threads=2500, so the system starts up fine, and runs for a few days.  But processes are occasionally stopped and restarted, causing the all-process clients to make new connections and consume more threads.  Eventually, I end up with an error like:

2013/02/09 12:35:14.085 E_LOG Sys_Mgr_Server|main||@iccsfwprod0002 [Process Monitor: iccsprod0001, ISP_Shot_Director] ProcessMonitorTask: Ping received unexpected exception for process: [nif/iccsprod0001/ISP_Shot_Director]
                org.omg.CORBA.NO_RESOURCES: (Pool)[0/2500] no idle threads but maximum number of threads reached (2500)  vmcid: 0x0  minor code: 0  completed: No
                Stack trace: org.omg.CORBA.NO_RESOURCES: (Pool)[0/2500] no idle threads but maximum number of threads reached (2500)  vmcid: 0x0  minor code: 0  completed: No
                at org.jacorb.util.threadpool.ThreadPool.putJob(ThreadPool.java:183)
                at org.jacorb.orb.giop.MessageReceptorPool.connectionCreated(MessageReceptorPool.java:84)
                at org.jacorb.orb.giop.ClientConnectionManager.getConnection(ClientConnectionManager.java:121)
                at org.jacorb.orb.Delegate.bind(Delegate.java:335)
                at org.jacorb.orb.Delegate.bind(Delegate.java:290)
                at org.jacorb.orb.Delegate.getObjectKey(Delegate.java:852)
                at org.jacorb.orb.ORB.findPOA(ORB.java:510)
                at org.jacorb.orb.Delegate.resolvePOA(Delegate.java:883)
                at org.jacorb.orb.Delegate.is_really_local(Delegate.java:1504)
                at org.jacorb.orb.Delegate.is_local(Delegate.java:1493)
                at org.omg.CORBA.portable.ObjectImpl._is_local(ObjectImpl.java:372)
                at iccs.base.corba.ConnectionHandler._is_local(ConnectionHandler.java:77)
                at iccs.idl.Idl_Sys_Mgr.Process._AgentStub.ping(_AgentStub.java:33)
                at iccs.services.system_manager.server.ProcessMonitorTask.run(ProcessMonitorTask.java:94)

>From looking at the JacORB code, it looks like client threads are never released; if the server goes away, the thread just waits around for it to maybe come back later.  But our processes always come up on a different port, and our JacORB processes come up with a unique implname, for reasons that I'm not going to go into.  So, we just slowly leak client threads.

We are using JacORB 2.3.1.

Is there a way to stop these threads from leaking?

Thanks,
Eric