How to synchronize between executors of an ExecutorService

Isaac Vero :

I have a list of client sockets, usually of size around 2000. These clients are dynamic, they come and go.

I have an ExecutorService with a fixed thread pool of 32 threads handling these threads. This executor service is responsible for decoding and sending the messages to be sent to these 2000 clients.

I want to prevent that two (or more) threads of the executor service are processing the same client at the same time.

One approach could be to introduce another bookkeeping thread (so I end up with 32 + 1 threads) which is responsible for calling ExecutorService.submit(mesage) when the previous message corresponding to the same client is done. But I am not sure if this will introduce a bottleneck, meaning that this newly introduced bookkeeping thread cannot keep up submitting messages.

Ideally, I don't want to pre-allocate a thread to a set of clients in advance, as the message load is not evenly distributed between the clients. It is also not known in advance.

What are approaches for this? Are they offered by java.util.concurrent functionalities?

update

This is a quick summary, as the comments pointed out that there were some misunderstandings:

• I don't want a single thread per client, as I would end up with 2000 threads.

• Ideally, I don't want to pre-allocate a thread to a set of clients, because message rate is not evenly distributed between all clients and not known in advance.

• Message order must be preserved.

• I believe it would not be good that thread A is waiting for thread B because B is already sending a message to the same client. In other words, at all times only one thread is processing one client.

Andreas :

When a thread (A) begins processing a message (#1), it needs to register the client id with a shared manager object. For each registered client, there is a queue.

When another thread (B) begins processing a message (#2) for the same client, the registration will detect that thread A is already processing, and will add message #2 to the queue for client. Thread B will then stop and process the next message.

When thread A is done with message #1, it will try to unregister, but since message #2 is queue, thread A will instead begin processing that message. After that, when it tries to unregister again, there are no queued messages and the thread will stop and process the next message.

It is up to the manager code to correctly synchronize access, so a second message is either processed by thread B, or handed off to thread A, without getting lost.

The above logic ensures that thread B will not wait for thread A, i.e. no idle time, and that message #2 is processed as soon as possible, i.e. with minimal delay, without processing two messages for the same client as the same time.

Message order for each client is retained. Globally, message order is of course not retained, because the processing of message #2 is delayed.

Note, there will be only one queue for each thread, so only 32 queues, and only "duplicate" messages are queue, so all queue will usually remain empty.

---

UPDATE

Example: For identification here, messages are named clientId.messageId where messageId is global.

Messages are submitted to the Executor (3 threads) in this order:

1.1, 2.2, 1.3, 2.4, 3.5, 1.6

1. Thread A picks up 1.1 and starts processing.

2. Thread B picks up 2.2 and starts processing.

3. Thread C picks up 1.3, adds it to thread A's queue, then returns.

4. Thread C picks up 2.4, adds it to thread B's queue, then returns.

5. Thread C picks up 3.5 and starts processing.

6. Thread A is done with message 1.1 and starts processing 1.3.

7. Thread C is done with message 3.5 and returns.

8. Thread C picks up 1.6, adds it to thread A's queue, then returns.
   Thread C is now idle.

9. Thread B is done with message 2.2 and starts processing 2.4.

10. Thread A is done with message 1.3 and starts processing 1.6.

11. Thread B is done with message 2.4 and returns.
    Thread B is now idle.

12. Thread A is done with message 1.6 and returns.
    Thread A is now idle.