Ideal Thread Pool Size

Why correct size of Thread Pool is important?

Tuning the size of a thread pool is largely a matter of avoiding two mistakes: having too few threads or too many threads.

Too many threads can degrade performance because when CPU switches from one thread to another there is overhead of context switching.

Processor utilization is not the only consideration in tuning the thread pool size. As the thread pool grows, you may encounter the limitations of the scheduler, available memory, or other system resources, such the number of sockets, open file handles, or database connections.

The optimum size of a thread pool depends on:

• Number of processors available 
• Nature of the tasks on the work queue. 

What is the Ideal Thread Pool size for Compute Bound operations?

On an N-processor system for a work queue that will hold entirely compute-bound tasks, you will generally achieve maximum CPU utilization with a thread pool of N or N+1 threads.

What is the Ideal Thread Pool size for IO Bound operations?

For tasks that may wait for I/O to complete — for example, a task that reads an HTTP request from a socket — you will want to increase the pool size beyond the number of available processors, because not all threads will be working at all times. 

Using profiling, you can estimate the ratio of waiting time (WT) to service time (ST) for a typical request. 

If we call this ratio WT/ST, for an N-processor system, you’ll want to have approximately N*(1+WT/ST) threads to keep the processors fully utilized.