Nginx solves the c10k problem by using multiplexing IO Technology (such as epoll of Linux, kqueue of FreeBSD, etc.), but the premise is that the request of nginx cannot have blocking operation, otherwise the whole nginx process will stop serving. However, blocking is inevitable in many cases. For example, when a client requests a static file, disk IO may lead to process blocking, which will lead to the performance degradation of nginx. To solve this problem, nginx implements the thread pool mechanism in version 1.7.11. Next, we will analyze how nginx solves the blocking operation problem through thread pool.
1, Enable thread pool feature
To use the thread pool function, first add the following configuration items to the configuration file:
location / {
root /html;
thread_pool default threads=32 max_queue=65536;
aio threads=default;
}
The above defines a thread pool named "default", which contains 32 threads, and the task queue supports up to 65536 requests. If the task queue is overloaded, Nginx will output the following error log and reject the request:
thread pool "default" queue overflow: N tasks waiting
If the above error occurs, it indicates that the load of the thread pool is very high. This problem can be solved by adding the number of threads. When the maximum processing capacity of the machine is reached, increasing the number of threads does not improve this problem.
2, Everything starts with "source"
The following mainly analyzes the source code of Nginx to understand the implementation principle of thread pool mechanism. Now let's learn about the two important data structures NGX of the Nginx thread pool_ thread_ pool_ T and ngx_thread_task_t.
ngx_thread_pool_t structure
struct ngx_thread_pool_s {
ngx_thread_mutex_t mtx;
ngx_thread_pool_queue_t queue;
ngx_int_t waiting;
ngx_thread_cond_t cond;
ngx_log_t *log;
ngx_str_t name;
ngx_uint_t threads;
ngx_int_t max_queue;
u_char *file;
ngx_uint_t line;
};
The purpose of each field is explained below:
- mtx: mutex lock, used to lock the task queue and avoid contention.
- Queue: task queue.
- Waiting: how many tasks are waiting to be processed.
- cond: used to notify the thread pool of tasks to be processed.
- Name: thread pool name.
- Threads: how many threads the thread pool consists of (number of threads).
- max_queue: the maximum number of tasks that the thread pool can handle.
ngx_thread_task_t structure
struct ngx_thread_task_s {
ngx_thread_task_t *next;
ngx_uint_t id;
void *ctx;
void (*handler)(void *data, ngx_log_t *log);
ngx_event_t event;
};
The purpose of each field is explained below:
- Next: point to the next task.
- ID: task ID.
- ctx: context of the task.
- handler: handle to the function that handles the task.
- Event: the event object associated with the task (the handler callback function of the event object will be called by the main thread after the thread pool is processed into a task).
3, Thread pool initialization
The following describes the initialization process of the thread pool
When Nginx starts, NGX will be called first_ thread_ pool_ init_ Worker () function to initialize the thread pool. ngx_ thread_ pool_ init_ The worker () function eventually calls ngx_thread_pool_init(), the source code is as follows:
static ngx_int_t
ngx_thread_pool_init(ngx_thread_pool_t *tp, ngx_log_t *log, ngx_pool_t *pool)
{
...
for (n = 0; n < tp->threads; n++) {
err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
if (err) {
ngx_log_error(NGX_LOG_ALERT, log, err,
"pthread_create() failed");
return NGX_ERROR;
}
}
...
return NGX_OK;
}
ngx_thread_pool_init() finally calls pthread_ The create () function creates a worker thread in the thread pool, and the worker thread will be deleted from NGX_ thread_ pool_ The cycle() function starts executing.
ngx_ thread_ pool_ The source code of the cycle() function is as follows:
static void *
ngx_thread_pool_cycle(void *data)
{
...
for ( ;; ) {
if (ngx_thread_mutex_lock(&tp->mtx, tp->log) != NGX_OK) {
return NULL;
}
tp->waiting--;
while (tp->queue.first == NULL) {
if (ngx_thread_cond_wait(&tp->cond, &tp->mtx, tp->log)
!= NGX_OK)
{
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NULL;
}
}
// Get a task object
task = tp->queue.first;
tp->queue.first = task->next;
if (tp->queue.first == NULL) {
tp->queue.last = &tp->queue.first;
}
if (ngx_thread_mutex_unlock(&tp->mtx, tp->log) != NGX_OK) {
return NULL;
}
// Processing tasks
task->handler(task->ctx, tp->log);
task->next = NULL;
ngx_spinlock(&ngx_thread_pool_done_lock, 1, 2048);
// Put the processed task into the completion queue
*ngx_thread_pool_done.last = task;
ngx_thread_pool_done.last = &task->next;
ngx_unlock(&ngx_thread_pool_done_lock);
(void) ngx_notify(ngx_thread_pool_handler); // Notify main thread
}
}
ngx_ thread_ pool_ The main task of the cycle () function is to get a task from the task queue to be processed, then to invoke the handler() function of the task object to process the task, then complete the task to the completion queue and pass the ngx_. Notify() notifies the main thread.
4, Add task to task queue
Through the above analysis, we know how the thread pool obtains and processes tasks from the task queue. But where do the tasks in the task queue come from? Because Nginx's mission is to process client requests, you can know that tasks are generated through client requests. That is, the task is created by the main thread (the main thread is responsible for processing client requests).
Main thread through NGX_ thread_ task_ The post() function adds a task to the task queue. The code is as follows:
ngx_int_t
ngx_thread_task_post(ngx_thread_pool_t *tp, ngx_thread_task_t *task)
{
...
if (ngx_thread_mutex_lock(&tp->mtx, tp->log) != NGX_OK) {
return NGX_ERROR;
}
// Notifies the thread pool that there are tasks to process
if (ngx_thread_cond_signal(&tp->cond, tp->log) != NGX_OK) {
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NGX_ERROR;
}
// Add task to task queue
*tp->queue.last = task;
tp->queue.last = &task->next;
tp->waiting++;
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NGX_OK;
}
ngx_ thread_ task_ The post () function first calls ngx_thread_cond_signal() notifies the thread of the thread pool that there are tasks to be processed, and then adds the tasks to the task queue. Some people may ask whether there will be an order problem when notifying the thread pool to add tasks to the task queue first. In fact, it's OK to do this because as long as the main thread doesn't call ngx_thread_mutex_unlock() unlocks the mutex, and the worker threads in the thread pool will not be from NGX_ thread_ cond_ Returned by wait().
5, Close out work
When the thread pool completes the task, it will put it in the completion queue (ngx_thread_pool_done), then call ngx_. Notify() notifies the main thread that a task has completed. After the main thread receives the notification, it will finish the work in the event module: call task.event.handler(). task.event.handler is set by the task creator;
For example, in NGX_ http_ copy_ NGX of filter module_ http_ copy_ thread_ Handler() function:
static ngx_int_t
ngx_http_copy_thread_handler(ngx_thread_task_t *task, ngx_file_t *file)
{
...
if (tp == NULL) {
if (ngx_http_complex_value(r, clcf->thread_pool_value, &name)
!= NGX_OK)
{
return NGX_ERROR;
}
tp = ngx_thread_pool_get((ngx_cycle_t *) ngx_cycle, &name);
}
task->event.data = r;
// Set the callback function of event
task->event.handler = ngx_http_copy_thread_event_handler;
if (ngx_thread_task_post(tp, task) != NGX_OK) {
return NGX_ERROR;
}
r->main->blocked++;
r->aio = 1;
return NGX_OK;
}
task.event.handler is set to ngx_http_copy_thread_event_handler, that is, when the task processing is completed, the main thread will call ngx_http_copy_thread_event_handler to finish the work.
6, Which operations use thread pools
So which operations will be processed using the thread pool. Generally speaking, disk IO will be processed using thread pool. In NGX_ http_ copy_ In the filter module, NGX is called_ thread_ Read() reads the contents of the file (when thread pooling is enabled), and ngx_thread_read() will let the thread pool handle the operation of reading the contents of the file.
ngx_ thread_ The read() code is as follows:
ssize_t
ngx_thread_read(ngx_thread_task_t **taskp, ngx_file_t *file, u_char *buf,
size_t size, off_t offset, ngx_pool_t *pool)
{
...
task = *taskp;
if (task == NULL) {
task = ngx_thread_task_alloc(pool, sizeof(ngx_thread_read_ctx_t));
if (task == NULL) {
return NGX_ERROR;
}
task->handler = ngx_thread_read_handler;
*taskp = task;
}
ctx = task->ctx;
...
ctx->fd = file->fd;
ctx->buf = buf;
ctx->size = size;
ctx->offset = offset;
if (file->thread_handler(task, file) != NGX_OK) {
return NGX_ERROR;
}
return NGX_AGAIN;
}
As you can see from the above code, the task handler is set to ngx_thread_read_handler, that is, NGX will be called in the thread pool_ thread_ read_ Handler () to read the contents of the file. File - > thread_ Handler () will call ngx_thread_task_post(), which has been analyzed earlier, ngx_thread_task_post() adds the task to the task queue.
