| The Paste HTTP Server Thread Pool |
| ================================= |
| |
| This document describes how the thread pool in ``paste.httpserver`` |
| works, and how it can adapt to problems. |
| |
| Note all of the configuration parameters listed here are prefixed with |
| ``threadpool_`` when running through a Paste Deploy configuration. |
| |
| Error Cases |
| ----------- |
| |
| When a WSGI application is called, it's possible that it will block |
| indefinitely. There's two basic ways you can manage threads: |
| |
| * Start a thread on every request, close it down when the thread stops |
| |
| * Start a pool of threads, and reuse those threads for subsequent |
| requests |
| |
| In both cases things go wrong -- if you start a thread every request |
| you will have an explosion of threads, and with it memory and a loss |
| of performance. This can culminate in really high loads, swapping, |
| and the whole site grinds to a halt. |
| |
| If you are using a pool of threads, all the threads can simply be used |
| up. New requests go into a queue to be processed, but since that |
| queue never moves forward everyone will just block. The site |
| basically freezes, though memory usage doesn't generally get worse. |
| |
| Paste Thread Pool |
| ----------------- |
| |
| The thread pool in Paste has some options to walk the razor's edge |
| between the two techniques, and to try to respond usefully in most |
| cases. |
| |
| The pool tracks all workers threads. Threads can be in a few states: |
| |
| * Idle, waiting for a request ("idle") |
| |
| * Working on a request |
| |
| * For a reasonable amount of time ("busy") |
| |
| * For an unreasonably long amount of time ("hung") |
| |
| * Thread that should die |
| |
| * An exception has been injected that should kill the thread, but it |
| hasn't happened yet ("dying") |
| |
| * An exception has been injected, but the thread has persisted for |
| an unreasonable amount of time ("zombie") |
| |
| When a request comes in, if there are no idle worker threads waiting |
| then the server looks at the workers; all workers are busy or hung. |
| If too many are hung, another thread is opened up. The limit is if |
| there are less than ``spawn_if_under`` busy threads. So if you have |
| 10 workers, ``spawn_if_under`` is 5, and there are 6 hung threads and |
| 4 busy threads, another thread will be opened (bringing the number of |
| busy threads back to 5). Later those threads may be collected again |
| if some of the threads become un-hung. A thread is hung if it has |
| been working for longer than ``hung_thread_limit`` (default 30 |
| seconds). |
| |
| Every so often, the server will check all the threads for error |
| conditions. This happens every ``hung_check_period`` requests |
| (default 100). At this time if there are more than enough threads |
| (because of ``spawn_if_under``) some threads may be collected. If any |
| threads have been working for longer than ``kill_thread_limit`` |
| (default 1800 seconds, i.e., 30 minutes) then the thread will be |
| killed. |
| |
| To kill a thread the ``ctypes`` module must be installed. This will |
| raise an exception (``SystemExit``) in the thread, which should cause |
| the thread to stop. It can take quite a while for this to actually |
| take effect, sometimes on the order of several minutes. This uses a |
| non-public API (hence the ``ctypes`` requirement), and so it might not |
| work in all cases. I've tried it in pure Python code and with a hung |
| socket, and in both cases it worked. As soon as the thread is killed |
| (before it is actually dead) another worker is added to the pool. |
| |
| If the killed thread lives longer than ``dying_thread_limit`` (default |
| 300 seconds, 5 minutes) then it is considered a zombie. |
| |
| Zombie threads are not handled specially unless you set |
| ``max_zombies_before_die``. If you set this and there are more than |
| this many zombie threads, then the entire process will be killed. |
| This is useful if you are running the server under some process |
| monitor, such as ``start-stop-daemon``, ``daemontools``, ``runit``, or |
| with ``paster serve --monitor``. To make the process die, it may run |
| ``os._exit``, which is considered an impolite way to exit a process |
| (akin to ``kill -9``). It *will* try to run the functions registered |
| with ``atexit`` (except for the thread cleanup functions, which are |
| the ones which will block so long as there are living threads). |
| |
| Notification |
| ------------ |
| |
| If you set ``error_email`` (including setting it globally in a Paste |
| Deploy ``[DEFAULT]`` section) then you will be notified of two error |
| conditions: when hung threads are killed, and when the process is |
| killed due to too many zombie threads. |
| |
| Missed Cases |
| ------------ |
| |
| If you have a worker pool size of 10, and 11 slow or hung requests |
| come in, the first 10 will get handed off but the server won't know |
| yet that they will hang. The last request will stay stuck in a queue |
| until another request comes in. When a later request comes later |
| (after ``hung_thread_limit`` seconds) the server will notice the |
| problem and add more threads, and the 11th request will come through. |
| |
| If a trickle of bad requests keeps coming in, the number of hung |
| threads will keep increasing. At 100 the ``hung_check_period`` may |
| not clean them up fast enough. |
| |
| Killing threads is not something Python really supports. Corruption |
| of the process, memory leaks, or who knows what might occur. For the |
| most part the threads seem to be killed in a fairly simple manner -- |
| an exception is raised, and ``finally`` blocks do get executed. But |
| this hasn't been tried much in production, so there's not much |
| experience with it. |
| |
| watch_threads |
| ------------- |
| |
| If you want to see what's going on in your process, you can install |
| the application ``egg:Paste#watch_threads`` (in the |
| ``paste.debug.watchthreads`` module). This lets you see requests and |
| how long they have been running. In Python 2.5 you can see tracebacks |
| of the running requests; before that you can only see request data |
| (URLs, User-Agent, etc). If you set ``allow_kill = true`` then you |
| can also kill threads from the application. The thread pool is |
| intended to run reliably without intervention, but this can help debug |
| problems or give you some feeling of what causes problems in the site. |
| |
| This does open up privacy problems, as it gives you access to all the |
| request data in the site, including cookies, IP addresses, etc. It |
| shouldn't be left on in a public setting. |
| |
| socket_timeout |
| -------------- |
| |
| The HTTP server (not the thread pool) also accepts an argument |
| ``socket_timeout``. It is turned off by default. You might find it |
| helpful to turn it on. |
| |
