背景
近期使用uwsgi启动django服务，发现在stop/reload uwsgi时会出现hangs问题，具体日志表现为：
...gracefully killing workers...
Gracefully killing worker 2 (pid: 390129)...
Gracefully killing worker 1 (pid: 390128)...
Gracefully killing worker 4 (pid: 390131)...
Gracefully killing worker 3 (pid: 390130)...
Wed Jul 31 20:13:14 2019 - worker 1 (pid: 390128) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 2 (pid: 390129) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 3 (pid: 390130) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 4 (pid: 390131) is taking too much time to die...NO MERCY !!!
...
在测试中发现无论用什么防线先uwsgi发送stop/reload信号，uwsgi都会进入一种假死状态，即不接收请求，也不结束进程。
期初以为uwsgi进程是在处理未完成的web请求。后来发现在没有任何请求时，uwsgi也会进入这种夯死状态。
问题原因
首先说明问题的原因是因何而起。
出现这种夯死的问题是由于在uwsgi中使用了线程导致。
Destroying threads in the POSIX world is basically no-way :) The right approach is letting your thread return when the destroy procedure is triggered. You have a bunch of ways to do it, but personally i am way more brutal and i generally set --worker-reload-mercy to a couple of seconds :)
在POSIX中销毁一个线程几乎是不可能实现的工作。正确的做法是：当接收到销毁信号后正确的让线程返回。通常有很多方式来实现着一点，但是更推荐的方式是设置一个--worker-reload-mercy时间。
解决方案
- 尽量避免在uwsgi中使用线程，首先使用线程会产生GIL锁问题，其次在使用线程时还要考虑如何让其正确退出。
- 设置reload-mercy和worker-reload-mercy两个参数。
--reload-mercy - used when reloading/stopping whole uWSGI instance
--worker-reload-mercy - used when reloading/stopping single worker
问题复现
这里创建了一个最简单的django服务，并用uwsgi来启动。
进程模式
首先，配置uwsgi为进程模式启动，这里创建了5个进程。
uwsgi配置文件如下：
[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
processes = 5
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log
现在来reload uwsgi服务，并查看uwsgi的日志。
...gracefully killing workers...
Gracefully killing worker 1 (pid: 13055)...
Gracefully killing worker 2 (pid: 13056)...
Gracefully killing worker 3 (pid: 13057)...
Gracefully killing worker 4 (pid: 13058)...
Gracefully killing worker 5 (pid: 13059)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......
结论：在uwsgi使用进程模式时，reload uwsgi不会出现夯死的问题。
线程模式
前面以进程方式启动uwsgi没有出现夯死问题，那么现在就试一下以线程模式启动wusgi。这里启动了5个进程，每个进程中又包含了两个线程。
uwsgi配置文件如下：
[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log
reload uwsgi服务，并观察日志输出。
...gracefully killing workers...
Gracefully killing worker 1 (pid: 14913)...
Gracefully killing worker 2 (pid: 14914)...
Gracefully killing worker 3 (pid: 14915)...
Gracefully killing worker 4 (pid: 14916)...
Gracefully killing worker 5 (pid: 14917)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......
结论：在uwsgi中使用线程模式也不会造成reload夯死的问题。
uwsgi app中使用线程
uwsgi的线程不会造成任何问题，那前文所指的线程究竟是什么？
现在uwsgi启动脚本中创建一个线程，在这种情况下尝试reload uwsgi并查看日志输出。
"""
WSGI config for laji_baclend project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/
"""
import os
import time
import threading
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'laji_baclend.settings')
def loop():
while True:
time.sleep(10)
t = threading.Thread(target=loop, name="loop_thread")
t.start()
application = get_wsgi_application()
随后以线程方式启动uwsgi，并reload uwsgi。
...gracefully killing workers...
Gracefully killing worker 1 (pid: 24390)...
Gracefully killing worker 2 (pid: 24391)...
Gracefully killing worker 3 (pid: 24394)...
Gracefully killing worker 4 (pid: 24396)...
Gracefully killing worker 5 (pid: 24398)...
Thu Aug 1 16:44:48 2019 - worker 1 (pid: 24390) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:44:48 2019 - worker 2 (pid: 24391) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:44:48 2019 - worker 3 (pid: 24394) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:44:48 2019 - worker 4 (pid: 24396) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:44:48 2019 - worker 5 (pid: 24398) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
......
结论：在uwsgi app中使用线程就导致reload夯死。
采用reload-mercy和worker-reload-mercy避免
若场景中非要在uwsgi app中使用线程，可以通过配置reload-mercy和worker-reload-mercy两个参数避免夯死的问题。
uwsgi配置文件如下：
[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log
reload-mercy = 1
worker-reload-mercy = 1
再次reload uwsgi服务，输出日志如下：
...gracefully killing workers...
Gracefully killing worker 1 (pid: 24766)...
Gracefully killing worker 2 (pid: 24767)...
Gracefully killing worker 3 (pid: 24768)...
Gracefully killing worker 4 (pid: 24769)...
Gracefully killing worker 5 (pid: 24770)...
Thu Aug 1 16:50:13 2019 - worker 1 (pid: 24766) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:50:13 2019 - worker 2 (pid: 24767) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:50:13 2019 - worker 3 (pid: 24768) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:50:13 2019 - worker 4 (pid: 24769) is taking too much time to die...NO MERCY !!!
Thu Aug 1 16:50:13 2019 - worker 5 (pid: 24770) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......
这里虽然也会出现NO MERCY问题，但是uwsgi在reload过程中并没有出现夯死的情况。
