On the fly binary upgrading is possible w/o losing any incoming requests and no service downtime. In order to upgrade the server binary on the fly, the new executable file should be put in place of an old binary first.
The new binary is installed in place.
% ls -aslitr /usr/local/nginx/sbin/ total 5504 132150 2612 -rwxr-xr-x 1 root root 2671562 Oct 18 12:31 nginx.old 132170 56 -rw-r--r-- 1 root root 55180 Oct 21 19:09 strings_nginx 132148 4 drwxr-xr-x 13 root root 4096 Feb 5 10:45 .. 132149 4 drwxr-xr-x 2 root root 4096 Feb 5 10:45 . 139728 2828 -rwxr-xr-x 1 root root 2892444 Feb 5 10:45 nginx
Version check of the old and new binary file.
% /usr/local/nginx/sbin/nginx.old -V nginx version: nginx/1.1.19 built by gcc 4.6.3 configure arguments: --add-module=../ngx-fancyindex
% /usr/local/nginx/sbin/nginx -V
nginx version: nginx/1.5-altair
built by gcc 4.6.3
configure arguments: --add-module=../headers-more-nginx-module-0.25/ \
--add-module=../ngx-fancyindex
We remotely check the running version that is listening on the machine (we enabled the `server_tokens' directive on purpose).
% nc -v host.tld 80 <<EOF HEAD / HTTP/1.0 EOF host.tld [FF.FF.FF.FF] 80 (http) open HTTP/1.1 200 OK Server: nginx/1.1.19 Date: Wed, 05 Feb 2014 09:19:55 GMT
The process basically consists of three steps, as follows:
1. Send a USR2 signal (upgrading the executable file) to the currently running master process (old master process).
1.1. The PID file of the old master process is renamed, appending the '.oldbin' suffix (e.g., `nginx.pid.oldbin').
1.2. A new binary is executed.
1.3. A new master process is started.
1.4. The new master process spawns its own worker processes.
2. Send a WINCH signal (gracefully shutdown the worker processes) to the old master process, to phase the old instance out.
2.1. The old master's worker processes will start to gracefully shut down.
2.2. After a while, only new worker processes will be left alive, handling future incoming requests.
3.
Get the nginx master process PID from its pid file:
% cat /usr/local/nginx/logs/nginx.pid 21977
Check the PID of old master process:
% ps ax | grep nginx 21977 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx 28389 ? S 0:00 nginx: worker process
Next, the USR2 signal (upgrading an executable file) will be sent to the master process.
Signal the old master process to start a new master process using the updated binary:
% kill -s USR2 21977
The master process first renames its PID file to a new file, appending the ".oldbin" suffix (e.g., `/usr/local/nginx/logs/nginx.pid.oldbin'), and then spans a new instance of a master process by running a new executable file (the file that we just overwritten in place) that in turn starts new worker processes.
% ps ax | grep nginx 18571 ? S 0:00 nginx: master process /usr/local/nginx/sbin/nginx 18573 ? S 0:00 nginx: worker process 21977 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx 28389 ? S 0:00 nginx: worker process
After that all worker processes (old and new ones) continue to accept requests. If the WINCH signal is sent to the first master process, it will send messages to its worker processes, requesting them to shut down gracefully, and they will start to exit.
{ When using the `rtsig' (real-time signals) connection processing method on Linux, the new processes may not accept connections even after the old master process was sent the WINCH signal. If that is the case, the USR1 signal (re-opening log files) should be sent to the new master process continuously, until the new processes start to accept connections. }
After some time, only the new worker processes will process requests.
Also, notice that the old master process doesn't close its LISTEN sockets, and it can be managed to start its worker processes again if needed (in case the upgrade isn't successful, or the new binary file doesn't perform as desired, for some reason).
Gracefully shut down old worker processes:
% kill -s WINCH 21977
% ps ax | grep nginx 18571 ? S 0:00 nginx: master process /usr/local/nginx/sbin/nginx 18573 ? S 0:00 nginx: worker process 21977 ? Ss 0:00 nginx: master process /usr/local/nginx/sbin/nginx
% nc -v host.tld 80 <<EOF HEAD / HTTP/1.0 EOF host.tld [FF.FF.FF.FF] 80 (http) open HTTP/1.1 200 OK Server: nginx/1.5.8 Date: Wed, 05 Feb 2014 09:20:41 GMT
If the new master process successfully exits, then the old master process discards the `.oldbin' suffix from the file name with the PID.
Therefore, if the upgrade was successful, the old master process can be sent the QUIT signal, so that only the new processes will stay.
Gracefully exit old master process:
% kill -s QUIT 21977
% ps ax | grep nginx 18571 ? S 0:00 nginx: master process /usr/local/nginx/sbin/nginx 18573 ? S 0:00 nginx: worker process 21585 pts/0 S+ 0:00 grep --color=auto nginx
Appendix: Building.
With my finger on the trigger, I run dot slash configure:
% ./configure --add-module=../headers-more-nginx-module-0.25/ --add-module=../ngx-fancyindex/
Magic stuff happening in the meantime.
Here's me replacing nginx.
% make install make -f objs/Makefile install make[1]: Entering directory `/root/build/nginx/nginx-1.5.8' test -d '/usr/local/nginx' || mkdir -p '/usr/local/nginx' test -d '/usr/local/nginx/sbin' || mkdir -p '/usr/local/nginx/sbin' test ! -f '/usr/local/nginx/sbin/nginx' || mv '/usr/local/nginx/sbin/nginx' '/usr/local/nginx/sbin/nginx.old' cp objs/nginx '/usr/local/nginx/sbin/nginx' test -d '/usr/local/nginx/conf' || mkdir -p '/usr/local/nginx/conf' cp conf/koi-win '/usr/local/nginx/conf' cp conf/koi-utf '/usr/local/nginx/conf' cp conf/win-utf '/usr/local/nginx/conf' test -f '/usr/local/nginx/conf/mime.types' || cp conf/mime.types '/usr/local/nginx/conf' cp conf/mime.types '/usr/local/nginx/conf/mime.types.default' test -f '/usr/local/nginx/conf/fastcgi_params' || cp conf/fastcgi_params '/usr/local/nginx/conf' cp conf/fastcgi_params '/usr/local/nginx/conf/fastcgi_params.default' test -f '/usr/local/nginx/conf/fastcgi.conf' || cp conf/fastcgi.conf '/usr/local/nginx/conf' cp conf/fastcgi.conf '/usr/local/nginx/conf/fastcgi.conf.default' test -f '/usr/local/nginx/conf/uwsgi_params' || cp conf/uwsgi_params '/usr/local/nginx/conf' cp conf/uwsgi_params '/usr/local/nginx/conf/uwsgi_params.default' test -f '/usr/local/nginx/conf/scgi_params' || cp conf/scgi_params '/usr/local/nginx/conf' cp conf/scgi_params '/usr/local/nginx/conf/scgi_params.default' test -f '/usr/local/nginx/conf/nginx.conf' || cp conf/nginx.conf '/usr/local/nginx/conf/nginx.conf' cp conf/nginx.conf '/usr/local/nginx/conf/nginx.conf.default' test -d '/usr/local/nginx/logs' || mkdir -p '/usr/local/nginx/logs' test -d '/usr/local/nginx/logs' || mkdir -p '/usr/local/nginx/logs' test -d '/usr/local/nginx/html' || cp -R html '/usr/local/nginx' test -d '/usr/local/nginx/logs' || mkdir -p '/usr/local/nginx/logs' make[1]: Leaving directory `/root/build/nginx/nginx-1.5.8'
