Ending a process without dropping a request

Every deploy kills something

A deploy is two events: a new process starts, and an old process stops. Almost all the attention goes to the first. The dropped requests come from the second.

When the old process is told to stop, it is usually in the middle of something: three HTTP requests half-answered, a database transaction open, a background job two seconds into a five-second task, a websocket holding somebody’s live page. What it does in the next few seconds decides whether your users see anything at all.

The two signals

SIGTERM is a polite request to stop. It can be caught, and that is the point: your code gets to decide what happens.

SIGKILL cannot be caught. The process ends immediately, mid-write, mid-request.

Every deploy system does the same thing: send SIGTERM, wait a grace period, then send SIGKILL. The grace period is 10 seconds in Docker and Kubernetes by default, 90 in systemd. If your requests take longer than the grace period, some of them are being cut off on every deploy, and the only sign is a small bump in 502s that looks like noise.

What graceful shutdown actually means

Four steps, in this order:

1. Fail the readiness check first. Start reporting not-ready so the load balancer stops sending new requests. This must happen before you stop accepting, and you have to wait: health checks poll every few seconds, so there is a window where the balancer still believes in you. Sleeping 5 seconds here removes more dropped requests than anything else in this lesson.
2. Stop accepting new connections, keep serving the ones in flight.
3. Finish what is open, with a deadline.
4. Close the database pool and exit.

In Node:

process.on('SIGTERM', async () => {
  await setTimeout(5000);            // let the LB notice we are unready
  server.close(async () => {          // stop accepting, drain in-flight
    await pool.end();
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 25_000).unref();  // hard deadline
});

The last line matters. A drain with no deadline is a process that never exits, and the platform will SIGKILL it anyway, later, and less predictably. Make your own timeout shorter than the platform’s grace period, and set the platform’s grace period longer than your slowest normal request.

The trap in containers

In a container your process is PID 1, and PID 1 does not get default signal handling from the kernel. If it has no SIGTERM handler, the signal is ignored entirely and the container always dies by SIGKILL after the grace period.

Worse, if your image starts the app through a shell (CMD npm start, or an ENTRYPOINT written as a shell string) the shell is PID 1 and does not forward signals to its child. Your handler is there and never runs.

Two fixes: use the exec form (CMD ["node", "server.js"]), and run with docker run --init or tini so a real init process sits at PID 1 and forwards signals.

This is worth testing directly, because it is invisible otherwise:

docker run --rm --name t myapp &
time docker stop t      # ~0.2s means it handled SIGTERM; ~10s means it did not

That single command has caught this bug in more projects than any amount of reading.

Background workers need the same thing, differently

A web request lasts milliseconds; a job might last minutes. Draining a worker means: stop pulling new jobs, finish the current one, then exit. If the current job is too long for the grace period, the right design is a job that checks a stop flag between steps and re-queues its remainder, which is only safe if the job is idempotent, which is the argument made in the queues lesson.

Schedulers and cron-like runners need one more: on a hard kill, the lock a job was holding must expire on its own, or the job never runs again.

What to check on your own service

• Does SIGTERM reach your code at all? Log a line in the handler and stop the process to see it.
• Is your drain deadline shorter than the platform grace period?
• Do you fail readiness before you stop accepting, with a pause in between?
• Does a deploy under a small load test produce zero failed requests? Run one with oha or hey, both free, and deploy while it runs. That is the only proof.