Achieving Zero Dropped In-Flight Requests During Rolling Pod Evictions

The Context: "The 502 Blip on Every Deployment"

On Sunday, August 30, 2026, during an automated midday rolling release, our real-time monitoring captured a recurring anomaly: every time Kubernetes deployed a new container image, our ingress gateways threw between 40 to 120 HTTP 502 Bad Gateway errors.

Our application developers swore they had configured graceful shutdown: server.shutdown = graceful was active in Spring Boot, and terminationGracePeriodSeconds was set to 60.

Yet, customers were seeing connections severed mid-checkout. Why? Because Kubernetes pod termination and ingress routing endpoint deletion are completely asynchronous operations running in parallel. When a pod receives SIGTERM, Kubernetes simultaneously removes the pod from the Endpoints slice. But iptables/eBPF routing updates take 1 to 3 seconds to propagate across cluster nodes! During that 2-second window, the ingress gateway was still enthusiastically forwarding fresh traffic to a pod that was already executing its shutdown hook and closing its listening sockets! Total baigan ka shutdown (a graceful shutdown that isn't graceful at all).

The Problem Statement

Eliminating transient HTTP 502 and TCP connection reset errors during Kubernetes node autoscaling, spot node evictions, and rolling image upgrades:

• Cluster Scale: 120 nodes running 2,400 pods across 3 availability zones.
• Continuous Traffic: 80,000 active concurrent WebSocket and HTTP/2 connections.
• Target Invariant: Exactly zero dropped requests during rolling deployments.
• Drain Timeout: Full termination cycle must complete within 45 seconds to maintain deployment velocity.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

1. The Race Condition Between Control Plane and Kubelet

When a pod is deleted:

1. Event A (Kubelet): The Kubelet sends SIGTERM to the container process. The process stops listening to its port.
2. Event B (Endpoints Controller): The API server updates the Endpoints / EndpointSlice object. Kube-proxy or the ingress controller watches this event and updates iptables or Envoy route tables.

Event B is fundamentally asynchronous and slower than Event A. If Event A happens before Event B finishes, incoming traffic lands on a closed socket, producing an instant 502 Bad Gateway or ECONNREFUSED.

2. The Mandatory preStop Sleep Barrier

The fix is mathematically deterministic: we must artificially delay the delivery of SIGTERM until the entire cluster network has finished removing the pod from routing tables.

We inject an explicit preStop lifecycle hook in the container specification:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: core-banking-api
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          image: banking-api:v2.4.0
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 3

During those 10 seconds of sleep:

• The container process is completely unaware that it is dying. It continues listening to its port and serving requests.
• The Kubernetes control plane updates EndpointSlice.
• Ingress proxies across all nodes receive the update and cleanly remove the pod from their load balancing pools.
• When sleep 10 finishes, zero new traffic is arriving at the pod. The subsequent SIGTERM can now drain existing in-flight connections with zero dropped packets.

3. HTTP/2 and WebSocket Connection Draining (GOAWAY)

For persistent HTTP/2 and gRPC streams:

• When the application receives SIGTERM, it must immediately transmit an HTTP/2 GOAWAY frame with last_stream_id = 2^31 - 1.
• It pauses for 1 second, then emits a second GOAWAY with the exact ID of the highest processed stream.
• This informs clients: "Do not send new requests over this TCP connection; finish current streams and reconnect to a fresh pod."

The "Baigan" Take (Expert Insights)

Many engineers argue: "Why add an extra 10-second sleep? It slows down my CI/CD deployment pipeline!"

That is baigan ki jaldi (reckless, impatient haste). What is the point of a 2-minute deployment pipeline if every single deployment drops 80 customer checkouts?

A 10-second preStop sleep is the cheapest, most battle-tested insurance policy in cloud native architecture. It transforms an unpredictable, race-condition-prone deployment into a zero-error, mathematically provable rolling update.

Performance Checklist

• Configure a mandatory preStop: exec: command: ["sleep", "10"] hook on all public ingress deployments.
• Set terminationGracePeriodSeconds to at least 45 seconds (greater than the sleep hook + max transaction duration).
• Transmit HTTP/2 and gRPC GOAWAY frames upon receiving SIGTERM to initiate orderly client stream migration.
• Ensure liveness probes do not prematurely kill draining pods during the shutdown grace period.
• Verify zero 502/504 errors during rolling releases using continuous automated Locust load testing.