Skip to content

fix(checkpointer): guard the LangGraph pool against stale (IPVS-killed) connections

Administrator requested to merge main into production

On Beta the app reaches Postgres through a Docker Swarm VIP / IPVS load balancer that drops idle TCP flows after ~900s. The checkpointer pool relied on psycopg_pool defaults (check=None, no keepalives, max_lifetime=3600), so after a quiet spell it handed LangGraph a socket IPVS had already killed → "server closed the connection unexpectedly" → the turn failed. Not a Postgres crash.

Fix (wiring.py::_pool_kwargs — only the checkpointer pool; no DB/schema/SQLAlchemy/ business-logic change):

  • check=AsyncConnectionPool.check_connection — health-check every connection on borrow (a real round-trip); a dead one is discarded + replaced before LangGraph sees it. Primary guard.
  • TCP keepalives (1 / idle 30 / interval 10 / count 3) — keep an idle flow alive through IPVS so it never reaches the ~900s timeout.
  • max_lifetime=600 (< 900, env-tunable) — belt-and-braces recycle.

Tests (evals/tests/test_checkpointer_pool.py):

  • offline: the pool is configured with all three guards (fails if any is dropped).
  • pg: terminate a pooled connection's backend server-side (faithful IPVS sim), then the pool discards it on borrow and recovers on a fresh backend — verified live. Persistence test still passes with the new config.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Merge request reports

Loading