hotfix: guard against + eliminate stale (IPVS-killed) Postgres connections

Production incident

Production Telegram bot is failing every turn for affected users with:

OperationalError: consuming input failed: server closed the connection unexpectedly

preceded by psycopg.pool | discarding closed connection, and the bot falls back to "The system is being updated. Please try again in a few minutes."

Root cause

The app reaches Postgres through the Docker Swarm overlay network's default vip endpoint mode, which routes every connection through a virtual IP load balanced by the kernel's IPVS. IPVS silently drops an established-but-idle TCP flow after ~900s — independent of whether Postgres itself is healthy. The LangGraph checkpointer's connection pool (ai/core/logics/assistant/graph/wiring.py) used psycopg_pool defaults (no borrow-time health check, no TCP keepalives, max_lifetime=3600s), so after any quiet period it handed LangGraph a connection whose socket IPVS had already killed. The first query on it fails mid-turn.

Fix — two complementary layers

1. App-level (commit 414ae7c, cherry-picked from beta's e5fa1e5, zero conflicts, identical diff): _pool_kwargs in wiring.py now sets:

• check=AsyncConnectionPool.check_connection — health-checks every connection on borrow; a dead one is discarded and replaced before LangGraph sees it.
• TCP keepalives (idle 30s / interval 10s / count 3) — keeps an idle flow alive through IPVS.
• max_lifetime=600s (< 900s, env-tunable) — belt-and-braces recycle.

2. Infra-level (commit 9ef2d10, new — not on beta yet either): endpoint_mode: dnsrr on the postgres service in both docker-compose-beta.yml and docker-compose-production.yml. This makes clients resolve the service's DNS name straight to the single running task's container IP instead of through the overlay's virtual IP — so the connection never passes through IPVS's idle-timeout logic in the first place. Only one Postgres replica ever runs in either stack, so dnsrr's lack of client-side load balancing across multiple tasks changes nothing else. Local docker-compose.yml (plain bridge-network dev stack, no Swarm/IPVS) needs no change.

Together: layer 2 removes the actual cause of the drop for this connection path; layer 1 is defense-in-depth so any future or transient stale connection (this DB or another) is caught and replaced instead of failing a user's turn.

Verification

• Cherry-pick (414ae7c) applied with zero conflicts; diff against main's wiring.py is byte-identical to what was verified on beta.
• Both compose files validated with docker compose config after the endpoint_mode change (no schema errors; nested correctly under deploy:, not top-level — the older endpoint_mode top-level placement is rejected by the current Compose Spec).
• Syntax-checked (py_compile) the Python changes in this environment.
• Could not execute the full Python test suite here (no Python 3.14 / uv environment available in this session) — the checkpointer commit ships its own regression coverage (ai/evals/tests/test_checkpointer_pool.py): an offline test asserting all three pool guards are present, and a pg-marked live test that terminates a pooled connection's backend server-side (faithfully simulating the IPVS drop) and asserts the pool discards it and recovers. Please let CI run these before merging.
• The dnsrr change has NOT been exercised against a live Swarm deploy in this session — recommend validating on beta first (redeploying the beta stack already exercises this compose file) before it reaches production.

Important — this alone does NOT deploy

Per .gitlab-ci.yml, CI only builds/deploys on the beta and production branches — main builds nothing. production is currently kept in sync with main via periodic Merge branch 'main' into 'production' merges (currently identical content). Merging this MR fixes main but will not stop the live incident until a follow-up main → production merge is also done (which also redeploys the stack, applying the new compose file). Happy to prepare that MR immediately once this one is merged, given the active incident.

Not done (by design)

• Not merged — opened for review only, per instruction.
• No production deploy, service restart, or Swarm stack redeploy performed.
• No live database or production system touched.