Skip to content

ci: make a failed deploy hook say why nothing was deployed

Maryam Ghorbani requested to merge fix/ci-deploy-hook-404 into main

The failure

Pipeline #33046 (production) failed in deploy with its entire diagnosis being:

curl: (22) The requested URL returned error: 404

What is actually wrong

All three build jobs succeeded and pushed :production images. Only the hook POST failed.

  • The same job, script, runner and variable name succeeds on beta (job 43730).
  • back-end/flight's production deploy answers 204 from the same runner.

So neither this CI file nor the production Portainer is broken. The production-scoped SERVICE_HOOK holds a webhook token Portainer does not have — that is Portainer's standard 404, whose body reads Unable to find a webhook with this token. curl -f prints only the status line and throws that body away, which is why the job said nothing useful.

The most likely cause: the production-safrat-assistant stack (docker-compose-production.yml) has never been created in the production Portainer, so no webhook for it exists yet and the variable holds a stale or copied-from-beta value.

What this MR changes

The real fix is infra-side and cannot be committed: create the stack, then put its webhook into the production-scoped SERVICE_HOOK. This MR fixes the part that lives in the repo — the job that could not tell you any of the above.

  • Inspect the status code instead of curl -f, and print Portainer's response body. Non-2xx still fails the job, exactly as before.
  • Log the target URL with the token segment redacted, so the trace shows which Portainer and which webhook endpoint the environment points at. A value copied from the other environment used to look identical to a dead one.
  • Spell out the three causes of a 404 and the fix, in the trace, at the moment of failure.
  • Retry only connection failures and 5xx (3 attempts). A 4xx is a configuration answer — re-POSTing a dead token is what made "just retry the deploy job" a waste of time.
  • Say plainly that a 2xx is not proof of rollout, with the verify command.
  • Document in the header that SERVICE_HOOK is per-environment and that the two values are not interchangeable.

What the failing job would print after this MR

POST https://<portainer-host>/api/stacks/webhooks/<redacted>  (environment: production)
HTTP 404
Portainer response: {"message":"Unable to find a webhook with this token","details":"Object not found inside the database"}
NOTHING WAS DEPLOYED - Portainer does not know this webhook.
The 'production'-scoped SERVICE_HOOK holds a token that is not registered. Usual causes:
  1. The stack does not exist yet in that Portainer - a webhook is created WITH the stack.
     Deploy docker-compose-production.yml as stack 'production-safrat-assistant' first.
  2. The stack was deleted and recreated - the webhook UUID changed and the old one 404s forever.
  3. The value belongs to the OTHER environment's Portainer, or it is a service webhook
     (/api/webhooks/<uuid>) pasted where a stack webhook (/api/stacks/webhooks/<uuid>) is expected.
Fix: copy the webhook from the stack's page in Portainer into the 'production'-scoped variable.

Verification

Ran the script against a fake Portainer under busybox ash (the deploy image's shell): 404, 204, 5xx-retry, connection-refused, unset and non-URL values each produce the right message and exit code. The config passes the GitLab CI lint API (valid: true).

Note this MR alone does not make production deploy — it makes the next failure legible. The stack + webhook still have to be created before production goes green.

Merge request reports