Incident: CORS failing only in production

A 204 preflight can still fail: replay it with the page's Origin, read Access-Control-Allow-Origin, and fix the allow-list's value, not the policy.

Before you read: predict

PAGE · 08:24 · the candidate app (https://app.examhall.io) has stopped saving answers to https://api.examhall.io; exams start at 09:00

Release 2026.10.06-1 went out at 08:10. Its one change on the API moves the allowed origins out of the code into a variable that each pod reads when it starts:

# production
CORS_ORIGINS: "https://app.examhall.io/,https://proctor.examhall.io/

Staging runs the same build and still saves. In production the browser console reports a CORS error on every autosave. The on-call engineer replays an autosave's preflight against production:

curl -i -X OPTIONS https://api.examhall.io/attempts/a-7/answers/q4 -H 'Origin: https://app.examhall.io' -H 'Access-Control-Request-Method: POST' -H 'Access-Control-Request-Headers: content-type'

Which status and CORS headers come back?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(','), credentials: true }));   // cors 2.8.6

The 30-second version

CORS fails in one environment after a config move: replay the preflight with the page's Origin and read its headers, not the 204. A missing Access-Control-Allow-Origin, with the other CORS headers present, means the allow-list refused the origin. Origin never has a path or slash; lists match exactly. Fix the value, refuse bad entries at start, check deploys from outside.

A 204 preflight can still fail: replay it with the page's Origin, read Access-Control-Allow-Origin, and fix the allow-list's value, not the policy.

How it works

Step 1 of 12

Read the symptom. Saves fail in one environment; the page's code sees a bare TypeError, while the console names the CORS check that failed.

Analogy: An alpine hut's bunk list

The alpine club moves its bookings to a new export, which writes every party's name with a stroke after it: "Okafor/". The hut warden checks each arriving party against the list, letter for letter.

At the hut	In the incident
the new export, a stroke after every name	the release's variable, a slash after every origin (steps 3, 8)
"arrived" written in the log for every party	the API answering every preflight 204 (step 4)
a bunk key only when the name matches letter for letter	`Access-Control-Allow-Origin` only on an exact match (steps 6, 8)
the name as the membership card prints it, never with a stroke	the `Origin` the browser writes (step 7)
phoning the hut and reading out one party's name	replaying the preflight with curl (step 5)
the sister hut, whose list was typed by hand	staging's clean value (step 9)
the warden's family, already inside, never checked	local calls through the dev proxy (step 9)
"give anyone a bunk"	`*`, which credentials refuse (step 10)

Breaks down when: a warden would notice every party failing and phone the club; the API notices nothing. It logs 204 for every preflight, and only the browser refuses.

Contrasting case: a gateway that answers OPTIONS with 401 before the CORS middleware runs. The console shows the same CORS error, but the replay shows a failing status, and the allow-list is innocent.

In real terms: when CORS fails in one environment only, compare that environment's allow-list value with the Origin the browser sends, byte for byte.

Examples, run for real

4.1 ExamHall EXH-3000: 08:24, production against staging

Release 2026.10.06-1 in both environments: one autosave from the candidate app in Chromium, the API's own log, then the on-call engineer's replay against production.

4.1 cors 2.8.6 · Chromium 153.0.8010.12 · release 2026.10.06-1 · production CORS_ORIGINS "https://app.examhall.io/,https://proctor.examhall.io/" · staging CORS_ORIGINS "https://staging.app.examhall.io,https://staging.proctor.examhall.io"
4.1 production · one autosave · api log: OPTIONS /attempts/a-7/answers/q4 (Origin https://app.examhall.io) -> 204, no Allow-Origin · POSTs logged 0 · page: TypeError
4.1 production · console: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
4.1 staging · one autosave · api log: OPTIONS /attempts/a-7/answers/q4 (Origin https://staging.app.examhall.io) -> 204, Allow-Origin https://staging.app.examhall.io; POST /attempts/a-7/answers/q4 (Origin https://staging.app.examhall.io) -> 201, Allow-Origin https://staging.app.examhall.io · POSTs logged 1 · page: reads 201 saved
4.1 replay against production: curl -i -X OPTIONS https://api.examhall.io/attempts/a-7/answers/q4 -H 'Origin: https://app.examhall.io' -H 'Access-Control-Request-Method: POST' -H 'Access-Control-Request-Headers: content-type'
4.1 replay answer: 204 · access-control-allow-credentials: true · access-control-allow-headers: content-type · access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE · vary: Origin, Access-Control-Request-Headers · access-control-allow-origin: absent

Production answered the preflight 204, and the browser still never sent the save: the header it needs was absent, while the others and Vary came back, so the list refused it (steps 4, 6). The replay shows why without a browser (step 5). Staging's value has no slashes, so the same build echoed the origin and saved (steps 8–9).

4.2 Ticketing MR !3300: refuse a bad entry at start

The box office's API shipped the same variable with release 4.0. MR !3300 adds a startup check that compares each entry with new URL(entry).origin and stops the process on a difference; the pipeline then fixes the value:

4.2 box office release 4.0 CORS_ORIGINS "https://tickets.example/,https://boxoffice.tickets.example/" · MR !3300 startup check: refused to start: "https://tickets.example/" is not an origin (new URL(entry).origin is "https://tickets.example")
4.2 corrected CORS_ORIGINS "https://tickets.example,https://boxoffice.tickets.example" · MR !3300 startup check: startup check passed
4.2 preflight from https://boxoffice.tickets.example to the corrected API: 204 · Access-Control-Allow-Origin https://boxoffice.tickets.example

The bad value now stops a pod before it serves a single request, and the corrected one echoes the box office's origin (steps 10–11).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A streaming service's release 3.2 moves its progress API's allowed origins into a variable:

Production's manifest, copied from a table in the team wiki:

CORS_ORIGINS: "https://watch.example, https://tv.watch.example

The web player (https://watch.example) and the TV app (https://tv.watch.example) both save progress with a JSON PUT and credentials.

After the release, which apps' preflights come back with an Access-Control-Allow-Origin that names the app?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(','), credentials: true }));   // cors 2.8.6

Question 2 of 7

Ten minutes into a flash sale, every payment from the checkout page on https://shop.example fails with a CORS error in the console. Release 8.1, ten minutes earlier, moved the payments API's allowed origins into a variable:

CORS_ORIGINS: "https://shop.example/,https://admin.shop.example/

The checkout page pays with:

fetch('https://api.shop.example/payments', { method: 'POST', credentials: 'include', headers: { 'Content-Type': 'application/json' }, body });

Which change makes payments from the checkout page go through again?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(','), credentials: true }));   // cors 2.8.6

Question 3 of 7

Release 7.3 of a chat API reads its allowed origins from a variable; the same build runs everywhere:

# staging
CORS_ORIGINS: "https://staging.chat.example
# production, and the developers' .env.local
CORS_ORIGINS: "https://chat.example/

Deployed, the web app on https://chat.example sends with fetch('https://api.chat.example/messages', …), and on staging the app on https://staging.chat.example with fetch('https://api.staging.chat.example/messages', …). Locally, the dev server serves the app on http://localhost:5173 with server.proxy: { '/api': 'http://localhost:8080' }, the API runs on http://localhost:8080, and the app sends with fetch('/api/messages', …). Every send is a JSON POST with credentials.

Where does sending a message work?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(','), credentials: true }));   // cors 2.8.6

Question 4 of 7

After EXH-3000, ExamHall's API trims each entry and gets a startup check on CORS_ORIGINS: if any entry fails the check, the pod refuses to start.

A week later the proctor console is added to production, its entry copied from the load balancer's listener (proctor.examhall.io:443):

# production
CORS_ORIGINS: "https://app.examhall.io, https://proctor.examhall.io:443
# staging
CORS_ORIGINS: "https://staging.app.examhall.io, https://staging.proctor.examhall.io

The pages run at https://app.examhall.io and https://proctor.examhall.io, and on staging at https://staging.app.examhall.io and https://staging.proctor.examhall.io.

Which check refuses production's value and accepts staging's?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(',').map((e) => e.trim()), credentials: true }));   // cors 2.8.6

Question 5 of 7

In the interview you are given EXH-3000's page and asked: "Saving works in staging and fails in production with a CORS error.

Go." What do you answer, in at most 150 words?

Your answer (optional, stays on this page)

Question 6 of 7

A box office's staff page holds seats through https://api.tickets.example. Staff open the page from the link in the staff handbook, https://Admin.Tickets.example, and the API's value was copied from the same handbook:

CORS_ORIGINS: "https://tickets.example,https://Admin.Tickets.example

fetch('https://api.tickets.example/holds', { method: 'POST', credentials: 'include', headers: { 'Content-Type': 'application/json' }, body });

The API's access log shows OPTIONS /holds 204 for every hold the staff page tries.

What happens to the hold?

app.use(cors({ origin: process.env.CORS_ORIGINS.split(','), credentials: true }));   // cors 2.8.6

Question 7 of 7

Since last night's platform changes, every comment that viewers post from a film club's watch-party page (https://party.watch.example) fails with a CORS error in the console. The on-call engineer replays a comment's preflight from outside:

curl -i -X OPTIONS https://api.watch.example/parties/p-9/comments -H 'Origin: https://party.watch.example' -H 'Access-Control-Request-Method: POST' -H 'Access-Control-Request-Headers: content-type'

The answer is 204 No Content, with no header besides Date and Connection.

Which cause does this preflight answer point to?

Go deeper

Pro tips

Reproduce the preflight with curl, not with the app. curl -i -X OPTIONS <api-url> -H 'Origin: <the page's origin>' -H 'Access-Control-Request-Method: POST', then read the Access-Control-* headers yourself; a browser hides them behind a TypeError (curriculum row's own tip). When: steps 5–6, before you touch the config. Status: EXECUTED.
On a refused preflight, look for Vary: Origin. cors 2.8.6 adds it when it checks a request's Origin against a list, so Vary: Origin beside a missing Access-Control-Allow-Origin says the list was consulted and refused the origin. When: step 6, reading the replay. Status: EXECUTED.

Source

Browser and library behaviour can change: the facts below are as of 2026-10-06. No item is keyed on a source; every key is executed. Valid as of 2026-10-06.

WHATWG Fetch Standard, CORS check: https://fetch.spec.whatwg.org/ (fetch-standard.txt), the curriculum row's first source. Quoted: "Let origin be the result of getting Access-Control-Allow-Origin from response's header list. If origin is null, then return failure."; "If the result of byte-serializing a request origin with request is not origin, then return failure."; "An ok status is a status in the range 200 to 299, inclusive." Supports steps 4, 6 and 8.
MDN, Cross-Origin Resource Sharing (CORS): https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS (mdn-cors.txt), the row's second source. Quoted: "When responding to a credentialed requests request, the server must specify an origin in the value of the Access-Control-Allow-Origin header, instead of specifying the * wildcard." Supports step 10.
expressjs cors README: https://github.com/expressjs/cors (expressjs-cors-readme.txt), the row's third source, and the 2.8.6 source in node_modules/cors/lib/index.js (isOriginAllowed: return origin === allowedOrigin;). Quoted: "Array - set origin to an array of valid origins."; "String - set origin to a specific origin."; "set origin to true to reflect the request origin"; "optionsSuccessStatus: Provides a status code to use for successful OPTIONS requests, since some legacy browsers (IE11, various SmartTVs) choke on 204."
WHATWG HTML Standard, serialization of an origin: https://html.spec.whatwg.org/multipage/browsers.html (html-origin.txt). Quoted: "let result be origin's scheme. Append "://" to result. Append origin's host, serialized, to result. If origin's port is non-null, append a U+003A COLON character (:), and origin's port". Supports step 7.
MDN, Origin header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Origin (mdn-origin.txt). Quoted: "The Origin header is similar to the Referer header, but does not disclose the path". Supports step 7.
MDN, Forbidden request header: https://developer.mozilla.org/en-US/docs/Glossary/Forbidden_request_header (mdn-forbidden-request-header.txt). Quoted: "A forbidden request header is an HTTP header name-value pair that cannot be set or modified programmatically in a request."
WHATWG URL Standard: https://url.spec.whatwg.org/ (url-standard.txt), for new URL() and its origin getter.
MDN, Access-Control-Allow-Origin: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Access-Control-Allow-Origin (mdn-acao.txt). Supports step 6.
Vite, Server Options, server.proxy: https://vite.dev/config/server-options (vite-server-options.txt). Quoted: "Any requests whose request path starts with that key will be proxied to the specified target."

Checked against the tools and docs as of 2026-10-06.

Predict first