Troubleshooting

Troubleshooting

UI loads but flows fail

Symptom: The frontend renders but executing a flow returns errors or the Looking Glass timeline stays empty.

Cause: The frontend cannot reach the gateway.

Fix:

1. Verify gateway health: curl http://localhost:8080/health
2. If using docker run, ensure both containers are on the same Docker network and the gateway container is named gateway (the nginx config resolves gateway:8080).
3. If using Docker Compose, the networking is automatic.

SPIFFE returns 503

Symptom: SPIFFE flows return 503 Service Unavailable or “SPIFFE not available”.

Cause: The SPIFFE service is running in demo mode without SPIRE infrastructure.

Fix: Start the SPIRE overlay alongside the main stack:

docker compose -f docker-compose.yml -f docker-compose.spiffe.yml up -d

Demo mode only returns metadata and health. Full SVID issuance, mTLS, and attestation flows require SPIRE.

Empty protocol list in the gateway

Symptom: GET /api/protocols returns an empty list or gateway reports no_upstreams.

Cause: Gateway cannot reach upstream services.

Fix:

1. Check upstream health: curl http://localhost:8080/health/upstreams
2. Verify the upstream environment variables are set correctly (e.g. FEDERATION_SERVICE_URL, SCIM_SERVICE_URL).
3. Ensure upstream services are healthy before the gateway starts, or wait for the gateway to refresh (it polls upstreams periodically).

CORS errors in the browser console

Symptom: Browser shows Access-Control-Allow-Origin errors when calling the API.

Cause: The frontend origin is not in the service’s CORS allow list.

Fix: Set SHOWCASE_CORS_ORIGINS on every backend service to include your frontend URL:

SHOWCASE_CORS_ORIGINS=http://localhost:3000,http://localhost:5173

If your frontend is at a custom domain, add that origin.

Looking Glass OID4VCI shows “Failed to fetch”

Symptom: Pre-authorized OID4VCI flows fail immediately with OID4VCI Execution Failed / Failed to fetch.

Cause: Looking Glass redeems offers through the wallet harness. A cross-origin call to wallet.protocolsoup.com that returns an application 502 can be rewritten by the edge without Access-Control-Allow-Origin, which the browser surfaces only as Failed to fetch.

Fix:

1. Confirm the frontend rewrite is present: browser Network tab should show POST /wallet-harness/api/import (same origin), not a direct call to wallet.protocolsoup.com.
2. Confirm WALLET_BACKEND_ORIGIN points at the live wallet harness and that wallet /health is warm.
3. Confirm the wallet has WALLET_MDOC_IACA_ROOT_PEM for mso_mdoc imports and can reach issuer JWKS at the advertised jwks_uri (Credential Issuer Metadata / RFC 8414) for JWT/SD-JWT imports. The wallet rejects mdoc storage when no IACA root is configured or when the document-signer chain does not chain to that root.

SCIM requests return 401

Symptom: All SCIM API calls return 401 Unauthorized.

Cause: SCIM endpoints require a bearer token.

Fix: Set SCIM_API_TOKEN on the SCIM service, then include it in every request:

curl -H "Authorization: Bearer <your-token>" \
  http://localhost:8080/scim/v2/Users

WebSocket connection fails for Looking Glass

Symptom: Looking Glass events don’t stream; WebSocket connection is refused or drops.

Cause: Reverse proxy or load balancer is not forwarding WebSocket upgrades.

Fix:

• If using the default frontend image, WebSocket proxying is built into the nginx config (/ws path).
• If using a custom reverse proxy, ensure it supports Connection: upgrade and Upgrade: websocket headers on the /ws path.
• Check that the WebSocket path matches: /ws/lookingglass/{session_id}.
• Preserve the Sec-WebSocket-Protocol header in both directions. Looking Glass offers protocolsoup-lookingglass-v1 plus protocolsoup-lookingglass-owner.<session_token> and the backend selects the first value after validating the owner capability.

Production federation service fails during startup

Symptom: Startup reports that OAUTH2_REPLAY_REDIS_URL is required, must use rediss:// or a Fly private Upstash URL, or cannot connect.

Cause: Production private_key_jwt replay protection requires one shared, TLS-protected Redis store so every service instance reserves jti values atomically. The same variable also backs the RFC 9449 DPoP proof jti replay stores for both oauth2 and OID4VCI (a distinct key prefix and instance from the private_key_jwt store), so a missing or unreachable Redis blocks DPoP-bound requests as well, not only private_key_jwt.

Fix: Provision reachable Redis with TLS and set its rediss:// URL as a platform secret, or use Fly’s private redis://fly-*.upstash.io:6379 endpoint over encrypted 6PN. Verify connectivity from the federation service. Other plaintext Redis URLs and in-memory production stores remain rejected. A runtime Redis error after startup fails the affected token or resource request closed with server_error, rather than silently accepting an unverified proof.

Containers exit with OOM

Symptom: Containers restart or get killed due to out-of-memory errors.

Fix: Backend services are lightweight Go binaries and typically need less than 256MB. The frontend nginx server needs minimal memory. If you see OOM in constrained environments:

• Increase the container memory limit.
• Run fewer services (use single-service mode instead of full stack).

Fly deploy fails: insufficient CPUs on the current host

Symptom: flyctl deploy builds the image, then fails while updating an existing Machine:

aborted: could not reserve resource for machine: insufficient CPUs available to fulfill request on the current host

Cause: The app has a Fly Volume. Volumes live on one physical host, so Fly must update that Machine in place. If that host is out of spare CPUs, the update cannot start. Retrying the same deploy keeps targeting the same packed host. ProtocolSoup hits this on protocolsoup-wallet (protocolsoup_wallet_data, holding /data/device-key.pem) and on protocolsoup (protocolsoup_data).

Fix: Copy the volume onto a different host, move the Machine, then deploy again. Production CI does this automatically via .github/scripts/fly-deploy-or-migrate-host.sh. To do it by hand:

fly volumes list -a protocolsoup-wallet
fly machines list -a protocolsoup-wallet

# Fork onto a host that can run this VM size (unique zone is the default).
fly volumes fork <volume-id> -a protocolsoup-wallet \
  --vm-cpu-kind shared --vm-cpus 1 --vm-memory 256

fly machine clone <machine-id> -a protocolsoup-wallet \
  --attach-volume <new-volume-id>:/data \
  --vm-cpu-kind shared --vm-cpus 1 --vm-memory 256

# Confirm /data (and the wallet device key, if present) on the new Machine,
# then retire the packed-host Machine and volume so deploy cannot reattach them.
fly ssh console -a protocolsoup-wallet --machine <new-machine-id> \
  --command "ls -la /data"
fly machines destroy <old-machine-id> -a protocolsoup-wallet --force
fly volumes destroy <old-volume-id> -a protocolsoup-wallet -y

flyctl deploy --config fly.wallet.toml --app protocolsoup-wallet --remote-only

Do not destroy the source volume until the new Machine is serving from the fork. The fork is a point-in-time copy; later writes on the old volume are not replicated.

Port conflicts

Symptom: bind: address already in use when starting a service.

Cause: Multiple services default to port 8080. When running services directly (not through compose), you need unique host ports.

Fix: Map each service to a different host port:

docker run -p 8080:8080 ghcr.io/parlesec/protocolsoup-federation:latest
docker run -p 8081:8080 ghcr.io/parlesec/protocolsoup-scim:latest
docker run -p 8082:8080 ghcr.io/parlesec/protocolsoup-ssf:latest

When using Docker Compose, port mapping is handled automatically.

SSF receiver not receiving events

Symptom: SSF push delivery runs but the receiver shows no events.

Cause: The SSF receiver runs on a separate port (8081).

Fix:

1. Ensure port 8081 is exposed: docker run -p 8080:8080 -p 8081:8081 ghcr.io/parlesec/protocolsoup-ssf:latest
2. Verify the receiver is listening: curl http://localhost:8081/health
3. Check that SSF_RECEIVER_TOKEN matches between transmitter configuration and receiver authentication.