Persistent 403 On All API Calls — Auth Token Not Accepted After Login (Android)

Metadata

• Date: 2026-04-18
• Status: root-cause-identified
• Severity: critical
• Related issue/ticket: N/A
• Owner: N/A

About

Overview: - On Android, API calls return 403 even after logging out and back in. The memory feed fails at app startup, and the session disappears mid-run (auth-gate kicks the user to login from Settings while the app appeared fully authenticated). - The 403 blocks all protected API access. The user can't view memories or interact with their data.

Technical Questions: - All 403s originate from the API Gateway Cognito authorizer (before any Lambda is invoked). Lambda AuthError returns status_code=401; API Gateway authorizer rejection returns 403. This means the token itself is being rejected at the Gateway level. - auth_retry_requested fires (not auth_retry_aborted), so the Cognito token refresh via refreshSession SUCCEEDS and returns a new idToken. Yet the retried request with the fresh token ALSO gets 403. This is the key anomaly. - The session disappears mid-run: logs show auth-gate redirect_to_login {"first_segment": "settings"} while the app had just successfully captured audio. The only code paths that call setSession(null) are signOut() and the bootstrap catch/fallthrough. One of these ran unexpectedly. - The app's .env has EXPO_PUBLIC_COGNITO_USER_POOL_ID=us-east-1_WvhXc4lRH and EXPO_PUBLIC_COGNITO_APP_CLIENT_ID=2f6n62s81ic3gm084mgi71b6hn. If AWS SSM /encache/auth/user_pool_id or /encache/auth/app_client_id no longer match these values (e.g., Cognito pool was recreated by Terraform), the Gateway would reject all mobile-issued tokens. - The 4-key SecureStore split (commit 719be108) changed how sessions are persisted. A partial write failure (one key missing) would cause getStoredSession to return null on next load, but this should surface as has_stored_session: false in bootstrap logs.

Resources: - main/app/lib/api/getApi.ts — response interceptor, attemptTokenRefresh, getOrStartRefresh - main/app/lib/api/auth/session.ts — getValidIdToken, persistSession, clearSession - main/app/lib/api/auth/secure-store.ts — getStoredSession, setStoredSession (4-key split) - main/app/lib/api/auth/auth-provider.tsx — bootstrap flow, setSession(null) paths - main/app/lib/api/auth/cognito.ts — EXPO_PUBLIC_COGNITO_USER_POOL_ID, EXPO_PUBLIC_COGNITO_APP_CLIENT_ID - main/server/template.yaml:68 — API Gateway authorizer uses {{resolve:ssm:/encache/auth/user_pool_id}} - main/server/layers/shared/python/shared/lambda_helpers.py:119 — _verify_jwt (returns 401, not 403 — confirms 403 = Gateway) - main/devops/main.tf:386 — SSM parameters /encache/auth/user_pool_id and /encache/auth/app_client_id

Steps to cause failure

flowchart LR
  AppStart --> FeedFires["Memory feed fires\n(bootstrap still loading)"]
  FeedFires --> NoToken["getValidIdToken() → null\n(sessionCache cold, SecureStore async)"]
  NoToken --> NoHeader["Request sent\nwithout Authorization header"]
  NoHeader --> GW403["API Gateway → 403\n(no auth header = Cognito rejects)"]
  GW403 --> Retry["Response interceptor:\nrefreshSession() → succeeds\nnew idToken obtained"]
  Retry --> Retry403["Retry with fresh token\n→ still 403\n(WHY? — open question)"]
  Retry403 --> Exhausted["auth_retry_exhausted"]

flowchart LR
  MidRun["App running normally\n(capture worked)"] --> NavSettings["User navigates to Settings"]
  NavSettings --> SessionNull["session React state = null\nhas_session: false"]
  SessionNull --> Redirect["Auth-gate kicks to login"]

System

flowchart TD
  MobileApp -->|EXPO_PUBLIC_COGNITO_USER_POOL_ID| CognitoPool["Cognito User Pool\nus-east-1_WvhXc4lRH"]
  CognitoPool -->|issues idToken| MobileApp
  MobileApp -->|Authorization: Bearer idToken| APIGW["API Gateway"]
  APIGW -->|validates token against| SSMPool["SSM /encache/auth/user_pool_id\n(resolved at deploy time)"]
  SSMPool -->|pool mismatch? → 403| APIGW
  APIGW -->|token valid → invokes| Lambda["Lambda\n(AuthError → 401, not 403)"]

Reproduction Details

1. Cold-start the app on Android.
2. Observe auth_retry_exhausted {"status": 403} for /memories/feed before capture starts.
3. Navigate to Settings → observe redirect_to_login {"first_segment": "settings"} and has_session: false.
4. Log out and back in via magic link.
5. Observe 403 persists on protected endpoints.

Reproduction test: N/A — requires live AWS + Cognito; cannot be unit-tested without mocking API Gateway.

Notes for PR

Two distinct failure modes identified:

Failure 1 — Race condition at startup (frontend): getValidIdToken() in the request interceptor returns null when sessionCache is cold and SecureStore hasn't finished loading during bootstrap. The memory feed fires immediately on mount before bootstrap() completes. The fix would be to gate the feed load on loading === false from AuthProvider, or to await session load before making the first API call.

Failure 2 — Persistent 403 even with fresh token (root cause TBD): The refreshed token (from refreshSession() → Cognito) is ALSO rejected by API Gateway. The most likely cause is a Cognito config mismatch: EXPO_PUBLIC_COGNITO_USER_POOL_ID/EXPO_PUBLIC_COGNITO_APP_CLIENT_ID in .env don't match the SSM values the backend was deployed with. To verify:

# Run in your AWS-authenticated terminal:
aws ssm get-parameter --name /encache/auth/user_pool_id --query Parameter.Value --output text
aws ssm get-parameter --name /encache/auth/app_client_id --query Parameter.Value --output text

Compare against .env: - EXPO_PUBLIC_COGNITO_USER_POOL_ID=us-east-1_WvhXc4lRH - EXPO_PUBLIC_COGNITO_APP_CLIENT_ID=2f6n62s81ic3gm084mgi71b6hn

If they differ, update .env to match SSM and rebuild the app. If they match, check CloudWatch Logs for the API Gateway authorizer to see the exact rejection reason.

Failure 3 — Session disappearing mid-run: The auth-gate fires redirect_to_login while the app had a valid session. The only setSession(null) paths outside of explicit signout are: bootstrap error catch, and bootstrap fallthrough (stored session missing + refresh fails). Bootstrap re-runs only if logger reference changes. Needs log evidence of bootstrap_started or token_refresh_failed firing mid-session.

Audit Log

Verification

• [ ] Reproduced failure before fix
• [ ] Reproduction test fails before fix
• [ ] Root cause identified with evidence
• [ ] Fix applied at source (no workaround-only patch)
• [ ] Reproduction test passes after fix
• [ ] Reproduction path now passes
• [ ] Regression test added/updated (or N/A with reason)
• [ ] Verified no duplicate solved-bug log exists for same root cause