# CMS ACCESS API — Resilience Configuration Guide

## Overview

The CmsClient implements production-grade error handling with structured exceptions, configurable retry logic, and per-environment timeout settings. This document describes all resilience knobs and recommended values.

---

## Structured Exception Hierarchy

All CMS errors inherit from `CmsApiError` and carry a `retryable` flag:

| Exception | Status Code(s) | Retryable | Action Required |
|-----------|---------------|-----------|-----------------|
| `CmsConnectionError` | N/A (network-level) | **Yes** | Transient — retry with backoff |
| `CmsServerError` | 500–599 | **Yes** | Transient — retry with backoff |
| `CmsRateLimitError` | 429 | **Yes** | Back off longer, then retry |
| `CmsAuthError` | 401, 403 | No | Check credentials/scope |
| `CmsValidationError` | 400 | No | Fix request payload |

### Usage Example

```python
from access_core.services.cms_client import CmsClient, CmsApiError

client = CmsClient()

try:
    result = client.post_operation("Patient", "$check-eligibility", params)
except CmsAuthError:
    # Credentials problem — don't retry
    logger.error("CMS auth failed — check CMS_API_CLIENT_ID/SECRET")
except CmsValidationError as exc:
    # Bad request data — fix and resubmit
    logger.error(f"Validation error: {exc.response_data}")
except CmsApiError as exc:
    if exc.retryable:
        logger.warning(f"Transient CMS error, retry later: {exc}")
    else:
        logger.error(f"CMS error (non-retryable): {exc}")
```

---

## Retry Configuration

### POST Operations (eligibility, alignment, reporting, unalignment)

| Setting | Env Var | Django Key | Default | Description |
|---------|---------|-----------|---------|-------------|
| `CMS_POST_MAX_RETRIES` | `CMS_POST_MAX_RETRIES` | `CMS_POST_MAX_RETRIES` | 3 | Max attempts for transient POST errors |
| `CMS_POLL_INTERVAL_SECONDS` | `CMS_POLL_INTERVAL_SECONDS` | `CMS_POLL_INTERVAL_SECONDS` | 5 | Base interval for exponential backoff (seconds) |

**Backoff formula:** `min(poll_interval × 2^(attempt-1), 300)` — caps at 5 minutes.

**What gets retried:**
- Connection errors (`requests.exceptions.ConnectionError`)
- Timeouts (`requests.exceptions.Timeout`)
- Server errors (HTTP 5xx)
- Rate limits (HTTP 429)

**What does NOT get retried:**
- Auth failures (401, 403) — raises `CmsAuthError` immediately
- Validation errors (400) — raises `CmsValidationError` immediately
- Client errors (4xx except 429) — raises generic `CmsApiError` immediately

### Polling ($submission-status)

| Setting | Env Var | Django Key | Default | Description |
|---------|---------|-----------|---------|-------------|
| `CMS_POLL_INTERVAL_SECONDS` | `CMS_POLL_INTERVAL_SECONDS` | `CMS_POLL_INTERVAL_SECONDS` | 5 | Base seconds until a row's next scheduled poll (`next_poll_at`) |

There is no blocking poll loop: services submit and return immediately, recording a PENDING `CMSApiSubmissionStatus` row. All `$submission-status` checks run in the polling worker — each `poll_cms_submissions` pass performs one status check per due row. Overlapping runs (worker vs manual `--force`) coordinate through an atomic per-row claim (`claim_submission_row`), so a submission is not status-checked twice at once; rows lost to another run are reported as skipped, and every final-result handler is idempotent under its own row lock / conditional update. Retryable failures (429/5xx) reschedule with exponential backoff; "in progress" responses reschedule `next_poll_at`; terminal 4xx marks the row ERROR.

### Documented CMS Polling Behavior

Per the **ACCESS API Operations Manual v0.9.12** (published in the official IG repo `DSACMS/cmmi-access-model`, file `ACCESS API Operations Manual.pdf`):

- All operations are asynchronous: POST returns HTTP 202 + a `Content-Location` header pointing at `$submission-status`.
- **Initial poll:** wait 5–10 seconds after submission before the first status check.
- **Subsequent polls:** every 10–30 seconds while receiving HTTP 202 Accepted.
- **Maximum duration:** implement a timeout (recommended: **5 minutes**) after which the client should stop polling and notify the user; "the actual timeout duration should be configured based on operational requirements."
- **Resume requirement:** under "Handling Network Interruptions" the manual directs clients to *"store submission IDs and associated request context before starting to poll"* and to *"implement resume logic that can continue polling after network restoration."*

**Implementation status:** **the resume pattern is implemented and polling is fully worker-driven.** Every submission is recorded in `CMSApiSubmissionStatus` (submission ID, endpoint, target record, next poll time) the moment the service submits; each poll attempt is logged to `CMSApiRequest`/`CMSApiResponse`. Services are submit-only — no request thread ever blocks on `$submission-status` — so there is no per-request timeout budget to tune: rows simply stay PENDING/IN_PROGRESS with a rescheduled `next_poll_at` until the worker applies a terminal result. The poller runs everywhere as a scheduled loop: staging/prod deploy a single-instance ECS service (`manage.py cms_polling_worker`) that invokes `poll_cms_submissions` every `CMS_POLL_LOOP_SECONDS` (default 60s); dev/local run it via the `cms-worker` service in `docker-compose.yml`. The command remains available for manual/one-off runs (`--force` bypasses `next_poll_at`). What remains is the transport gap: the submit POST still executes synchronously inside signal handlers (bounded by `CMS_POST_MAX_RETRIES`, seconds rather than minutes) and the webhook SQS queue has no consumer yet (see Review Guide Appendix A #3/#4).

**Note on processing time:** CMS does not document a multi-hour processing SLA for ACCESS operations; the 5-minute recommended client timeout is the authoritative figure. The third-party mock server's README speculates the real API may take "minutes/hours" — treat that as unverified until the production endpoint exists (no public ACCESS sandbox is available yet).

**Do not confuse with BCDA:** `sandbox.bcda.cms.gov` is the *Beneficiary Claims Data* API — a different CMS product for bulk claims retrieval (data lands 2–14 days after submission, updates weekly). It is **not** an ACCESS Model endpoint. No working public URL exists for `CMS_API_BASE_URL` today; dev/staging point at the local mock, and the value will be set when CMS publishes the real API.

---

## Timeout Configuration

| Setting | Env Var | Django Key | Default | Description |
|---------|---------|-----------|---------|-------------|
| `CMS_REQUEST_TIMEOUT_SECONDS` | `CMS_REQUEST_TIMEOUT_SECONDS` | `CMS_REQUEST_TIMEOUT_SECONDS` | 60 | Per-request timeout (POST, GET poll, OAuth) |

OAuth token requests are capped at `min(request_timeout, 30)` seconds.

### Recommended Values Per Environment

| Environment | Timeout | POST Retries | Worker Loop | Notes |
|-------------|---------|--------------|-------------|-------|
| **Development** | 60s | 1 (disabled) | 2s (compose default) | Fast feedback loop; mock server is local |
| **Staging** | 120s | 3 | 60s | Realistic timeouts; moderate retry budget |
| **Production** | 180s | 3 | 60s | Durable re-poll via the worker (see above) |

---

## Signal Exception Logging

All CMS integration signals (alignment, unalignment, reporting) now log with correlation IDs and structured format:

```
[cms-signal] alignment triggered for IntakeRecord #abc-123 | corr=f4a2b8c9d1e7
[cms-signal] alignment succeeded for IntakeRecord #abc-123 | corr=f4a2b8c9d1e7 — Beneficiary=xyz-789
[cms-signal] unalignment failed for Beneficiary #xyz-789 | corr=a1b2c3d4e5f6 | type=CmsServerError | retryable=True
[cms-signal] reporting skipped for OAPMeasureObservation #obs-001 — recent unalignment exists
```

**Log fields:**
- `[cms-signal]` prefix — filter with `grep '\[cms-signal\]'` or structured log parsers
- Operation: `alignment`, `unalignment`, `reporting`
- Entity type + ID: traceable to Django model instance
- Correlation ID: 12-char hex, unique per signal invocation
- Exception type + retryability flag (on errors)

---

## Malformed Response Handling

The client gracefully handles non-FHIR responses:

| Scenario | Behavior |
|----------|----------|
| HTML error page (500) | Captured in `response_data["raw"]`; raises `CmsServerError` |
| Empty body on 200 | Returns `{"raw": "", "_http_status": 200}` — no crash |
| Invalid JSON | Wrapped as `{"raw": resp.text}`; logged at ERROR level |
| Partial response | Handled same as invalid JSON; the row stays open and is retried on its next scheduled poll |

---

## Environment Variables Summary

```bash
# Base URL — no public ACCESS sandbox exists yet; the mock is the working target.
# (sandbox.bcda.cms.gov is a different CMS product, NOT an ACCESS endpoint.)
CMS_API_BASE_URL=http://l.nso.me:3001          # Mock server (dev)

# Auth (optional for mock; required for real API)
CMS_API_CLIENT_ID=                               # Empty = skip OAuth
CMS_API_CLIENT_SECRET=                           # Empty = skip OAuth
CMS_FHIR_VERSION_PATH=""                         # "" for mock, "R4" when the real ACCESS endpoint is available
# URL path templates — empty = built-in defaults matching the CMS Operations Manual.
# The doc's example URLs are not guaranteed, so if the real endpoints differ, set these:
CMS_API_POST_PATH=                               # default /fhir/{version}/{resource_type}/{id_suffix}/{operation}
CMS_API_SUBMISSION_STATUS_PATH=                  # mock form: /fhir/Patient/$submission-status?submissionID={submission_id}
CMS_API_TOKEN_TTL_SECONDS=3540                   # Token cache TTL (seconds)

# Retry & Resilience
CMS_POST_MAX_RETRIES=3                           # POST retry attempts
CMS_POLL_INTERVAL_SECONDS=5                      # Seconds until a row's next scheduled poll
CMS_POLL_LOOP_SECONDS=60                         # Worker loop cadence (cms_polling_worker)
CMS_REQUEST_TIMEOUT_SECONDS=60                   # Per-request timeout
```

---

## Monitoring Recommendations

1. **Alert on non-retryable errors:** `CmsAuthError`, `CmsValidationError` indicate misconfiguration
2. **Track retry rates:** High retry frequency suggests infrastructure issues (network, CMS downtime)
3. **Correlation IDs:** Use to trace full lifecycle from signal → POST → poll → outcome
4. **Stale submissions:** Submissions awaiting a result persist as PENDING/IN_PROGRESS `CMSApiSubmissionStatus` rows — the `<name>-worker` ECS service (or the compose `cms-worker` in dev/local) drains them automatically (`cms_polling_worker`, every `CMS_POLL_LOOP_SECONDS`; manual fallback: `manage.py poll_cms_submissions --force`). Alert when rows sit past their `next_poll_at` while the worker is stopped or failing.