Side-partner operations¶

This page covers day-2 operational expectations for a live Path C integration: token rotation, the partner-tenant binding lifecycle, incident behaviors you can expect from FG.AI’s adapter during failures, capacity expectations, and the retirement procedure when you onboard as a FlexSync connector.

For onboarding, see side-partner-onboarding. For architecture, see side-partner-architecture. For mapping patterns, see side-partner-interface-mapping.

Token rotation¶

Both sides rotate credentials periodically per security policy. The procedures:

When you rotate your `(partner_app_id, partner_app_secret)`¶

Notify FG.AI partner-engineering via the agreed operational channel before the rotation goes live on your side.
Provide the new credentials via the agreed secure channel.
FG.AI updates SecretsManager + force-refreshes the adapter’s cached token. Verification: the next outbound POST succeeds with the new credentials.
If both old and new credentials are valid during a rotation window, the adapter can use either. If you immediately invalidate the old credentials, coordinate timing tightly with FG.AI to avoid an auth-failure window.

When FG.AI rotates the partner-bearer-JWT credentials FG.AI issued to you (for `wms-partners/v1/`)¶

FG.AI notifies you via the agreed operational channel.
FG.AI provides the new (partner_app_id, partner_app_secret) via the agreed secure channel.
Update your inbound polling client to use the new credentials.
Verify the next poll against wms-partners/v1/inventory-delta returns valid data.

Rotate during low-volume operational windows when possible.

When FG.AI rotates the KMS signing key (transparent to you)¶

The RS256 signing key behind wms-partners/v1/authenticate is rotated on a periodic schedule (typically annual; or out-of-cycle on suspected compromise). The rotation is designed to be transparent to verifying partners:

During a rotation window (default 7 days), the JWKS endpoint at GET /wms-partners/v1/.well-known/jwks.json returns BOTH the new and the retiring RSA public keys.
Already-issued JWTs continue validating against the retiring key (matched by their kid header claim) until they expire naturally.
Newly-issued JWTs are signed with the new key and carry a new kid.
After the rotation window, the retiring key is removed from the JWKS response.

Your JWT library (Spring Security NimbusJwtDecoder, Node jose, Python PyJWT, Go golang-jwt, etc.) handles kid selection automatically as long as you (a) refresh JWKS on kid not found (most libraries do this by default) and (b) respect the Cache-Control: max-age=300 header rather than caching JWKS indefinitely. No action required from you for routine rotations.

If a kid lookup against a freshly-refreshed JWKS still fails, contact FG.AI partner-engineering — that indicates a token issued against a key that has fully retired (your client is presenting a stale token) OR a partition mismatch (intl JWKS being used to verify a CN-issued token, or vice versa).

Partner-tenant binding lifecycle¶

Every (partner, tenant) pair is recorded in the WMS partner-registry with a three-state lifecycle. You can think of this as the contract between your engagement and FG.AI’s infrastructure:

State	Meaning	Operational behavior
`active`	The adapter is running in production; events flow normally.	Normal steady-state.
`migrating`	The tenant is being migrated to a FlexSync connector (post-FlexSync-v1 GA).	Adapter still runs; FlexSync connector is being provisioned in parallel. Coordinated cutover window.
`retired`	Migration complete; the adapter’s Helm release for this tenant is decommissioned.	The row is preserved for audit; FlexSync now serves the tenant.

Transitions:

active → migrating is initiated by FG.AI when FlexSync v1 GA’s and your migration is scheduled.
migrating → retired is initiated after FG.AI verifies the FlexSync connector is operating correctly during the comparison window.
active → retired (skipping migrating) is rare; happens when a tenant is fully deprovisioned (you no longer serve them, or you exit the engagement) without a FlexSync alternative.

The MAX_SIDE_PARTNERS=2 budget counts unique partners with any active or migrating binding. A retired-only partner doesn’t count.

Adapter incident behaviors you can expect¶

During failures, the adapter behaves in well-defined ways. This section explains what you’ll observe and what FG.AI is doing.

Authentication failures (your `/authenticate` returns 401 or your auth-failed code)¶

The adapter:

Invalidates its cached token.
Retries the outbound POST once with a fresh token.
If the second attempt also fails: fires an internal alert + routes the message to the dead-letter table.

What you’ll observe: a small number of 401s in your auth-endpoint logs during a credential rotation. If you see a sustained 401 storm from the adapter, contact FG.AI partner-engineering — credentials may be out of sync or there’s a clock-skew issue.

Your endpoint returns 5xx (“system error”)¶

The adapter:

Retries with exponential backoff per the configured retry policy (typical defaults: 5 attempts; 1s initial, 30s max backoff).
If retries exhaust, routes to the dead-letter table.
If the failure rate exceeds the configured threshold (default 50% over 10 calls), opens a circuit breaker for a configured duration (default 60 seconds) to protect your endpoint from a thundering-herd retry storm.

What you’ll observe: retries with backoff during a brief outage; quiet period (no retries) when the circuit is open; resumption when the circuit closes. Do NOT contact FG.AI to “override” the circuit breaker — it’s protecting your endpoint from a flood when you come back up.

After recovery, FG.AI operators can manually replay the dead-letter table to drive deferred events to your endpoint. You may see a burst of catch-up POSTs.

Your endpoint rate-limits the adapter (429 or your equivalent)¶

The adapter backs off. If sustained, FG.AI partner-engineering reaches out to coordinate capacity (longer backoffs on FG.AI’s side, larger shift windows reducing event rate, your side increasing capacity, etc.).

Dead-letter table growth¶

If the adapter cannot drain its dead-letter table because outbound POSTs systematically fail (sustained outage, persistent shape mismatch, credential issue), FG.AI operators see the table grow. They contact you when growth exceeds operational thresholds. Do NOT assume FG.AI will bulk-discard dead-lettered events without coordination — partner-engineering reviews them with you to determine cause and disposition.

Postgres state loss (the per-`(partner, tenant)` schema is corrupted or restored from backup)¶

The adapter recovers:

The cached token table is recoverable — the adapter re-authenticates on the next outbound POST. Brief latency spike, no business impact.
The aggregation state is recoverable by replaying Kafka from the last shift-window-close boundary. Per-tenant consumer group makes this surgical.
The dedup table loss MAY produce duplicate POSTs to you. FG.AI coordinates with you on duplicate-tolerance per your idempotency policy. If your inbound consumer is strictly idempotent, the recovery completes without incident. If your inbound consumer is not strictly idempotent, FG.AI freezes the consumer until state is restored from backup. Plan your inbound consumer to be idempotent.

Capacity expectations¶

The per-(partner, tenant) pod has a small steady-state footprint:

~256MB memory request baseline
~50m CPU request baseline
Burst capacity per Resilience4j config

Your outbound endpoint capacity should be sized for:

Per-document postback: one POST per closed document in the relevant FG.AI document type
Inventory transitions postback: one POST per shift window (typically per warehouse, configurable)
Catch-up bursts after outages: if the adapter has been in a circuit-open state for the full open-duration window, the catch-up volume after recovery can be 2-3x normal for a few minutes

Your /authenticate endpoint sees periodic refresh requests (typically once per TTL minus refresh-lead — default ~115 minutes for a 120-minute TTL) plus one-off refreshes on 401 events.

Reconciliation expectations¶

The Path C integration is one-way per leg. The adapter does not round-trip your inbound writes back to you, and treats the synchronous HTTP response on outbound (status code + your response-body business code per interface-mapping § Error code envelope conventions) as the only application-level ack — no separate async confirmation message is expected from your side after the POST completes. This means:

Reconciliation is your responsibility on the inbound leg. Periodic comparison between your authoritative state and what you wrote into FG.AI catches drift.
Reconciliation is your responsibility on the outbound leg. Poll wms-partners/v1/inventory-delta to confirm FG.AI’s state matches what you’ve received via outbound postbacks. Discrepancies likely indicate an incident on the adapter or an aggregation timing issue.

If you observe persistent reconciliation drift, contact FG.AI partner-engineering with the specific events that don’t match. Drift is investigated; it is not a known steady-state condition.

FlexSync v1 (the path you migrate to when FlexSync GA’s) includes reconciliation as a framework capability. Until then, you reconcile on your side.

Retirement procedure (when you onboard as a FlexSync connector)¶

When FlexSync v1 GA’s and you are scheduled to onboard as a FlexSync {your}-wms connector, the adapter’s per-tenant Helm releases are decommissioned. Per the Path C agreement, this is a contractual milestone, not optional cleanup.

The procedure (you will be a participant, not a passive observer):

Pre-cutover¶

FG.AI confirms FlexSync v1 is in production.
FG.AI provisions your FlexSync connector instance(s) in parallel.
FlexSync smoke-tests against your dev environment.

Migrating (binding `active → migrating`)¶

FG.AI updates your partner_tenant_binding rows to status='migrating'.
The adapter pod(s) continue running normally — no event flow change.
The FlexSync connector begins shadow-consuming the same Kafka topics in parallel.
FG.AI compares adapter outbound vs FlexSync outbound for a comparison window (typically 48 hours). Discrepancies investigated with you.

Cutover¶

FG.AI schedules a cutover window with you (typically off-peak hours).
At cutover time T:
- Adapter side: drains Kafka consumer; commits offsets; waits for in-flight outbound POSTs to ack-or-DLT.
- FlexSync side: becomes the authoritative outbound source.
- Adapter side: pod stopped (scaled to zero replicas).
You verify outbound POSTs continue arriving — sourced from FlexSync, not the adapter.

Retired (binding `migrating → retired`)¶

FG.AI updates partner_tenant_binding to status='retired'.
Per-tenant Helm release deleted.
Per-tenant Postgres schema dropped after backup confirmation.
Per-tenant SecretsManager entry deleted (with 30-day recovery window).
Per-tenant IAM role + Kafka ACLs removed.

After your last tenant retires¶

When your last FG.AI tenant transitions to retired:

FG.AI archives the wes-{your}-adapter GitHub repo (history preserved; not deleted).
FG.AI updates the audit record for your engagement.
The MAX_SIDE_PARTNERS=2 budget reclaims your slot.
Your partner row in the WMS partner-registry remains, with integration_topology transitioned to FLEXSYNC — FlexSync continues serving you.

When to contact FG.AI partner-engineering¶

Reach out via the agreed operational channel when:

You observe a sustained authentication failure storm from the adapter.
You observe the adapter retrying past its expected backoff (suggests a configuration drift).
You observe missing outbound POSTs for events you know occurred in FG.AI (suggests an aggregation or dedup issue).
You observe reconciliation drift between wms-partners/v1/inventory-delta and what the adapter has posted to you.
You are planning capacity changes on your side (rate-limit changes, endpoint URL changes, credential rotations).
You have an active incident that may affect what you can accept from the adapter (so FG.AI can throttle or pause outbound coordinated with your recovery).
You are scheduling FlexSync onboarding (so FG.AI can initiate the migration sequence).

When you contact FG.AI, include:

Your partner code + FG.AI tenant ID(s) in scope
Approximate time range of the observed behavior
Sample event identifiers if you have them (your taskNo, woCode, or equivalent — FG.AI can correlate to the source Kafka offsets)

Steady-state observability you can run on your side¶

You can observe Path C operational health from your side via:

Inbound endpoint logs — auth-endpoint hits, postback receipts, response codes you return. Spike in 401 from the adapter indicates a credential or clock issue. Sustained 5xx from you reflects your own incident.
wms-partners/v1/inventory-delta polling — your authoritative reconciliation surface. Poll rate is your choice; FG.AI does not impose a minimum cadence.
Periodic comparison between your authoritative state and what the adapter has posted to you. Tolerance is partner-specific; document during onboarding.

FG.AI maintains operational dashboards on its side covering token-refresh health, outbound POST success rate, circuit-breaker state, DLT growth, Kafka consumer lag, and per-tenant audit-event throughput. FG.AI partner-engineering can share specific metrics on request.