Closes#45624
Part 1 of #45553 -- see there for the full behavioral contract and
Oracle.
## Changes
- New `orbit/pkg/backoff` package: shared, stateful exponential backoff
tracker with jitter, thread-safe, per-path isolation. This package will
serve all agent components that need backoff (orbit API, fleetd paths,
and potentially osquery TLS), but for now only Fleet Desktop uses it. We
are introducing it incrementally to reduce risk.
- Integrated into Fleet Desktop's `checkToken` retry loop -- the exact
tight-retry path that caused the #44816 DB outage. The main
ping/DesktopSummary loop does not need backoff (Ping is unauthenticated
with no DB cost; DesktopSummary already runs at most every 5 min).
- On error: interval doubles each failure (1s, 2s, 4s, 8s, ...) capped
at 5 minutes
- On success: resets immediately to normal polling interval
- Each communication path tracks its own backoff independently
## Manual testing
### Automated tests (17 total, all pass with -race)
\`\`\`
go test ./orbit/pkg/backoff/ -v -race -count=1 # 17 tests, 0 failures
make lint-go-incremental # 0 issues
\`\`\`
- 14 logic tests (exponential doubling, cap, jitter, reset, per-path
isolation, concurrent access, overflow detection, garbage input
flooring)
- 3 real-time ticker tests (actual time.Ticker with wall-clock
measurements)
### Local TUF end-to-end test (macOS)
Set up local TUF server via \`tools/tuf/test/main.sh\` with
\`SYSTEMS=macos FLEET_DESKTOP=1 GENERATE_PKG=1\`. This builds orbit and
Desktop from this branch, generates \`fleet-osquery.pkg\` with local TUF
root keys. Installed the package on macOS, enrolled to a local Fleet
server.
**Test: corrupt token to simulate #44816 expired-token scenario**
Wrote invalid token to \`/opt/orbit/identifier\`, then watched Desktop
and orbit logs.
Desktop backoff (exponential doubling):
\`\`\`
11:57:21 ERR get device URL, backing off next_retry=2.044s (1s * 2^1 +
jitter)
11:57:29 ERR get device URL, backing off next_retry=4.061s (1s * 2^2 +
jitter)
11:57:39 ERR get device URL, backing off next_retry=8.744s (1s * 2^3 +
jitter)
\`\`\`
Orbit detects and rotates the token:
\`\`\`
11:57:42 INF token TTL expired, rotating token
\`\`\`
Desktop recovers instantly:
\`\`\`
11:57:48 DBG enabling tray items
\`\`\`
Previously Desktop would have retried every 5s indefinitely (#44816).
With backoff, retry intervals double each failure and recovery is
immediate on the first success.
### Build verification
\`\`\`
go build ./orbit/cmd/desktop/ # compiles clean
go build ./orbit/cmd/orbit/ # compiles clean
\`\`\`
---
# Checklist for submitter
- [x] Changes file added for user-visible changes in \`orbit/changes/\`.
- [x] Input data is properly validated, no SQL changes, no JS changes.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops (backoff caps at 5 min).
- [x] Added/updated automated tests (17 tests, all pass with \`-race\`).
- [x] QA'd all new/changed functionality manually (local TUF e2e on
macOS).
## fleetd/orbit/Fleet Desktop
- [x] If the change applies to only one platform, confirmed that
\`runtime.GOOS\` is used as needed to isolate changes (backoff is
platform-agnostic).
- [x] Verified that fleetd runs on macOS (local TUF install + e2e test).
Linux/Windows need QA.
- [ ] Verified auto-update works from the released version of component
to the new version.
Sharon Katz
4221903eb2