main
1
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Sharon Katz
|
4221903eb2 |
Add exponential backoff to Fleet Desktop server polling (#45623)
Closes #45624 Part 1 of #45553 -- see there for the full behavioral contract and Oracle. ## Changes - New `orbit/pkg/backoff` package: shared, stateful exponential backoff tracker with jitter, thread-safe, per-path isolation. This package will serve all agent components that need backoff (orbit API, fleetd paths, and potentially osquery TLS), but for now only Fleet Desktop uses it. We are introducing it incrementally to reduce risk. - Integrated into Fleet Desktop's `checkToken` retry loop -- the exact tight-retry path that caused the #44816 DB outage. The main ping/DesktopSummary loop does not need backoff (Ping is unauthenticated with no DB cost; DesktopSummary already runs at most every 5 min). - On error: interval doubles each failure (1s, 2s, 4s, 8s, ...) capped at 5 minutes - On success: resets immediately to normal polling interval - Each communication path tracks its own backoff independently ## Manual testing ### Automated tests (17 total, all pass with -race) \`\`\` go test ./orbit/pkg/backoff/ -v -race -count=1 # 17 tests, 0 failures make lint-go-incremental # 0 issues \`\`\` - 14 logic tests (exponential doubling, cap, jitter, reset, per-path isolation, concurrent access, overflow detection, garbage input flooring) - 3 real-time ticker tests (actual time.Ticker with wall-clock measurements) ### Local TUF end-to-end test (macOS) Set up local TUF server via \`tools/tuf/test/main.sh\` with \`SYSTEMS=macos FLEET_DESKTOP=1 GENERATE_PKG=1\`. This builds orbit and Desktop from this branch, generates \`fleet-osquery.pkg\` with local TUF root keys. Installed the package on macOS, enrolled to a local Fleet server. **Test: corrupt token to simulate #44816 expired-token scenario** Wrote invalid token to \`/opt/orbit/identifier\`, then watched Desktop and orbit logs. Desktop backoff (exponential doubling): \`\`\` 11:57:21 ERR get device URL, backing off next_retry=2.044s (1s * 2^1 + jitter) 11:57:29 ERR get device URL, backing off next_retry=4.061s (1s * 2^2 + jitter) 11:57:39 ERR get device URL, backing off next_retry=8.744s (1s * 2^3 + jitter) \`\`\` Orbit detects and rotates the token: \`\`\` 11:57:42 INF token TTL expired, rotating token \`\`\` Desktop recovers instantly: \`\`\` 11:57:48 DBG enabling tray items \`\`\` Previously Desktop would have retried every 5s indefinitely (#44816). With backoff, retry intervals double each failure and recovery is immediate on the first success. ### Build verification \`\`\` go build ./orbit/cmd/desktop/ # compiles clean go build ./orbit/cmd/orbit/ # compiles clean \`\`\` --- # Checklist for submitter - [x] Changes file added for user-visible changes in \`orbit/changes/\`. - [x] Input data is properly validated, no SQL changes, no JS changes. - [x] Timeouts are implemented and retries are limited to avoid infinite loops (backoff caps at 5 min). - [x] Added/updated automated tests (17 tests, all pass with \`-race\`). - [x] QA'd all new/changed functionality manually (local TUF e2e on macOS). ## fleetd/orbit/Fleet Desktop - [x] If the change applies to only one platform, confirmed that \`runtime.GOOS\` is used as needed to isolate changes (backoff is platform-agnostic). - [x] Verified that fleetd runs on macOS (local TUF install + e2e test). Linux/Windows need QA. - [ ] Verified auto-update works from the released version of component to the new version. |