Sharon Katz
4221903eb2
Closes #45624 Part 1 of #45553 -- see there for the full behavioral contract and Oracle. ## Changes - New `orbit/pkg/backoff` package: shared, stateful exponential backoff tracker with jitter, thread-safe, per-path isolation. This package will serve all agent components that need backoff (orbit API, fleetd paths, and potentially osquery TLS), but for now only Fleet Desktop uses it. We are introducing it incrementally to reduce risk. - Integrated into Fleet Desktop's `checkToken` retry loop -- the exact tight-retry path that caused the #44816 DB outage. The main ping/DesktopSummary loop does not need backoff (Ping is unauthenticated with no DB cost; DesktopSummary already runs at most every 5 min). - On error: interval doubles each failure (1s, 2s, 4s, 8s, ...) capped at 5 minutes - On success: resets immediately to normal polling interval - Each communication path tracks its own backoff independently ## Manual testing ### Automated tests (17 total, all pass with -race) \`\`\` go test ./orbit/pkg/backoff/ -v -race -count=1 # 17 tests, 0 failures make lint-go-incremental # 0 issues \`\`\` - 14 logic tests (exponential doubling, cap, jitter, reset, per-path isolation, concurrent access, overflow detection, garbage input flooring) - 3 real-time ticker tests (actual time.Ticker with wall-clock measurements) ### Local TUF end-to-end test (macOS) Set up local TUF server via \`tools/tuf/test/main.sh\` with \`SYSTEMS=macos FLEET_DESKTOP=1 GENERATE_PKG=1\`. This builds orbit and Desktop from this branch, generates \`fleet-osquery.pkg\` with local TUF root keys. Installed the package on macOS, enrolled to a local Fleet server. **Test: corrupt token to simulate #44816 expired-token scenario** Wrote invalid token to \`/opt/orbit/identifier\`, then watched Desktop and orbit logs. Desktop backoff (exponential doubling): \`\`\` 11:57:21 ERR get device URL, backing off next_retry=2.044s (1s * 2^1 + jitter) 11:57:29 ERR get device URL, backing off next_retry=4.061s (1s * 2^2 + jitter) 11:57:39 ERR get device URL, backing off next_retry=8.744s (1s * 2^3 + jitter) \`\`\` Orbit detects and rotates the token: \`\`\` 11:57:42 INF token TTL expired, rotating token \`\`\` Desktop recovers instantly: \`\`\` 11:57:48 DBG enabling tray items \`\`\` Previously Desktop would have retried every 5s indefinitely (#44816). With backoff, retry intervals double each failure and recovery is immediate on the first success. ### Build verification \`\`\` go build ./orbit/cmd/desktop/ # compiles clean go build ./orbit/cmd/orbit/ # compiles clean \`\`\` --- # Checklist for submitter - [x] Changes file added for user-visible changes in \`orbit/changes/\`. - [x] Input data is properly validated, no SQL changes, no JS changes. - [x] Timeouts are implemented and retries are limited to avoid infinite loops (backoff caps at 5 min). - [x] Added/updated automated tests (17 tests, all pass with \`-race\`). - [x] QA'd all new/changed functionality manually (local TUF e2e on macOS). ## fleetd/orbit/Fleet Desktop - [x] If the change applies to only one platform, confirmed that \`runtime.GOOS\` is used as needed to isolate changes (backoff is platform-agnostic). - [x] Verified that fleetd runs on macOS (local TUF install + e2e test). Linux/Windows need QA. - [ ] Verified auto-update works from the released version of component to the new version.
Orbit is a lightweight osquery installer and autoupdater. With Orbit, it's easy to deploy osquery, manage configurations, and keep things up-to-date. Orbit eases the deployment of osquery connected with a Fleet server, and is a (near) drop-in replacement for osquery in a variety of deployment scenarios.
Orbit is the recommended agent for Fleet. But Orbit can be used with or without Fleet, and Fleet can be used with or without Orbit.
How to build from source
To build orbit we use goreleaser.
For reference, here are the build configuration files:
- Goreleaser github workflow
- Goreleaser configuration file for each platform:
Following are the commands to build in case you can't use goreleaser.
IMPORTANT: We recommend you build orbit natively and not cross compile to avoid any build or runtime errors.
macOS
CGO_ENABLED=1 \
CODESIGN_IDENTITY=$CODESIGN_IDENTITY \
ORBIT_VERSION=$VERSION \
ORBIT_BINARY_PATH=./orbit-macos \
go run ./orbit/tools/build/build.go
Windows
CGO_ENABLED=0 \
GOOS=windows \
GOARCH=amd64 \
go build \
-trimpath \
-ldflags="-s -w -X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=$VERSION \
-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Commit=$COMMIT \
-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Date=$DATE" \
-o ./orbit.exe ./orbit/cmd/orbit
Linux
CGO_ENABLED=1 \
GOOS=linux \
GOARCH=amd64 \
go build \
-trimpath \
-ldflags="-s -w -X github.com/fleetdm/fleet/v4/orbit/pkg/build.Version=$VERSION \
-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Commit=$COMMIT \
-X github.com/fleetdm/fleet/v4/orbit/pkg/build.Date=$DATE" \
-o ./orbit-linux ./orbit/cmd/orbit
Bugs
To report a bug or request a feature, create an issue in the fleet GitHub repository.
Orbit Development
Run Orbit From Source
To execute orbit from source, use the following commands:
Connect to a Fleet server
Modify the fleet-url and enroll-secret as appropriate:
go run github.com/fleetdm/fleet/v4/orbit/cmd/orbit \
--dev-mode \
--disable-updates \
--root-dir /tmp/orbit \
--fleet-url https://localhost:8080 \
--insecure \
--enroll-secret Pz3zC0NMDdZfb3FtqiLgwoexItojrYh/ \
-- --verbose
Using a custom flagfile
With a flagfile.txt for osqueryd:
go run github.com/fleetdm/fleet/v4/orbit/cmd/orbit \
--dev-mode \
--disable-updates \
--root-dir /tmp/orbit \
-- --flagfile=flagfile.txt --verbose
Open an interactive shell to run SQL
This can be useful for building/testing extension tables:
go run github.com/fleetdm/fleet/v4/orbit/cmd/orbit \
--dev-mode \
--disable-updates \
--root-dir /tmp/orbit \
shell
Generate Installer Packages from Orbit Source
The fleetctl package command generates installers by fetching the targets/executables from a TUF repository.
To generate an installer that contains an Orbit built from source you need to setup a local TUF repository.
The following document explains how you can generate a TUF repository, and installers that use it tools/tuf/test.
FAQs
How does Orbit compare with Kolide Launcher?
Orbit is inspired by the success of Kolide Launcher, and approaches a similar problem domain with new strategies informed by the challenges encountered in real world deployments. Orbit does not share any code with Launcher.
- Both Orbit and Launcher use The Update Framework specification for managing updates. Orbit utilizes the official go-tuf library, while Launcher has it's own implementation of the specification.
- Orbit can be deployed as a (near) drop-in replacement for osquery, supporting full customization of the osquery flags. Launcher heavily manages the osquery flags making deployment outside of Fleet or Kolide's SaaS difficult.
- Orbit prefers the battle-tested plugins of osquery. Orbit uses the built-in logging, configuration, and live report plugins, while Launcher uses custom implementations.
- Orbit prefers the built-in osquery remote APIs. Launcher utilizes a custom gRPC API that has led to issues with character encoding, load balancers/proxies, and request size limits.
- Orbit encourages use of the osquery performance Watchdog, while Launcher disables the Watchdog.
Additionally, Orbit aims to tackle problems out of scope for Launcher:
- Configure updates via release channels, providing more granular control over agent versioning.
- Manage osquery startup flags from a remote (Fleet) server.
- Support for deploying and updating osquery extensions.
- Manage osquery versions from a remote (Fleet) server.
Is Orbit Free?
Yes! Orbit is licensed under an MIT license and all uses are encouraged.
How does orbit update osquery? And how do the stable and edge channels get triggered to update osquery on a self hosted Fleet instance?
Orbit uses a configurable update server. We expect that many folks will just use the update server we manage (similar to what Kolide does with Launcher's update server). We are also offering tooling for self-managing an update server as part of Fleet Premium (the subscription offering).
Community
Chat
Please join us in the #fleet channel on osquery Slack.
