Sharon Katz
18d3481604
**Related issue:** Resolves #45320 # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Summary `filterExtensionsForHost` (called on every Orbit config fetch, ~30s per host) had an N+1 query pattern: it called `HostMemberOfAllLabels` once per extension in a loop, issuing a separate DB query for each. This PR replaces the N queries with a single batch query via a new `HostMembershipForLabels` datastore method that returns which labels (from a given list) the host belongs to. Extension filtering then happens in-memory. ### Changes - **New datastore method** `HostMembershipForLabels(ctx, hostID, labelNames) -> map[string]bool` -- single `SELECT l.name FROM labels l JOIN label_membership` query - **Updated `filterExtensionsForHost`** in `server/service/orbit.go` -- collects all unique label names across extensions, calls the new method once, filters in-memory - **No API, UI, CLI, agent, or schema changes** -- purely server-side internal optimization. Full backward compatibility: old agents work with new servers and vice versa (no protocol change). ## Benchmark Ran a local end-to-end benchmark against the live `POST /api/fleet/orbit/config` endpoint to measure the real-world impact. **Setup:** - MacBook (Fleet server + Docker MySQL 8.0 + Redis, all localhost) - 50 enrolled Orbit hosts (darwin), 5 label-scoped extensions, all hosts members of all 5 labels - 500 requests at concurrency 10, cycling through all 50 orbit_node_keys - Built Fleet binary from `main` (before) and this PR branch (after), same database and test data **Results:** | Metric | Before (main) | After (this PR) | Improvement | |--------|:---:|:---:|:---:| | Avg latency | 25.33 ms | 17.46 ms | **-31%, 1.45x faster** | | P50 latency | 24.55 ms | 16.52 ms | **-33%, 1.49x faster** | | P95 latency | 34.42 ms | 27.53 ms | **-20%, 1.25x faster** | | Throughput | 390.6 req/s | 564.2 req/s | **+44%** | ### Extrapolation to 100,000 hosts At 100k hosts with a 30-second check-in interval (3,333 req/s steady state): | Metric | Before | After | |--------|--------|-------| | Server host capacity (measured MacBook) | 11,718 | 16,926 (+44%) | | Label-check DB queries/sec | **16,665** (5/req) | **3,333** (1/req) | | **DB queries eliminated** | | **13,332/sec (80% reduction)** | The improvement scales linearly with extension count: | Extensions | DB queries eliminated/sec | Reduction | |:---:|---:|:---:| | 5 | 13,332 | 80% | | 10 | 29,997 | 90% | | 15 | 46,662 | 93% | | 20 | 63,327 | 95% | > **Note:** These are conservative localhost numbers. In production, where each DB round-trip includes real network latency, the per-request latency improvement would be more pronounced because each eliminated query saves a network hop. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) ### Automated - New `testHostMembershipForLabels` MySQL integration test covering: empty input, full membership, partial membership, nonexistent labels, nonexistent host, host with no memberships - Existing `testHostMemberOfAllLabels` unchanged and unaffected ### Manual QA 1. Fleet Premium instance with 2+ Orbit-enrolled hosts 2. Configure 3+ osquery extensions with different label scoping 3. Verify each host receives only the extensions whose label requirements it meets 4. Verify extensions with no label scoping are included for all hosts <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Improved Orbit configuration loading by batching host label membership checks into a single query for extension label filtering. * **Behavior** * Extension availability and filtering behavior remains the same, with more efficient processing when multiple extensions use labels. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 lines
124 B
Plaintext
2 lines
124 B
Plaintext
- Improved performance of Orbit config endpoint by batching extension label-membership checks into a single database query.
|