72c224bd787bb0344cfb5ccbb3a94d87327283bf
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Lucas Manuel Rodriguez
|
001b57cb9d |
Optimize memory usage in CVE chart cron job (#50385)
Resolves #50266. At production numbers the table looks like this - 20,691 CVEs × 83,000 hosts, ~268M raw (cve, host) rows (software + OS joins combined): ``` ┌─────────────────────────┬─────────────────────────┬───────────────────────┐ │ Shape of host IDs │ Old (map[string][]uint) │ New (roaring bitmaps) │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Dense (contiguous runs) │ 2,479 MB │ 4.5 MB │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (random) │ 2,488 MB │ 282 MB │ └─────────────────────────┴─────────────────────────┴───────────────────────┘ ``` A few things worth noting about how these map to your real data: - The old cost is shape-independent: ~2.5 GB retained just for the result map (268M rows × 8 bytes plus append slack), and the peak during collection is higher still because append doubling leaves garbage behind. That's the number that was blowing up the cron. - The new sparse figure is an overstated worst case. Your 268M rows include duplicates — multiple vulnerable software rows per host for the same CVE (the multi-kernel case) and overlap between the software and OS joins. The old code retained every raw row; the bitmap dedupes on Add, so it's bounded by unique pairs, and real fleets with AUTO_INCREMENT host IDs sit much closer to the dense row than the sparse one. - The new representation also has a hard ceiling the old one doesn't: a roaring bitmap over 83k host IDs maxes out around 16 KB per CVE regardless of contents, so even a pathological dataset caps at ~330 MB for all 20,691 CVEs — versus the old form growing linearly with join rows, unbounded. TL;DR: At scale the change is roughly a 550× reduction in the realistic (dense) case, and at minimum ~9× in the theoretical worst case. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance** - Reduced memory usage for CVE chart data collection. - Improved efficiency when processing large CVE and affected-host datasets. - **Bug Fixes** - Preserved correct CVE filtering, duplicate-host handling, disabled-fleet exclusions, and empty-result behavior. - Added coverage for CVEs sourced from both software and operating-system data. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Sharon Katz
|
f492a6a41d |
Enforce API-only endpoint restrictions on chart routes (#49477)
# Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [x] Timeouts are implemented and retries are limited to avoid infinite loops - [x] If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes ## Summary Enforced API-only endpoint restrictions on chart endpoints, matching the pattern already used by the activity bounded context. Also added `RouteTemplateRequestFunc` to chart route server options so the middleware can read the matched mux route template from context. ## Testing - [x] Added/updated automated tests - [x] QA'd all new/changed functionality manually ### Reproduction Created an API-only user with a restrictive endpoint allow-list (only `GET /api/v1/fleet/hosts`). Confirmed that: - Allowed endpoint (`/api/latest/fleet/hosts`) returns 200 - Non-allowed cataloged endpoint (`/api/latest/fleet/users`) returns 403 - Chart endpoint (`/api/latest/fleet/charts/uptime`) returned 200 before the fix (the bug) - After the fix, chart endpoint correctly returns 403 ### Unit test Added a test case in `server/service/middleware/auth/api_only_test.go` that verifies an API-only user with endpoint restrictions is denied access to chart endpoints not in their allow-list. The chart endpoint is included in the test catalog (matching production), so the test exercises the allow-list rejection path. All 17 tests in the auth middleware package pass. ### Local verification 1. Confirmed the chart middleware in `cmd/fleet/serve.go` previously called `auth.AuthenticatedUser(svc, next)` without `APIOnlyEndpointCheck` wrapping 2. Verified the activity bounded context (same file) already uses `auth.APIOnlyEndpointCheck(next)` as the correct pattern 3. Applied the same wrapping to the chart middleware 4. Added `RouteTemplateRequestFunc` to `server/chart/internal/service/endpoint_utils.go` so the route template is available in context (required by `APIOnlyEndpointCheck`) 5. Ran `go test ./server/service/middleware/auth/ -v` with all 17 tests passing 6. Ran `make lint-go-incremental` with 0 issues |
||
|
Scott Gress
|
997a4097c4 | Add docs for chart bounded context (#47877) | ||
|
Scott Gress
|
6336443f37 | Report mobile devices in "hosts online" (#47222) | ||
|
Lucas Manuel Rodriguez
|
705f4db3a3 |
Fix data race in CVE tests (#48070)
Fixes: https://github.com/fleetdm/fleet/actions/runs/28003942578. New run: https://github.com/fleetdm/fleet/actions/runs/28026451929. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved test fixture logic for more reliable test execution. **Note:** This release contains internal testing improvements with no end-user-facing changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
0301aea831 |
Add more filtering to Vulnerability Exposure chart (frontend) (#47674)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44746 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [x] Changing the filters in the UI causes the related API params to be set - [x] Changing the software filters causes the "filtered" tooltip to show up and include info about software filters - [x] Changing the host filters causes the "filtered" tooltip to show up and include info about host filters - [x] Changing both host and software filters causes the "filtered" tooltip to show up and include info about both filters - [x] CVE search works and utilizes infinite scroll <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added software category filtering options to vulnerability charts. * Added EPSS range filtering with validation to refine results. * Added known exploit toggle and CVE exclusion capabilities. * Improved filter status display with tabbed interface. * **Tests** * Added comprehensive test coverage for software filtering and validation logic. * **Style** * Enhanced filter UI styling and interactivity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c370a9672b |
Add CVE chart filtering and non-critical CVE data collection (backend) (#47470)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44746 # Details * Adds the ability to filter historical CVE data by software type, EPSS, CVSS, CVE ID (exclude only) and "has known exploit" * Hard-codes the CVSS filter to 9.0+ for now, since that's the only data that's been collected thus far * Un-gates the collection code so that it will collect CVE data for _all_ severities (but still in the restricted set of software) Related PRs [update the front-end](https://github.com/fleetdm/fleet/pull/47674) to allow sending these filters, and [update GitOps](https://github.com/fleetdm/fleet/pull/47634) to allow changing the default filters. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually ### Manual test plan — CVE chart filtering (backend smoke test) #### Setup - Premium dev server running with a few hosts carrying vulnerable software (so `cve_meta` / `software_cve` / `operating_system_vulnerabilities` are populated) - Chart data present — collector ran once, or seeded: `go run ./tools/charts-backfill --dataset cve --use-tracked-cves --days 7` - API token exported and helper set: ```bash BASE=https://localhost:8080/api/v1/fleet/charts peak() { curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/$1" | jq '[.data[].value] | max'; } #### Checks (compare against the no-filter baseline) - [x] Baseline returns data — GET /charts/cve?days=7 returns a data series; .filters is empty/default - [x] Severity force-pinned to critical — cve?days=7 and cve?days=7&severity_min=0&severity_max=10 give identical peaks (no low-severity leak; client severity ignored) - [x] Category narrowing — software_categories=browsers ≤ baseline; software_categories=os,browsers,office,adobe == baseline - [x] OS category includes kernel — software_categories=os returns OS-vuln + Linux-kernel CVE counts - [x] Known-exploit narrowing — known_exploit=true ≤ baseline - [x] EPSS narrowing — epss_min=0.9 ≤ baseline; epss_min=0&epss_max=1 == baseline (EPSS is 0.0–1.0 on the API) - [x] Exclude is subtractive + tolerant — excluding a visible CVE lowers/keeps counts; exclude_cves=CVE-0000-00000 == baseline (no-op) - [x] Filters echo back — filtered requests return applied values under .filters - [x] Uptime untouched — GET /charts/uptime?days=7 returns its normal series - [x] Free-tier safety (optional) — on non-Premium, /charts/cve returns an empty series, no error - [x] > 0 rows from: SELECT COUNT(DISTINCT scd.entity_id) AS below_critical FROM host_scd_data scd JOIN cve_meta cm ON cm.cve = scd.entity_id WHERE scd.dataset='cve' AND cm.cvss_score < 9.0; - (confirms lower-severity CVEs are stored) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary of changes * **New Features** * Added advanced CVE chart request filters: software categories, known-exploit flag, EPSS min/max, severity min/max, and excluded CVEs. * Expanded CVE chart coverage to use the full “collectible” CVE set, with filtering applied when serving chart data. * **Tests** * Added coverage for collecting collectible CVEs and resolving chart entities based on filter combinations and exclusions. * **Chores** * Updated CVE chart backfill to use collectible CVE discovery. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
6f8942f8da |
Skip unneeded query when getting CVE chart (#45813)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45720 # Details When requesting CVE chart data, we were making a call to get the set of tracked CVEs to filter the data by. Currently we're only _collecting_ data for the tracked CVEs, so there's no reason to make this call at all. When we add more filtering options and start collecting more data, we'll need a call like this again, and will likely need to start caching the results. Otherwise it's a multi-second cost per query on large deployments. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - removed some outdated tests - replaced with a test that checks that when an entity filter returns no items (an empty, rather than nil slice) we get empty buckets returned rather than getting data for all entities. This is a regression test for when we add filtering back. - [X] QA'd all new/changed functionality manually - validated that chart still loads and shows the same data. - tried it on a load test env and saw dramatic API request time improvement <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Optimized CVE chart data retrieval by eliminating redundant queries, reducing unnecessary database operations and improving performance. * Fixed entity ID filtering logic to correctly handle edge cases and prevent unintended filter interactions across metrics. * **Tests** * Added test coverage for chart data queries with empty entity filters. * Improved test isolation to ensure metric-specific filtering behavior is properly separated. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45813?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
d7fa35e417 |
Implement roaring bitmaps for historical data collection (#45709)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45715 # Details This PR refactors the way the charts module stores historical data to use the [roaring bitmap](https://github.com/RoaringBitmap/roaring) package instead of saving raw bitmaps. See [this blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives) to learn how roaring compresses data, but TL;DR for our purposes it represents a huge improvement especially for larger deployments where host ID numbers may be very large. In testing, some data was reduced 96%. The majority of the changes in this PR are straight swapping of types from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and updating the internals of our bit math helpers to use roaring methods instead of native AND and OR methods. I've tried to comment on all functional changes. Since the charts have been shipped already, so there will be data in the wild in the prior "dense" format, the code still handles dense bitmaps on _read_, but will always _write_ roaring bitmaps. The majority of the data will therefore have turned over within 30 days on its own, but I plan on a follow-up PR that will transform open rows when the cron runs so that we should be guaranteed to turn over completely within 30 days. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - Tests updated to accommodate the new format, and existing unchanged tests act as proof against regression - [X] QA'd all new/changed functionality manually - Using a tool that dumps the `host_scd_data` rows data into a JSON file (with the keys being entity_id+data and the values being host IDs on that date), compared the data from main branch and this and confirmed they're identical - With a host count of ~9000, some of which have IDs of over 1,000,000, the data storage requirements were: * 82,558,976 bytes for dense * 2,867,200 for roaring (a 96% decrease) For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - should hugely improve - [X] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented roaring bitmaps in historical data collection to optimize bitmap handling for chart data aggregation * Added encoding support to bitmap storage schema for flexible data representation <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
a29ba6befc | Update data collection interval and strategy (#45293) | ||
|
Scott Gress
|
24e5baf21f |
Only collect data about tracked CVEs (#45247)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45163 # Details Limits CVE data collection to only those CVEs which we report on in the chart. This is a performance optimization necessitated by the large amount of data that bigger fleets may generate. The plan is to implement a data compression strategy so that we can go back to collecting full CVE data soon. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] Ran some collection jobs and verified that only tracked CVEs were represented in "open" rows. - [ ] Ran load test w/ new code For unreleased bug fixes in a release candidate, one of: - [ ] Confirmed that the fix is not expected to adversely impact load test results should improve results! - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Enhancements** * CVE vulnerability tracking is now scoped to a curated set of critical vulnerabilities, improving the relevance of security impact data displayed across your systems. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45247) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
64a50d0c16 |
Fix relative spread + calendar dates on checkerboard (#44959)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44958 # Details Fixes two issues on the checkerboard: 1. Ensures that the chart shows data going back 30 calendar days (not 720 hours) if it has it 2. Leaves `0` values out of the chart color band calculations in "relative" color mode # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually **Before** Colors clustered in top 3 levels, empty boxes in first column: <img width="705" height="416" alt="image" src="https://github.com/user-attachments/assets/b867a1a9-4c52-4b96-92fd-04e7848c6295" /> **After** Colors spread over all levels, no empty boxes in first column: <img width="707" height="412" alt="image" src="https://github.com/user-attachments/assets/c67e11c5-6dd8-4e66-9c32-9d9213ccb24f" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Charts request an extra day to ensure full calendar-day coverage across timezones. * Checkerboard visualization excludes empty/no-data slots from relative color scaling so color ramps reflect non-zero data. * Calendar view trims leading partial days so the displayed window matches the selected range. * **New Features** * Chart date-range selection expanded to support any value from 1–31 days. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
684becade8 |
Allow disabling chart datasets: backend (#44769)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44077 # Details This PR implements enforcement of the "disable dataset" feature. When a dataset is disabled globally, we: * Stop collecting all data for that dataset (the `Collect` method for that dataset is not called in the cron job) * Remove all previously-collected data for the dataset via an asynchronous job When a dataset is disabled for one or more fleets, we: * Provide the list of disabled fleets as an argument to each dataset's `Collect` method. Each dataset is responsible for filtering out hosts in the most efficient way possible * Scrub the data for the relevant datasets using a bitmask, so that all hosts from the disabled fleets are removed from the data. This is done via an asynchronous job. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [X] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [ ] QA'd all new/changed functionality manually ### Prerequisites / Test Setup - [ ] Fleet running with at least 3 teams (call them T1, T2, T3) and ≥3 hosts in each, plus ≥2 hosts with no team - [ ] At least one host on each team has reported recent uptime (within the bucket window) - [ ] At least one host in each team is affected by a tracked CVE (so `host_scd_data` for `dataset='cve'` will have non-empty bitmaps) - [ ] AppConfig: both `features.historical_data.uptime` and `features.historical_data.vulnerabilities` start as `true`; same for every team - [ ] Let the collection cron run at least one full tick to populate baseline rows in `host_scd_data` for both `uptime` and `cve` - [ ] Note the current row count per dataset: `SELECT dataset, COUNT(*) FROM host_scd_data GROUP BY dataset;` --- ### 1. Cron Skips Globally-Disabled Datasets #### 1.1 Global disable of `uptime` - [x] Disable globally: `PATCH /api/v1/fleet/config` with `features.historical_data.uptime = false` - [x] Verify activity feed shows `disabled_historical_dataset` for `uptime` (existing behavior) - [x] Wait for next collection tick (or trigger it via fleetctl debug if available) - [x] Confirm **no new rows** appear for `dataset='uptime'`: `SELECT MAX(valid_from) FROM host_scd_data WHERE dataset='uptime';` should not advance after the disable - [x] Confirm cron still writes `cve` rows on the same tick (per-dataset isolation) - [x] Re-enable: PATCH `historical_data.uptime = true` - [x] Verify next tick resumes writing `uptime` rows #### 1.2 Global disable of `vulnerabilities` - [x] Repeat 1.1 with `features.historical_data.vulnerabilities` - [x] Confirm `cve` writes stop, `uptime` continues #### 1.3 Both disabled globally - [x] Disable both globally - [x] Confirm cron tick produces zero new rows for either dataset - [x] Confirm cron does not error or get stuck - [x] Re-enable both --- ### 2. Per-Fleet Disable — Cron Filters at SQL #### 2.1 Single team disabled for one dataset - [x] Disable uptime for T1 only: PATCH team T1 with `features.historical_data.uptime = false` - [x] Verify scoped `disabled_historical_dataset` activity emitted for T1 - [x] Wait for next cron tick / trigger cron - [x] Pick a host known to be in T1 (call it `H_T1`); confirm its bit is NOT set in any `uptime` row written *after* the disable by filtering the chart to that host - [x] Pick a host in T2 (`H_T2`); confirm its bit IS still set in the same rows (T2 is not disabled) - [x] Pick a no-team host (`H_none`); confirm its bit IS still set (no-team hosts follow the global value) #### 2.2 Same fleet, different dataset - [x] With T1's uptime disabled, confirm T1's hosts ARE still written into `cve` rows on subsequent ticks (per-dataset isolation) #### 2.3 All teams disabled, global on, no-team hosts - [x] Disable uptime on every team (T1, T2, T3) - [x] Confirm next tick still writes a row containing only no-team hosts' bits (global is on, no-team hosts always count) - [x] Re-enable uptime on all teams --- ### 3. Global Scrub — DELETE #### 3.1 Successful global scrub - [x] Note baseline: `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';` (should be > 5000 to exercise the loop; if not, manually insert filler rows or run multiple cron ticks) - [x] Disable uptime globally via the API - [x] Wait for the worker to pick up the scrub / trigger the job - [x] Confirm the count drops to 0: `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';` - [x] Confirm rows for **other datasets** are untouched - [ ] Test again but disable via GitOps --- ### 4. Per-Fleet Scrub — ANDNOT #### 4.1 Single-fleet scrub clears bits - [x] Identify hosts in T1 and record their IDs (call this set `S`) - [x] Pre-disable, confirm at least one `host_scd_data` row for `dataset='uptime'` has bits set at positions in `S` by filtering the chart to those hosts - [x] Disable uptime on T1 only, via the API - [x] Wait for the scrub to run / trigger it - [x] Confirm: every existing row for `dataset='uptime'` now has NO bits set at any position in `S`. Spot-check by filtering the chart to those hosts - [x] Confirm rows for `dataset='cve'` (different dataset) are untouched - [x] Confirm bits for hosts in T2/T3 (not disabled) are still set - [x] Run test again but disable via GitOps #### 4.2 Multi-fleet scrub via GitOps batch - [x] Apply a GitOps spec that flips cve to false on T1 and T3 in a single apply - [x] Wait for scrub(s) to complete - [x] Confirm bits for the union of T1∪T2 hosts are cleared from every row of `dataset='cve'` - [x] Confirm T2 hosts' bits remain set --- ### 5. Activity Feed Cross-Check - [x] Each global flip emits exactly one `disabled_historical_dataset` activity (existing behavior, unchanged) - [x] Each per-team flip emits one scoped activity with the team's ID and name - [x] PATCH submitting unchanged values emits **no** activity and causes **no** scrub (no `host_scd_data` data change observed after the cron tick) - [x] No new "scrub completed" or "scrub started" activity is emitted (out of scope for v1) - [x] Re-enable flips emit `enabled_historical_dataset` activities and do NOT emit any scrub-related activity --- ### 6. Regression Spot Checks - [x] With everything enabled (default), the chart UI renders the same data as before this change (no behavior change in the "all on" case) - [x] AppConfig YAML round-trip (`fleetctl apply`) is benign: applying the unchanged config produces no scrub jobs and no activities - [x] GitOps apply with `historical_data` omitted from team specs defaults to `true` (per the gitops-api change) and does not trigger spurious scrubs - [x] After a full disable+scrub of cve, the `host_scd_data` table has no `dataset='cve'` rows; the chart UI for "vulnerable hosts over time" shows an empty/zero state without errors --- <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Chart collection now supports per-dataset scoping and honors team-level disables; new scrub jobs are registered and worker handlers added. * New dataset scrub operations: global and fleet-scoped scrubs; scrubs can be enqueued and are deduplicated to avoid duplicate pending jobs. Historical-data changes enqueue scrubs after save (errors logged, non-blocking). * **Tests** * Added unit tests for scope resolution, scrub enqueue/dedup behavior, scrub workers, scrub application, and low-level blob scrub logic. * **Documentation** * Added OpenSpec metadata for the chart scrub change. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
5e7f5a7584 |
Optimize data collection: add index and batch deletes (#44692)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44609 # Details This PR optimizes the historical data collection system in two ways: 1. Adds an additional index on the `host_scd_data` table allowing more efficient lookups of rows by their `valid_to`, to optimize both closing out open rows and deleting old rows 2. Implements batching in the job that deletes old rows, so that it no longer blocks writes if the collection job happens to happen at the same time as the cleanup job # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [ ] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [ ] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually SQL explains -- before: ``` +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | 1 | DELETE | host_scd_data | NULL | ALL | NULL | NULL | NULL | NULL | 144320 | 100.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ | 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range | uniq_entity_bucket | 604 | const,const | 3030 | 100.00 | Using where | +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ ``` Using a test set of data (~144k "open" rows), UPDATES happened at 9 ops per second. after: ``` +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ | 1 | DELETE | host_scd_data | NULL | range | idx_valid_to_dataset | idx_valid_to_dataset | 5 | const | 55749 | 100.00 | Using where | +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ | 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range,idx_valid_to_dataset | idx_valid_to_dataset | 609 | const,const,const | 4 | 100.00 | Using where; Using temporary | +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ ``` Using the same test set of data, UPDATES happened at 4,910 ops per second. For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results this should significantly improve results! - [ ] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [ ] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [ ] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Cleanup now runs in controlled, ordered batches, removing only closed/historical records while respecting cancellation; error reporting for cleanup was strengthened. * Added a new composite index on historical data to improve cleanup and query performance. * **Tests** * Added tests and test helpers validating batched cleanup behavior, preservation of open records, multi-batch operation, and cancellation handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
4334017b38 |
Add Vulnerabilities exposure dataset (#44124)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #43769 # Details Adds methods to collect data for the `cve` dataset. As with all sets this is collected at hourly granularity, but unlike the `uptime` set, the `cve` set uses the "snapshot" strategy so that we record at most one change (the most recent) per hour. For this first iteration, we are _recording_ data for all CVEs (i.e., which hosts were exposed to which CVEs at a given time), but we are only _reporting_ a subset of CVEs for the dashboard chart. See [this comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405) for more info. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers` and didn't find any outside of the expected - [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261), generated chart: <img width="706" height="421" alt="image" src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Host vulnerability (CVE) chart added to the dashboard; CVE chart data collection is now active. * Critical CVE tracking surfaces high-severity vulnerabilities. * **Improvements** * CVE chart refreshes every 3 hours (was daily) for more timely insights. * Snapshot collection reconciles and closes prior data during empty runs to keep charts accurate. * CVE queries may produce zero datapoints when no tracked CVEs exist, without affecting other metrics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
28908e6083 |
Dashboard charts backend (#43910)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #42812 # Details This PR implements a new bounded context, `chart`, with a single endpoint `/charts`. The context encompasses a framework for recording and querying and aggregating historical data for Fleet hosts, and returning that data via the API for the purpose of charting. This initial iteration has a full implementation of a dataset called "uptime" which captures which hosts were online hour-by-hour (online meaning, having been "seen" at some point during that hour). It has a partial implementation of a "cve" dataset which will capture which hosts were vulnerable to which CVEs during a given day. ### Data storage Data is stored in an SCD (slowly-changing dimension) format in the `host_scd_data` table, where the main "value" in a row is stored in the `host_bitmap` column, which is a `mediumblob` where each bit encodes a host ID (bit one represents host ID 1, bit 1444 represents host ID 1444, etc.). The set of bits set on a row represents that hosts for which that dataset is "on" during a given time period represented by the `valid_from` (inclusive) and `valid_to` (exclusive) dates, where a `valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000 meaning that the row is still "open" (the value represents everything from `valid_from` to the present). Additionally an `entity_id` column can be used for datasets with multiple dimensions, e.g. CVE exposure or software usage which would have entity IDs representing CVEs or software items respectively. ### Data collection Data is collected via a cron job that runs every 10 minutes. Each dataset has its own `Collect` method which will sample the data for the given moment. For example the "uptime" dataset gathers the set of hosts that are online at the moment, and the "cve" dataset will gather the set of hosts that are vulnerable to each CVE at that moment. The sample can then be recorded using one of two strategies: * `accumulate`: bitwise OR the sample with any data already recorded for the current hour, or add a new pre-closed row for that hour. * `snapshot`: if there is no open row, create one with the sample and `valid_to set` to the sentinel. Otherwise: * If the sample has the same value as the current open row, do nothing * If the sample has a different value and the current open row's `valid_from` is within the same hour, update the current row's value * If the sample has a different value and the current open row's `valid_from` is not within the same hour, close the current open row and start a new one with `valid_from` = the start of the current hour ### Data retrieval 1. Gets the set of host IDs to retrieve data for. This starts with the set of host IDs in the requested fleet (or all the hosts a user has access to if no `fleet_id` param was passed to the `/charts` endpoint), and further whittled down by any filter options supplied with the request (labels, platforms, etc.). 2. Finds all `host_scd_data` rows for the requested dataset and date range (i.e. all rows whose `valid_from` is < the date range end and `valid_to` is > the date range start). 3. Calculates the date ranges of the "buckets" to return datapoints for. For the uptime chart we default to 3-hour buckets, so we want 8 buckets per day. 4. Iterates over each bucket and finds the row or rows from host_scd_data that cover that bucket range. For datasets using the "accumulate" strategy, the values for those rows are ORed together. For "snapshot"s, we take the one active at the bucket end time to represent the bucket (e.g. "which hosts had a given CVE at the end of the day") ### Tools This PR includes two dev tools that don't require deep review: * **chart-backfill** - used to backfill data to various datasets for testing * **charts-collect** - used to collect data from a live server via the API and put into a local hosts_scd_data table # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - With [front-end branch](https://github.com/fleetdm/fleet/pull/43878) <img width="712" height="434" alt="image" src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c" /> ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added charting bounded context: HTTP API for metrics (uptime, CVE), dataset registry, hosted dataset collection, background collection/cleanup with opt-out env. * New utilities: host bitmap operations and string-list/uint-list parsers. * New CLI tools to collect and backfill chart data. * **Database** * Migration and schema to store host time-series SCD chart data. * **Tests** * Extensive unit and integration tests for service, storage, caching, cron, and utilities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
