Discharge7443/fleet
1
0
Fork 0
Code Issues Pull Requests Actions 19 Packages Projects Releases Wiki Activity
26,791 Commits 2 Branches 0 Tags
imgbot
Commit Graph
5 Commits
Author SHA1 Message Date
Scott Gress 6336443f37 Report mobile devices in "hosts online" (#47222) 2026-06-24 07:50:35 -07:00
Scott Gress c370a9672b Add CVE chart filtering and non-critical CVE data collection (backend) (#47470) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #44746 

# Details

* Adds the ability to filter historical CVE data by software type, EPSS,
CVSS, CVE ID (exclude only) and "has known exploit"
* Hard-codes the CVSS filter to 9.0+ for now, since that's the only data
that's been collected thus far
* Un-gates the collection code so that it will collect CVE data for
_all_ severities (but still in the restricted set of software)

Related PRs [update the
front-end](https://github.com/fleetdm/fleet/pull/47674) to allow sending
these filters, and [update
GitOps](https://github.com/fleetdm/fleet/pull/47634) to allow changing
the default filters.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually

### Manual test plan — CVE chart filtering (backend smoke test)

#### Setup
- Premium dev server running with a few hosts carrying vulnerable
software (so `cve_meta` / `software_cve` /
  `operating_system_vulnerabilities` are populated)
- Chart data present — collector ran once, or seeded: `go run
./tools/charts-backfill --dataset cve
  --use-tracked-cves --days 7`
  - API token exported and helper set:
    ```bash
    BASE=https://localhost:8080/api/v1/fleet/charts
peak() { curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/$1" | jq
'[.data[].value] | max'; }

####  Checks (compare against the no-filter baseline)

- [x] Baseline returns data — GET /charts/cve?days=7 returns a data
series; .filters is empty/default
- [x] Severity force-pinned to critical — cve?days=7 and
cve?days=7&severity_min=0&severity_max=10 give identical
  peaks (no low-severity leak; client severity ignored)
- [x] Category narrowing — software_categories=browsers ≤ baseline;
software_categories=os,browsers,office,adobe ==
  baseline
- [x] OS category includes kernel — software_categories=os returns
OS-vuln + Linux-kernel CVE counts
  - [x] Known-exploit narrowing — known_exploit=true ≤ baseline
- [x] EPSS narrowing — epss_min=0.9 ≤ baseline; epss_min=0&epss_max=1 ==
baseline (EPSS is 0.0–1.0 on the API)
- [x] Exclude is subtractive + tolerant — excluding a visible CVE
lowers/keeps counts; exclude_cves=CVE-0000-00000
  == baseline (no-op)
- [x] Filters echo back — filtered requests return applied values under
.filters
- [x] Uptime untouched — GET /charts/uptime?days=7 returns its normal
series
- [x] Free-tier safety (optional) — on non-Premium, /charts/cve returns
an empty series, no error
  - [x] > 0 rows from:
  SELECT COUNT(DISTINCT scd.entity_id) AS below_critical
  FROM host_scd_data scd JOIN cve_meta cm ON cm.cve = scd.entity_id
  WHERE scd.dataset='cve' AND cm.cvss_score < 9.0;
  - (confirms lower-severity CVEs are stored)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary of changes

* **New Features**
* Added advanced CVE chart request filters: software categories,
known-exploit flag, EPSS min/max, severity min/max, and excluded CVEs.
* Expanded CVE chart coverage to use the full “collectible” CVE set,
with filtering applied when serving chart data.

* **Tests**
* Added coverage for collecting collectible CVEs and resolving chart
entities based on filter combinations and exclusions.

* **Chores**
  * Updated CVE chart backfill to use collectible CVE discovery.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-19 10:52:03 -05:00
Scott Gress d7fa35e417 Implement roaring bitmaps for historical data collection (#45709) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #45715 

# Details

This PR refactors the way the charts module stores historical data to
use the [roaring bitmap](https://github.com/RoaringBitmap/roaring)
package instead of saving raw bitmaps. See [this
blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives)
to learn how roaring compresses data, but TL;DR for our purposes it
represents a huge improvement especially for larger deployments where
host ID numbers may be very large. In testing, some data was reduced
96%.

The majority of the changes in this PR are straight swapping of types
from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and
updating the internals of our bit math helpers to use roaring methods
instead of native AND and OR methods. I've tried to comment on all
functional changes.

Since the charts have been shipped already, so there will be data in the
wild in the prior "dense" format, the code still handles dense bitmaps
on _read_, but will always _write_ roaring bitmaps. The majority of the
data will therefore have turned over within 30 days on its own, but I
plan on a follow-up PR that will transform open rows when the cron runs
so that we should be guaranteed to turn over completely within 30 days.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- Tests updated to accommodate the new format, and existing unchanged
tests act as proof against regression
- [X] QA'd all new/changed functionality manually
- Using a tool that dumps the `host_scd_data` rows data into a JSON file
(with the keys being entity_id+data and the values being host IDs on
that date), compared the data from main branch and this and confirmed
they're identical
- With a host count of ~9000, some of which have IDs of over 1,000,000,
the data storage requirements were:
     * 82,558,976 bytes for dense
     * 2,867,200 for roaring (a 96% decrease)

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
  - should hugely improve
- [X] Alerted the release DRI if additional load testing is needed

## Database migrations

- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Implemented roaring bitmaps in historical data collection to optimize
bitmap handling for chart data aggregation
* Added encoding support to bitmap storage schema for flexible data
representation

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-19 09:34:29 -05:00
Scott Gress 684becade8 Allow disabling chart datasets: backend (#44769) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #44077 

# Details

This PR implements enforcement of the "disable dataset" feature.  

When a dataset is disabled globally, we:

* Stop collecting all data for that dataset (the `Collect` method for
that dataset is not called in the cron job)
* Remove all previously-collected data for the dataset via an
asynchronous job

When a dataset is disabled for one or more fleets, we:

* Provide the list of disabled fleets as an argument to each dataset's
`Collect` method. Each dataset is responsible for filtering out hosts in
the most efficient way possible
* Scrub the data for the relevant datasets using a bitmask, so that all
hosts from the disabled fleets are removed from the data. This is done
via an asynchronous job.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, unreleased

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [X] Timeouts are implemented and retries are limited to avoid infinite
loops

## Testing

- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [ ] QA'd all new/changed functionality manually

  ### Prerequisites / Test Setup

- [ ] Fleet running with at least 3 teams (call them T1, T2, T3) and ≥3
hosts in each, plus ≥2 hosts with no team
- [ ] At least one host on each team has reported recent uptime (within
the bucket window)
- [ ] At least one host in each team is affected by a tracked CVE (so
`host_scd_data` for `dataset='cve'` will have non-empty bitmaps)
- [ ] AppConfig: both `features.historical_data.uptime` and
`features.historical_data.vulnerabilities` start as `true`; same for
every team
- [ ] Let the collection cron run at least one full tick to populate
baseline rows in `host_scd_data` for both `uptime` and `cve`
- [ ] Note the current row count per dataset: `SELECT dataset, COUNT(*)
FROM host_scd_data GROUP BY dataset;`

  ---

  ### 1. Cron Skips Globally-Disabled Datasets

  #### 1.1 Global disable of `uptime`

- [x] Disable globally: `PATCH /api/v1/fleet/config` with
`features.historical_data.uptime = false`
- [x] Verify activity feed shows `disabled_historical_dataset` for
`uptime` (existing behavior)
- [x] Wait for next collection tick (or trigger it via fleetctl debug if
available)
  - [x] Confirm **no new rows** appear for `dataset='uptime'`:
`SELECT MAX(valid_from) FROM host_scd_data WHERE dataset='uptime';`
        should not advance after the disable
- [x] Confirm cron still writes `cve` rows on the same tick (per-dataset
isolation)
  - [x] Re-enable: PATCH `historical_data.uptime = true`
  - [x] Verify next tick resumes writing `uptime` rows

  #### 1.2 Global disable of `vulnerabilities`

  - [x] Repeat 1.1 with `features.historical_data.vulnerabilities`
  - [x] Confirm `cve` writes stop, `uptime` continues

  #### 1.3 Both disabled globally

  - [x] Disable both globally
  - [x] Confirm cron tick produces zero new rows for either dataset
  - [x] Confirm cron does not error or get stuck
  - [x] Re-enable both

  ---

  ### 2. Per-Fleet Disable — Cron Filters at SQL

  #### 2.1 Single team disabled for one dataset

  - [x] Disable uptime for T1 only: PATCH team T1 with
        `features.historical_data.uptime = false`
- [x] Verify scoped `disabled_historical_dataset` activity emitted for
T1
  - [x] Wait for next cron tick / trigger cron
- [x] Pick a host known to be in T1 (call it `H_T1`); confirm its bit is
NOT set in any `uptime` row written *after* the disable by filtering the
chart to that host
- [x] Pick a host in T2 (`H_T2`); confirm its bit IS still set in the
same rows (T2 is not disabled)
- [x] Pick a no-team host (`H_none`); confirm its bit IS still set
(no-team hosts follow the global value)

  #### 2.2 Same fleet, different dataset

- [x] With T1's uptime disabled, confirm T1's hosts ARE still written
into `cve` rows on subsequent ticks (per-dataset isolation)

  #### 2.3 All teams disabled, global on, no-team hosts

  - [x] Disable uptime on every team (T1, T2, T3)
- [x] Confirm next tick still writes a row containing only no-team
hosts' bits (global is on, no-team hosts always count)
  - [x] Re-enable uptime on all teams

  ---

  ### 3. Global Scrub — DELETE

  #### 3.1 Successful global scrub

  - [x] Note baseline:
        `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';`
        (should be > 5000 to exercise the loop; if not, manually
        insert filler rows or run multiple cron ticks)
  - [x] Disable uptime globally via the API
  - [x] Wait for the worker to pick up the scrub / trigger the job
  - [x] Confirm the count drops to 0:
        `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';`
  - [x] Confirm rows for **other datasets** are untouched
  - [ ] Test again but disable via GitOps

  ---

  ### 4. Per-Fleet Scrub — ANDNOT

  #### 4.1 Single-fleet scrub clears bits

  - [x] Identify hosts in T1 and record their IDs (call this set `S`)
- [x] Pre-disable, confirm at least one `host_scd_data` row for
`dataset='uptime'` has bits set at positions in `S` by filtering the
chart to those hosts
  - [x] Disable uptime on T1 only, via the API
  - [x] Wait for the scrub to run / trigger it
- [x] Confirm: every existing row for `dataset='uptime'` now has NO bits
set at any position in `S`. Spot-check by filtering the chart to those
hosts
- [x] Confirm rows for `dataset='cve'` (different dataset) are untouched
  - [x] Confirm bits for hosts in T2/T3 (not disabled) are still set
  - [x] Run test again but disable via GitOps

  #### 4.2 Multi-fleet scrub via GitOps batch

- [x] Apply a GitOps spec that flips cve to false on T1 and T3 in a
single apply
  - [x] Wait for scrub(s) to complete
- [x] Confirm bits for the union of T1∪T2 hosts are cleared from every
row of `dataset='cve'`
  - [x] Confirm T2 hosts' bits remain set

  ---

  ### 5. Activity Feed Cross-Check

  - [x] Each global flip emits exactly one `disabled_historical_dataset`
        activity (existing behavior, unchanged)
  - [x] Each per-team flip emits one scoped activity with the team's
        ID and name
  - [x] PATCH submitting unchanged values emits **no** activity and
        causes **no** scrub (no `host_scd_data` data change observed
        after the cron tick)
  - [x] No new "scrub completed" or "scrub started" activity is
        emitted (out of scope for v1)
  - [x] Re-enable flips emit `enabled_historical_dataset` activities
        and do NOT emit any scrub-related activity

  ---

  ### 6. Regression Spot Checks

  - [x] With everything enabled (default), the chart UI renders the
        same data as before this change (no behavior change in the
        "all on" case)
  - [x] AppConfig YAML round-trip (`fleetctl apply`) is benign:
        applying the unchanged config produces no scrub jobs and no
        activities
  - [x] GitOps apply with `historical_data` omitted from team specs
        defaults to `true` (per the gitops-api change) and does not
        trigger spurious scrubs
  - [x] After a full disable+scrub of cve, the `host_scd_data` table
        has no `dataset='cve'` rows; the chart UI for "vulnerable
        hosts over time" shows an empty/zero state without errors

  ---


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Chart collection now supports per-dataset scoping and honors
team-level disables; new scrub jobs are registered and worker handlers
added.
* New dataset scrub operations: global and fleet-scoped scrubs; scrubs
can be enqueued and are deduplicated to avoid duplicate pending jobs.
Historical-data changes enqueue scrubs after save (errors logged,
non-blocking).
* **Tests**
* Added unit tests for scope resolution, scrub enqueue/dedup behavior,
scrub workers, scrub application, and low-level blob scrub logic.
* **Documentation**
  * Added OpenSpec metadata for the chart scrub change.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-07 08:52:35 -05:00
Scott Gress 5e7f5a7584 Optimize data collection: add index and batch deletes (#44692) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #44609

# Details

This PR optimizes the historical data collection system in two ways:

1. Adds an additional index on the `host_scd_data` table allowing more
efficient lookups of rows by their `valid_to`, to optimize both closing
out open rows and deleting old rows
2. Implements batching in the job that deletes old rows, so that it no
longer blocks writes if the collection job happens to happen at the same
time as the cleanup job

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a, unreleased
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops

## Testing

- [ ] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [X] QA'd all new/changed functionality manually

SQL explains -- before:

```
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table         | partitions | type | possible_keys | key  | key_len | ref  | rows   | filtered | Extra       |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+
|  1 | DELETE      | host_scd_data | NULL       | ALL  | NULL          | NULL | NULL    | NULL | 144320 |   100.00 | Using where |
+----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+

+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
| id | select_type | table         | partitions | type  | possible_keys                        | key                | key_len | ref         | rows | filtered | Extra       |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
|  1 | UPDATE      | host_scd_data | NULL       | range | uniq_entity_bucket,idx_dataset_range | uniq_entity_bucket | 604     | const,const | 3030 |   100.00 | Using where |
+----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+
```

Using a test set of data (~144k "open" rows), UPDATES happened at 9 ops
per second.

after:

```
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
| id | select_type | table         | partitions | type  | possible_keys        | key                  | key_len | ref   | rows  | filtered | Extra       |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+
|  1 | DELETE      | host_scd_data | NULL       | range | idx_valid_to_dataset | idx_valid_to_dataset | 5       | const | 55749 |   100.00 | Using where |
+----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+

+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
| id | select_type | table         | partitions | type  | possible_keys                                             | key                  | key_len | ref               | rows | filtered | Extra                        |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
|  1 | UPDATE      | host_scd_data | NULL       | range | uniq_entity_bucket,idx_dataset_range,idx_valid_to_dataset | idx_valid_to_dataset | 609     | const,const,const |    4 |   100.00 | Using where; Using temporary |
+----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+
```

Using the same test set of data, UPDATES happened at 4,910 ops per
second.

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
this should significantly improve results!
- [ ] Alerted the release DRI if additional load testing is needed

## Database migrations

- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Cleanup now runs in controlled, ordered batches, removing only
closed/historical records while respecting cancellation; error reporting
for cleanup was strengthened.
* Added a new composite index on historical data to improve cleanup and
query performance.
* **Tests**
* Added tests and test helpers validating batched cleanup behavior,
preservation of open records, multi-batch operation, and cancellation
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-05 08:29:47 -05:00