Discharge7443/fleet
1
0
Fork 0
Code Issues Pull Requests Actions 24 Packages Projects Releases Wiki Activity
26,790 Commits 2 Branches 0 Tags
main
Commit Graph
3 Commits
Author SHA1 Message Date
Scott Gress c370a9672b Add CVE chart filtering and non-critical CVE data collection (backend) (#47470) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #44746 

# Details

* Adds the ability to filter historical CVE data by software type, EPSS,
CVSS, CVE ID (exclude only) and "has known exploit"
* Hard-codes the CVSS filter to 9.0+ for now, since that's the only data
that's been collected thus far
* Un-gates the collection code so that it will collect CVE data for
_all_ severities (but still in the restricted set of software)

Related PRs [update the
front-end](https://github.com/fleetdm/fleet/pull/47674) to allow sending
these filters, and [update
GitOps](https://github.com/fleetdm/fleet/pull/47634) to allow changing
the default filters.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually

### Manual test plan — CVE chart filtering (backend smoke test)

#### Setup
- Premium dev server running with a few hosts carrying vulnerable
software (so `cve_meta` / `software_cve` /
  `operating_system_vulnerabilities` are populated)
- Chart data present — collector ran once, or seeded: `go run
./tools/charts-backfill --dataset cve
  --use-tracked-cves --days 7`
  - API token exported and helper set:
    ```bash
    BASE=https://localhost:8080/api/v1/fleet/charts
peak() { curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/$1" | jq
'[.data[].value] | max'; }

####  Checks (compare against the no-filter baseline)

- [x] Baseline returns data — GET /charts/cve?days=7 returns a data
series; .filters is empty/default
- [x] Severity force-pinned to critical — cve?days=7 and
cve?days=7&severity_min=0&severity_max=10 give identical
  peaks (no low-severity leak; client severity ignored)
- [x] Category narrowing — software_categories=browsers ≤ baseline;
software_categories=os,browsers,office,adobe ==
  baseline
- [x] OS category includes kernel — software_categories=os returns
OS-vuln + Linux-kernel CVE counts
  - [x] Known-exploit narrowing — known_exploit=true ≤ baseline
- [x] EPSS narrowing — epss_min=0.9 ≤ baseline; epss_min=0&epss_max=1 ==
baseline (EPSS is 0.0–1.0 on the API)
- [x] Exclude is subtractive + tolerant — excluding a visible CVE
lowers/keeps counts; exclude_cves=CVE-0000-00000
  == baseline (no-op)
- [x] Filters echo back — filtered requests return applied values under
.filters
- [x] Uptime untouched — GET /charts/uptime?days=7 returns its normal
series
- [x] Free-tier safety (optional) — on non-Premium, /charts/cve returns
an empty series, no error
  - [x] > 0 rows from:
  SELECT COUNT(DISTINCT scd.entity_id) AS below_critical
  FROM host_scd_data scd JOIN cve_meta cm ON cm.cve = scd.entity_id
  WHERE scd.dataset='cve' AND cm.cvss_score < 9.0;
  - (confirms lower-severity CVEs are stored)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary of changes

* **New Features**
* Added advanced CVE chart request filters: software categories,
known-exploit flag, EPSS min/max, severity min/max, and excluded CVEs.
* Expanded CVE chart coverage to use the full “collectible” CVE set,
with filtering applied when serving chart data.

* **Tests**
* Added coverage for collecting collectible CVEs and resolving chart
entities based on filter combinations and exclusions.

* **Chores**
  * Updated CVE chart backfill to use collectible CVE discovery.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-19 10:52:03 -05:00
Scott Gress d7fa35e417 Implement roaring bitmaps for historical data collection (#45709) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #45715 

# Details

This PR refactors the way the charts module stores historical data to
use the [roaring bitmap](https://github.com/RoaringBitmap/roaring)
package instead of saving raw bitmaps. See [this
blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives)
to learn how roaring compresses data, but TL;DR for our purposes it
represents a huge improvement especially for larger deployments where
host ID numbers may be very large. In testing, some data was reduced
96%.

The majority of the changes in this PR are straight swapping of types
from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and
updating the internals of our bit math helpers to use roaring methods
instead of native AND and OR methods. I've tried to comment on all
functional changes.

Since the charts have been shipped already, so there will be data in the
wild in the prior "dense" format, the code still handles dense bitmaps
on _read_, but will always _write_ roaring bitmaps. The majority of the
data will therefore have turned over within 30 days on its own, but I
plan on a follow-up PR that will transform open rows when the cron runs
so that we should be guaranteed to turn over completely within 30 days.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- Tests updated to accommodate the new format, and existing unchanged
tests act as proof against regression
- [X] QA'd all new/changed functionality manually
- Using a tool that dumps the `host_scd_data` rows data into a JSON file
(with the keys being entity_id+data and the values being host IDs on
that date), compared the data from main branch and this and confirmed
they're identical
- With a host count of ~9000, some of which have IDs of over 1,000,000,
the data storage requirements were:
     * 82,558,976 bytes for dense
     * 2,867,200 for roaring (a 96% decrease)

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
  - should hugely improve
- [X] Alerted the release DRI if additional load testing is needed

## Database migrations

- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Implemented roaring bitmaps in historical data collection to optimize
bitmap handling for chart data aggregation
* Added encoding support to bitmap storage schema for flexible data
representation

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-19 09:34:29 -05:00
Scott Gress 28908e6083 Dashboard charts backend (#43910) 
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #42812 

# Details

This PR implements a new bounded context, `chart`, with a single
endpoint `/charts`. The context encompasses a framework for recording
and querying and aggregating historical data for Fleet hosts, and
returning that data via the API for the purpose of charting.

This initial iteration has a full implementation of a dataset called
"uptime" which captures which hosts were online hour-by-hour (online
meaning, having been "seen" at some point during that hour). It has a
partial implementation of a "cve" dataset which will capture which hosts
were vulnerable to which CVEs during a given day.

### Data storage

Data is stored in an SCD (slowly-changing dimension) format in the
`host_scd_data` table, where the main "value" in a row is stored in the
`host_bitmap` column, which is a `mediumblob` where each bit encodes a
host ID (bit one represents host ID 1, bit 1444 represents host ID 1444,
etc.). The set of bits set on a row represents that hosts for which that
dataset is "on" during a given time period represented by the
`valid_from` (inclusive) and `valid_to` (exclusive) dates, where a
`valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000
meaning that the row is still "open" (the value represents everything
from `valid_from` to the present). Additionally an `entity_id` column
can be used for datasets with multiple dimensions, e.g. CVE exposure or
software usage which would have entity IDs representing CVEs or software
items respectively.

### Data collection

Data is collected via a cron job that runs every 10 minutes. Each
dataset has its own `Collect` method which will sample the data for the
given moment. For example the "uptime" dataset gathers the set of hosts
that are online at the moment, and the "cve" dataset will gather the set
of hosts that are vulnerable to each CVE at that moment. The sample can
then be recorded using one of two strategies:

* `accumulate`: bitwise OR the sample with any data already recorded for
the current hour, or add a new pre-closed row for that hour.
* `snapshot`: if there is no open row, create one with the sample and
`valid_to set` to the sentinel. Otherwise:
  * If the sample has the same value as the current open row, do nothing
* If the sample has a different value and the current open row's
`valid_from` is within the same hour, update the current row's value
* If the sample has a different value and the current open row's
`valid_from` is not within the same hour, close the current open row and
start a new one with `valid_from` = the start of the current hour

### Data retrieval 

1. Gets the set of host IDs to retrieve data for. This starts with the
set of host IDs in the requested fleet (or all the hosts a user has
access to if no `fleet_id` param was passed to the `/charts` endpoint),
and further whittled down by any filter options supplied with the
request (labels, platforms, etc.).
2. Finds all `host_scd_data` rows for the requested dataset and date
range (i.e. all rows whose `valid_from` is < the date range end and
`valid_to` is > the date range start).
3. Calculates the date ranges of the "buckets" to return datapoints for.
For the uptime chart we default to 3-hour buckets, so we want 8 buckets
per day.
4. Iterates over each bucket and finds the row or rows from
host_scd_data that cover that bucket range. For datasets using the
"accumulate" strategy, the values for those rows are ORed together. For
"snapshot"s, we take the one active at the bucket end time to represent
the bucket (e.g. "which hosts had a given CVE at the end of the day")

### Tools

This PR includes two dev tools that don't require deep review:

* **chart-backfill** - used to backfill data to various datasets for
testing
* **charts-collect** - used to collect data from a live server via the
API and put into a local hosts_scd_data table

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [X] QA'd all new/changed functionality manually
  - With [front-end branch](https://github.com/fleetdm/fleet/pull/43878)
<img width="712" height="434" alt="image"
src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c"
/>

## Database migrations

- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added charting bounded context: HTTP API for metrics (uptime, CVE),
dataset registry, hosted dataset collection, background
collection/cleanup with opt-out env.
* New utilities: host bitmap operations and string-list/uint-list
parsers.
  * New CLI tools to collect and backfill chart data.

* **Database**
  * Migration and schema to store host time-series SCD chart data.

* **Tests**
* Extensive unit and integration tests for service, storage, caching,
cron, and utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-23 12:43:23 -05:00