Files

T

Scott Gress c370a9672b Add CVE chart filtering and non-critical CVE data collection (backend) (#47470 )

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #44746 

# Details

* Adds the ability to filter historical CVE data by software type, EPSS,
CVSS, CVE ID (exclude only) and "has known exploit"
* Hard-codes the CVSS filter to 9.0+ for now, since that's the only data
that's been collected thus far
* Un-gates the collection code so that it will collect CVE data for
_all_ severities (but still in the restricted set of software)

Related PRs [update the
front-end](https://github.com/fleetdm/fleet/pull/47674) to allow sending
these filters, and [update
GitOps](https://github.com/fleetdm/fleet/pull/47634) to allow changing
the default filters.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually

### Manual test plan — CVE chart filtering (backend smoke test)

#### Setup
- Premium dev server running with a few hosts carrying vulnerable
software (so `cve_meta` / `software_cve` /
  `operating_system_vulnerabilities` are populated)
- Chart data present — collector ran once, or seeded: `go run
./tools/charts-backfill --dataset cve
  --use-tracked-cves --days 7`
  - API token exported and helper set:
    ```bash
    BASE=https://localhost:8080/api/v1/fleet/charts
peak() { curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/$1" | jq
'[.data[].value] | max'; }

####  Checks (compare against the no-filter baseline)

- [x] Baseline returns data — GET /charts/cve?days=7 returns a data
series; .filters is empty/default
- [x] Severity force-pinned to critical — cve?days=7 and
cve?days=7&severity_min=0&severity_max=10 give identical
  peaks (no low-severity leak; client severity ignored)
- [x] Category narrowing — software_categories=browsers ≤ baseline;
software_categories=os,browsers,office,adobe ==
  baseline
- [x] OS category includes kernel — software_categories=os returns
OS-vuln + Linux-kernel CVE counts
  - [x] Known-exploit narrowing — known_exploit=true ≤ baseline
- [x] EPSS narrowing — epss_min=0.9 ≤ baseline; epss_min=0&epss_max=1 ==
baseline (EPSS is 0.0–1.0 on the API)
- [x] Exclude is subtractive + tolerant — excluding a visible CVE
lowers/keeps counts; exclude_cves=CVE-0000-00000
  == baseline (no-op)
- [x] Filters echo back — filtered requests return applied values under
.filters
- [x] Uptime untouched — GET /charts/uptime?days=7 returns its normal
series
- [x] Free-tier safety (optional) — on non-Premium, /charts/cve returns
an empty series, no error
  - [x] > 0 rows from:
  SELECT COUNT(DISTINCT scd.entity_id) AS below_critical
  FROM host_scd_data scd JOIN cve_meta cm ON cm.cve = scd.entity_id
  WHERE scd.dataset='cve' AND cm.cvss_score < 9.0;
  - (confirms lower-severity CVEs are stored)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Summary of changes

* **New Features**
* Added advanced CVE chart request filters: software categories,
known-exploit flag, EPSS min/max, severity min/max, and excluded CVEs.
* Expanded CVE chart coverage to use the full “collectible” CVE set,
with filtering applied when serving chart data.

* **Tests**
* Added coverage for collecting collectible CVEs and resolving chart
entities based on filter combinations and exclusions.

* **Chores**
  * Updated CVE chart backfill to use collectible CVE discovery.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-06-19 10:52:03 -05:00

main.go

Add CVE chart filtering and non-critical CVE data collection (backend) (#47470 )

2026-06-19 10:52:03 -05:00

README.md

…

README.md

charts-backfill

Generates synthetic chart data for development and testing. Writes rows to host_scd_data using ON DUPLICATE KEY UPDATE, so it is safe to re-run. All writes use the roaring bitmap encoding (encoding_type = 1).

Usage

go run ./tools/charts-backfill --dataset uptime --days 30
go run ./tools/charts-backfill --dataset uptime --days 7 --host-ids 1,2,3
go run ./tools/charts-backfill --dataset cve --days 30 --use-tracked-cves
go run ./tools/charts-backfill --dataset cve --days 30 --entity-ids CVE-2024-1,CVE-2024-2
go run ./tools/charts-backfill --mysql-dsn "fleet:fleet@tcp(localhost:3306)/fleet"

Flags

Flag	Default	Description
`--dataset`	`uptime`	Dataset name (`uptime`, `policy`, `cve`, ...)
`--days`	`30`	Number of days to backfill
`--start-date`	`now - days`	Start date (`YYYY-MM-DD`)
`--entity-ids`	`""`	Comma-separated entity IDs (e.g. CVE IDs); `""` for non-entity datasets
`--use-tracked-cves`	`false`	For `--dataset cve`, auto-discover entity IDs from the production tracked-CVE query (joins `software_cve` / `operating_system_vulnerabilities` against the curated software matchers; requires vulnerability data to be populated). Overrides `--entity-ids`.
`--host-ids`	all hosts	Comma-separated host IDs to include
`--mysql-dsn`	local dev	MySQL connection string

Datasets

Backfill mode matches the live collector's sample strategy for each dataset:

Accumulate, hourly (default; uptime, policy): 24 independent rows per day per entity, each a fresh random sample. valid_to is set to one hour past valid_from.
Snapshot, state-segment (cve): per-entity state-segment rows shaped like real CVE data. Each entity gets an initial host set; for each subsequent day, with ~5% probability the set is churned (~10% drop, ~10% add). Each contiguous run of unchanged days collapses to a single row. The final segment per entity leaves valid_to at the open sentinel so the live collector compares against it on its next tick instead of inserting over the top. Pair with --use-tracked-cves to mirror production CVE selection.

Density (fraction of hosts marked active) for the initial sample varies by dataset — see densityRange in main.go. Snapshot churn parameters (snapshotFlipsPerDayPerEntity, snapshotChurnFraction) are also defined there.