<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #31820 and #39898 Vulnerability processing performance improvements, and added OTEL spans to the vulnerabilities cron job. Optimized the two main bottlenecks in the vulnerability cron job: CPE matching and CVE insertion. In my loadtest testing (10K hosts), the overall initial vulnerabilities job went from over 2 hours down to 53 minutes, and the number of spans (DB accesses) went from ~2 million to ~90K. 1. CPE matching (TranslateSoftwareToCPE): replaced the goqu query builder with hand-written SQL using raw database/sql queries. Replaced UNION with separate queries because case number 3 was an expensive full text match operation and in most cases we did not need to do that. 2. CVE insertion (TranslateCPEToCVE and other places): replaced individual INSERT INTO software_cve ... VALUES (?,?,?,?) calls with batch inserts of 500 rows each, using the existing BatchProcessSimple helper. Same pattern applied to OS vulnerability inserts using the existing InsertOSVulnerabilities batch method. Functional equivalence verified using osquery perf dataset locally. Both changes produce identical output (22,366 CPEs, 131,233 CVEs) when compared against the old code using a before/after comparison tool. - CPE caveats: bugs #39898 and https://github.com/fleetdm/fleet/issues/39899 found # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [x] Added/updated automated tests - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded tracing for automated vulnerability workflows to improve observability. * **Performance** * Bulk/batched processing for software and OS vulnerability inserts to speed ingestion and downstream tasks. * More efficient CPE lookup and read-optimized database access for faster translations. * **Bug Fixes** * Improved error recording and read-after-write consistency to reduce missed or duplicate vulnerability notifications. * **Tests** * Test suite updated to support batch insertion semantics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
CPE Translations
CPE Translations are rules to address bugs when translating Fleet software to Common Platform Enumerations (CPEs) which are used to identify software in the National Vulnerability Database (NVD)
To improve accuracy when mapping software to CVEs, we can add data to cpe_translations.json
How CPE translations work
CPE Translations are defined in cpe_translations.json and currently released in
GitHub once a day. The rules are specified in JSON format and
and each rule consists of a software and a filter object.
software defines matching logic on what Fleet Software this rule should apply to. You can use one
or more of the below attributes to match on. Each attribute is an array of string or regex
matches (a regex string is identified by a leading and trailing /).
A match on the attribute is found if at least 1 item in the array matches. If multiple
attributes are defined, then a match is needed for each attribute. (ie. name == Zoom.app &&
source == apps)
software attributes:
name: A software name attributebundle_identifier: A software bundle_identifier attribute (macOS only)source: A software source attribute (ie.apps,chrome_extensions, etc...)
example: Search Fleet software for items that match: (bundle_identifier == us.zoom.xos) AND (source = apps)
"software": {
"bundle_identifier": ["us.zoom.xos"],
"source": ["apps"]
}
If the software rule matches, then Fleet will search known NVD CPEs (stored in a local sqlite database) using the
specified filters or skip the software item based on the filter specified.
filter attributes:
product: array of strings to search by product field. If not specified, the software name is used.vendor: array of strings to search by vendor fieldtarget_sw: array of strings to search by target_sw fieldpart: string to override the default "a" Part valueskip: boolean; software is skipped iftrue. This overrides any other filters set.
Like the software matching logic, filter items are matched by OR within the array, and AND between filter items
example: Query the CPE database for a CPE that matches: (product == zoom OR product == meetings) AND (vendor == zoom) AND (target == macos OR target == mac_os)
"filter": {
"product": ["zoom", "meetings"],
"vendor": ["zoom"],
"target_sw": ["macos", "mac_os"]
}
Testing CPE Translations (end-to-end)
-
make the appropriate changes to cpe_translations
-
host this file on a local web server
go run ./tools/file-server/main.go 8082 ./server/vulnerabilities/nvd/ -
(re)launch your local fleet server with one of the following
Config method
vulnerabilities: cpe_translations_url: "http://localhost:8082/cpe_translations.json"Environment method
FLEET_VULNERABILITIES_CPE_TRANSLATIONS_URL="http://localhost:8082/cpe_translations.json" ./build/fleet serve --dev --dev_license --logging_debug -
trigger a vulnerabilities scan
fleetctl trigger --name vulnerabilities