Lucas Manuel Rodriguez
001b57cb9d
Resolves #50266. At production numbers the table looks like this - 20,691 CVEs × 83,000 hosts, ~268M raw (cve, host) rows (software + OS joins combined): ``` ┌─────────────────────────┬─────────────────────────┬───────────────────────┐ │ Shape of host IDs │ Old (map[string][]uint) │ New (roaring bitmaps) │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Dense (contiguous runs) │ 2,479 MB │ 4.5 MB │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (random) │ 2,488 MB │ 282 MB │ └─────────────────────────┴─────────────────────────┴───────────────────────┘ ``` A few things worth noting about how these map to your real data: - The old cost is shape-independent: ~2.5 GB retained just for the result map (268M rows × 8 bytes plus append slack), and the peak during collection is higher still because append doubling leaves garbage behind. That's the number that was blowing up the cron. - The new sparse figure is an overstated worst case. Your 268M rows include duplicates — multiple vulnerable software rows per host for the same CVE (the multi-kernel case) and overlap between the software and OS joins. The old code retained every raw row; the bitmap dedupes on Add, so it's bounded by unique pairs, and real fleets with AUTO_INCREMENT host IDs sit much closer to the dense row than the sparse one. - The new representation also has a hard ceiling the old one doesn't: a roaring bitmap over 83k host IDs maxes out around 16 KB per CVE regardless of contents, so even a pathological dataset caps at ~330 MB for all 20,691 CVEs — versus the old form growing linearly with join rows, unbounded. TL;DR: At scale the change is roughly a 550× reduction in the realistic (dense) case, and at minimum ~9× in the theoretical worst case. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance** - Reduced memory usage for CVE chart data collection. - Improved efficiency when processing large CVE and affected-host datasets. - **Bug Fixes** - Preserved correct CVE filtering, duplicate-host handling, disabled-fleet exclusions, and empty-result behavior. - Added coverage for CVEs sourced from both software and operating-system data. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 lines
48 B
Plaintext
2 lines
48 B
Plaintext
* Optimized memory usage of CVE chart cron job.
|