Lucas Manuel Rodriguez
001b57cb9d
Resolves #50266. At production numbers the table looks like this - 20,691 CVEs × 83,000 hosts, ~268M raw (cve, host) rows (software + OS joins combined): ``` ┌─────────────────────────┬─────────────────────────┬───────────────────────┐ │ Shape of host IDs │ Old (map[string][]uint) │ New (roaring bitmaps) │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Dense (contiguous runs) │ 2,479 MB │ 4.5 MB │ ├─────────────────────────┼─────────────────────────┼───────────────────────┤ │ Sparse (random) │ 2,488 MB │ 282 MB │ └─────────────────────────┴─────────────────────────┴───────────────────────┘ ``` A few things worth noting about how these map to your real data: - The old cost is shape-independent: ~2.5 GB retained just for the result map (268M rows × 8 bytes plus append slack), and the peak during collection is higher still because append doubling leaves garbage behind. That's the number that was blowing up the cron. - The new sparse figure is an overstated worst case. Your 268M rows include duplicates — multiple vulnerable software rows per host for the same CVE (the multi-kernel case) and overlap between the software and OS joins. The old code retained every raw row; the bitmap dedupes on Add, so it's bounded by unique pairs, and real fleets with AUTO_INCREMENT host IDs sit much closer to the dense row than the sparse one. - The new representation also has a hard ceiling the old one doesn't: a roaring bitmap over 83k host IDs maxes out around 16 KB per CVE regardless of contents, so even a pathological dataset caps at ~330 MB for all 20,691 CVEs — versus the old form growing linearly with join rows, unbounded. TL;DR: At scale the change is roughly a 550× reduction in the realistic (dense) case, and at minimum ~9× in the theoretical worst case. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance** - Reduced memory usage for CVE chart data collection. - Improved efficiency when processing large CVE and affected-host datasets. - **Bug Fixes** - Preserved correct CVE filtering, duplicate-host handling, disabled-fleet exclusions, and empty-result behavior. - Added coverage for CVEs sourced from both software and operating-system data. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
63 lines
2.6 KiB
Go
63 lines
2.6 KiB
Go
package chart
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"github.com/RoaringBitmap/roaring"
|
|
"github.com/fleetdm/fleet/v4/server/chart/api"
|
|
)
|
|
|
|
// UptimeDataset implements api.Dataset for host uptime tracking.
|
|
type UptimeDataset struct{}
|
|
|
|
func (u *UptimeDataset) Name() string { return "uptime" }
|
|
func (u *UptimeDataset) DefaultResolutionHours() int { return 3 }
|
|
func (u *UptimeDataset) SampleStrategy() api.SampleStrategy { return api.SampleStrategyAccumulate }
|
|
func (u *UptimeDataset) DefaultVisualization() string { return "checkerboard" }
|
|
|
|
func (u *UptimeDataset) Collect(ctx context.Context, store api.DatasetStore, now time.Time, disabledFleetIDs []uint) error {
|
|
hostIDs, err := store.FindOnlineHostIDs(ctx, now, disabledFleetIDs)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if len(hostIDs) == 0 {
|
|
return nil
|
|
}
|
|
bucketStart := now.UTC().Truncate(time.Hour)
|
|
return store.RecordBucketData(ctx, u.Name(), bucketStart, time.Hour, u.SampleStrategy(),
|
|
// The empty string key means "all entities" since uptime isn't tracked per host.
|
|
// The value is a bitmap of host IDs that were active in this bucket.
|
|
map[string]*roaring.Bitmap{"": NewBitmap(hostIDs)})
|
|
}
|
|
|
|
// CVEDataset implements api.Dataset for host CVE tracking.
|
|
type CVEDataset struct{}
|
|
|
|
func (c *CVEDataset) Name() string { return api.MetricCVE }
|
|
func (c *CVEDataset) DefaultResolutionHours() int { return 3 }
|
|
func (c *CVEDataset) SampleStrategy() api.SampleStrategy { return api.SampleStrategySnapshot }
|
|
func (c *CVEDataset) DefaultVisualization() string { return "line" }
|
|
|
|
func (c *CVEDataset) Collect(ctx context.Context, store api.DatasetStore, now time.Time, disabledFleetIDs []uint) error {
|
|
// Collect CVEs at all severities on the curated set of tracked software and
|
|
// OS vulnerabilities. Display-time narrowing (critical-only this round,
|
|
// plus user filters) happens at read time via ResolveCVEChartEntities.
|
|
tracked, err := store.CollectibleCVEs(ctx)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// The store sets bits while streaming the vulnerability joins, so peak
|
|
// memory here is one bitmap per CVE — never the raw (CVE, host) pairs.
|
|
bitmaps, err := store.AffectedHostIDsByCVE(ctx, disabledFleetIDs, tracked)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
bucketStart := now.UTC().Truncate(time.Hour)
|
|
// Always call RecordBucketData, even when bitmaps is empty: snapshot
|
|
// semantics use an empty input to close any open rows for entities no
|
|
// longer in the tracked set (recordSnapshot's "absent entities" branch).
|
|
return store.RecordBucketData(ctx, c.Name(), bucketStart, time.Hour, c.SampleStrategy(), bitmaps)
|
|
}
|