Scott Gress
d7fa35e417
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45715 # Details This PR refactors the way the charts module stores historical data to use the [roaring bitmap](https://github.com/RoaringBitmap/roaring) package instead of saving raw bitmaps. See [this blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives) to learn how roaring compresses data, but TL;DR for our purposes it represents a huge improvement especially for larger deployments where host ID numbers may be very large. In testing, some data was reduced 96%. The majority of the changes in this PR are straight swapping of types from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and updating the internals of our bit math helpers to use roaring methods instead of native AND and OR methods. I've tried to comment on all functional changes. Since the charts have been shipped already, so there will be data in the wild in the prior "dense" format, the code still handles dense bitmaps on _read_, but will always _write_ roaring bitmaps. The majority of the data will therefore have turned over within 30 days on its own, but I plan on a follow-up PR that will transform open rows when the cron runs so that we should be guaranteed to turn over completely within 30 days. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - Tests updated to accommodate the new format, and existing unchanged tests act as proof against regression - [X] QA'd all new/changed functionality manually - Using a tool that dumps the `host_scd_data` rows data into a JSON file (with the keys being entity_id+data and the values being host IDs on that date), compared the data from main branch and this and confirmed they're identical - With a host count of ~9000, some of which have IDs of over 1,000,000, the data storage requirements were: * 82,558,976 bytes for dense * 2,867,200 for roaring (a 96% decrease) For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - should hugely improve - [X] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented roaring bitmaps in historical data collection to optimize bitmap handling for chart data aggregation * Added encoding support to bitmap storage schema for flexible data representation <!-- end of auto-generated comment: release notes by coderabbit.ai -->
64 lines
2.5 KiB
Go
64 lines
2.5 KiB
Go
package chart
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"github.com/RoaringBitmap/roaring"
|
|
"github.com/fleetdm/fleet/v4/server/chart/api"
|
|
)
|
|
|
|
// UptimeDataset implements api.Dataset for host uptime tracking.
|
|
type UptimeDataset struct{}
|
|
|
|
func (u *UptimeDataset) Name() string { return "uptime" }
|
|
func (u *UptimeDataset) DefaultResolutionHours() int { return 3 }
|
|
func (u *UptimeDataset) SampleStrategy() api.SampleStrategy { return api.SampleStrategyAccumulate }
|
|
func (u *UptimeDataset) DefaultVisualization() string { return "checkerboard" }
|
|
|
|
func (u *UptimeDataset) Collect(ctx context.Context, store api.DatasetStore, now time.Time, disabledFleetIDs []uint) error {
|
|
hostIDs, err := store.FindOnlineHostIDs(ctx, now, disabledFleetIDs)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if len(hostIDs) == 0 {
|
|
return nil
|
|
}
|
|
bucketStart := now.UTC().Truncate(time.Hour)
|
|
return store.RecordBucketData(ctx, u.Name(), bucketStart, time.Hour, u.SampleStrategy(),
|
|
// The empty string key means "all entities" since uptime isn't tracked per host.
|
|
// The value is a bitmap of host IDs that were active in this bucket.
|
|
map[string]*roaring.Bitmap{"": NewBitmap(hostIDs)})
|
|
}
|
|
|
|
// CVEDataset implements api.Dataset for host CVE tracking.
|
|
type CVEDataset struct{}
|
|
|
|
func (c *CVEDataset) Name() string { return "cve" }
|
|
func (c *CVEDataset) DefaultResolutionHours() int { return 3 }
|
|
func (c *CVEDataset) SampleStrategy() api.SampleStrategy { return api.SampleStrategySnapshot }
|
|
func (c *CVEDataset) DefaultVisualization() string { return "line" }
|
|
|
|
func (c *CVEDataset) Collect(ctx context.Context, store api.DatasetStore, now time.Time, disabledFleetIDs []uint) error {
|
|
// Only track the CVEs that the chart API currently returns.
|
|
// TODO: implement bitmap compression so we can track all CVEs.
|
|
tracked, err := store.TrackedCriticalCVEs(ctx)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
hostIDsByCVE, err := store.AffectedHostIDsByCVE(ctx, disabledFleetIDs, tracked)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
bitmaps := make(map[string]*roaring.Bitmap, len(hostIDsByCVE))
|
|
for cve, hostIDs := range hostIDsByCVE {
|
|
bitmaps[cve] = NewBitmap(hostIDs)
|
|
}
|
|
bucketStart := now.UTC().Truncate(time.Hour)
|
|
// Always call RecordBucketData, even when bitmaps is empty: snapshot
|
|
// semantics use an empty input to close any open rows for entities no
|
|
// longer in the tracked set (recordSnapshot's "absent entities" branch).
|
|
return store.RecordBucketData(ctx, c.Name(), bucketStart, time.Hour, c.SampleStrategy(), bitmaps)
|
|
}
|