Files
fleet/server/chart/internal/testutils/testutils.go
T
Scott Gress d7fa35e417 Implement roaring bitmaps for historical data collection (#45709)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #45715 

# Details

This PR refactors the way the charts module stores historical data to
use the [roaring bitmap](https://github.com/RoaringBitmap/roaring)
package instead of saving raw bitmaps. See [this
blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives)
to learn how roaring compresses data, but TL;DR for our purposes it
represents a huge improvement especially for larger deployments where
host ID numbers may be very large. In testing, some data was reduced
96%.

The majority of the changes in this PR are straight swapping of types
from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and
updating the internals of our bit math helpers to use roaring methods
instead of native AND and OR methods. I've tried to comment on all
functional changes.

Since the charts have been shipped already, so there will be data in the
wild in the prior "dense" format, the code still handles dense bitmaps
on _read_, but will always _write_ roaring bitmaps. The majority of the
data will therefore have turned over within 30 days on its own, but I
plan on a follow-up PR that will transform open rows when the cron runs
so that we should be guaranteed to turn over completely within 30 days.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.

## Testing

- [X] Added/updated automated tests
- Tests updated to accommodate the new format, and existing unchanged
tests act as proof against regression
- [X] QA'd all new/changed functionality manually
- Using a tool that dumps the `host_scd_data` rows data into a JSON file
(with the keys being entity_id+data and the values being host IDs on
that date), compared the data from main branch and this and confirmed
they're identical
- With a host count of ~9000, some of which have IDs of over 1,000,000,
the data storage requirements were:
     * 82,558,976 bytes for dense
     * 2,867,200 for roaring (a 96% decrease)

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
  - should hugely improve
- [X] Alerted the release DRI if additional load testing is needed

## Database migrations

- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Implemented roaring bitmaps in historical data collection to optimize
bitmap handling for chart data aggregation
* Added encoding support to bitmap storage schema for flexible data
representation

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-19 09:34:29 -05:00

146 lines
4.9 KiB
Go

// Package testutils provides shared test utilities for the chart bounded context.
package testutils
import (
"log/slog"
"testing"
"time"
"github.com/fleetdm/fleet/v4/server/chart"
common_mysql "github.com/fleetdm/fleet/v4/server/platform/mysql"
mysql_testing_utils "github.com/fleetdm/fleet/v4/server/platform/mysql/testing_utils"
"github.com/jmoiron/sqlx"
"github.com/stretchr/testify/require"
)
// TestDB holds the database connection for tests.
type TestDB struct {
DB *sqlx.DB
Logger *slog.Logger
}
// SetupTestDB creates a test database with the Fleet schema loaded. Tests are
// skipped automatically when MYSQL_TEST is not set.
func SetupTestDB(t *testing.T, testNamePrefix string) *TestDB {
t.Helper()
testName, opts := mysql_testing_utils.ProcessOptions(t, &mysql_testing_utils.DatastoreTestOptions{
UniqueTestName: testNamePrefix + "_" + t.Name(),
})
mysql_testing_utils.LoadDefaultSchema(t, testName, opts)
config := mysql_testing_utils.MysqlTestConfig(testName)
db, err := common_mysql.NewDB(config, &common_mysql.DBOptions{}, "")
require.NoError(t, err)
t.Cleanup(func() { db.Close() })
return &TestDB{
DB: db,
Logger: slog.New(slog.DiscardHandler),
}
}
// Conns returns DBConnections for creating a datastore.
func (tdb *TestDB) Conns() *common_mysql.DBConnections {
return &common_mysql.DBConnections{Primary: tdb.DB, Replica: tdb.DB}
}
// TruncateTables clears the tables used by the chart bounded context.
func (tdb *TestDB) TruncateTables(t *testing.T) {
t.Helper()
mysql_testing_utils.TruncateTables(t, tdb.DB, tdb.Logger, nil,
"host_scd_data", "hosts", "host_seen_times", "nano_enrollments", "teams")
}
// InsertSCDRow inserts a single host_scd_data row for tests. host_bitmap is
// stored as an empty blob since cleanup tests don't care about its contents.
func (tdb *TestDB) InsertSCDRow(t *testing.T, dataset, entityID string, validFrom, validTo time.Time) {
t.Helper()
ctx := t.Context()
_, err := tdb.DB.ExecContext(ctx, `
INSERT INTO host_scd_data (dataset, entity_id, host_bitmap, valid_from, valid_to)
VALUES (?, ?, ?, ?, ?)
`, dataset, entityID, []byte{}, validFrom, validTo)
require.NoError(t, err)
}
// InsertSCDRowWithBlob inserts a host_scd_data row with a caller-supplied
// chart.Blob (bytes + encoding) and returns the auto-assigned id.
func (tdb *TestDB) InsertSCDRowWithBlob(t *testing.T, dataset, entityID string, blob chart.Blob, validFrom, validTo time.Time) uint {
t.Helper()
ctx := t.Context()
res, err := tdb.DB.ExecContext(ctx, `
INSERT INTO host_scd_data (dataset, entity_id, host_bitmap, encoding_type, valid_from, valid_to)
VALUES (?, ?, ?, ?, ?, ?)
`, dataset, entityID, blob.Bytes, blob.Encoding, validFrom, validTo)
require.NoError(t, err)
id, err := res.LastInsertId()
require.NoError(t, err)
require.GreaterOrEqual(t, id, int64(0), "AUTO_INCREMENT should never produce a negative id")
return uint(id) //nolint:gosec // G115: id is a positive AUTO_INCREMENT primary key
}
// InsertSCDRowWithHostIDs is a convenience wrapper for tests that just want to
// store a set of host IDs — produces a roaring-encoded row.
func (tdb *TestDB) InsertSCDRowWithHostIDs(t *testing.T, dataset, entityID string, hostIDs []uint, validFrom, validTo time.Time) uint {
t.Helper()
return tdb.InsertSCDRowWithBlob(t, dataset, entityID, chart.HostIDsToBlob(hostIDs), validFrom, validTo)
}
// DenseBlob builds a legacy dense-encoded chart.Blob for the given host IDs.
// Used to seed pre-migration fixtures that exercise the dense decode path.
// Production writes always go through chart.HostIDsToBlob (roaring).
func DenseBlob(ids []uint) chart.Blob {
if len(ids) == 0 {
return chart.Blob{Encoding: chart.EncodingDense}
}
var maxID uint
for _, id := range ids {
if id > maxID {
maxID = id
}
}
bytes := make([]byte, maxID/8+1)
for _, id := range ids {
bytes[id/8] |= 1 << (id % 8)
}
return chart.Blob{Bytes: bytes, Encoding: chart.EncodingDense}
}
// SCDBlob returns the host_bitmap + encoding_type for the given row id.
func (tdb *TestDB) SCDBlob(t *testing.T, id uint) chart.Blob {
t.Helper()
ctx := t.Context()
type row struct {
HostBitmap []byte `db:"host_bitmap"`
EncodingType uint8 `db:"encoding_type"`
}
var r row
err := tdb.DB.GetContext(ctx, &r, `SELECT host_bitmap, encoding_type FROM host_scd_data WHERE id = ?`, id)
require.NoError(t, err)
return chart.Blob{Bytes: r.HostBitmap, Encoding: r.EncodingType}
}
// SCDHostIDs returns the decoded host IDs for the given row id.
func (tdb *TestDB) SCDHostIDs(t *testing.T, id uint) []uint {
t.Helper()
rb, err := chart.DecodeBitmap(tdb.SCDBlob(t, id))
require.NoError(t, err)
return chart.BitmapToHostIDs(rb)
}
// CountSCDRows returns the total number of rows in host_scd_data.
func (tdb *TestDB) CountSCDRows(t *testing.T) int {
t.Helper()
ctx := t.Context()
var n int
err := tdb.DB.GetContext(ctx, &n, `SELECT COUNT(*) FROM host_scd_data`)
require.NoError(t, err)
return n
}