diff --git a/server/chart/CLAUDE.md b/server/chart/CLAUDE.md new file mode 100644 index 0000000000..5ba3ec09f3 --- /dev/null +++ b/server/chart/CLAUDE.md @@ -0,0 +1,23 @@ +# Chart bounded context + +Before working in `server/chart/`, read [README.md](./README.md) — it covers the +architecture, the SCD + bitmap data model, sample strategies, and a step-by-step +guide for adding new datasets and charts. + +Quick reminders (full detail in the README): + +- This is a **self-contained bounded context**. No chart package may import + `server/fleet` or `server/contexts/viewer` — `arch_test.go` enforces this. Bridge + to legacy Fleet via a narrow `api` interface implemented in `server/acl/chartacl`. +- External code only touches `bootstrap.New` and the `api` package. +- Data lives in one table, `host_scd_data` (slowly-changing-dimension type-2), with + host-sets stored as roaring bitmaps. Work in op form (`*roaring.Bitmap`); only + cross the storage boundary via `BitmapToBlob` / `DecodeBitmap`. +- Mind the **nil vs empty-slice** semantics on `HostFilter.TeamIDs` and + `GetSCDData` `entityIDs` — they are load-bearing; don't normalize them away. +- Snapshot collectors must call `RecordBucketData` even with an empty map (it closes + open rows). Accumulate collectors may skip empty input. +- Adding a dataset: implement `api.Dataset` in `datasets.go`, add any store method to + **both** `api.DatasetStore` and `internal/types.Datastore`, register it in + `cmd/fleet/serve.go`, and wire config gating + scrub in + `server/fleet/historical_data.go` if it's opt-in/out. diff --git a/server/chart/README.md b/server/chart/README.md new file mode 100644 index 0000000000..0ee2d6324b --- /dev/null +++ b/server/chart/README.md @@ -0,0 +1,458 @@ +# Chart bounded context + +Time-series charting for the Fleet dashboard. Given a metric like "how many hosts +were online" or "how many hosts were affected by a critical CVE," this package +records a per-hour history and serves it back as bucketed data points the frontend +renders as a line, checkerboard, etc. + +This is a **self-contained bounded context** (like `server/activity/` and +`server/mdm/`), not the traditional `server/fleet` → `server/service` → +`server/datastore` layering. It owns its own types, datastore, service, HTTP +transport, and bootstrap, and it must not import `server/fleet` or +`server/contexts/viewer` directly — an architecture test enforces this (see +[Dependency rules](#dependency-rules)). + +## Table of contents + +- [Mental model](#mental-model) +- [Package layout](#package-layout) +- [Dependency rules](#dependency-rules) +- [Data model: SCD type-2 + bitmaps](#data-model-scd-type-2--bitmaps) +- [Sample strategies](#sample-strategies) +- [The write path (collection)](#the-write-path-collection) +- [The read path (GetChartData)](#the-read-path-getchartdata) +- [Scoping, config gating, and scrubbing](#scoping-config-gating-and-scrubbing) +- [How to add a new dataset](#how-to-add-a-new-dataset) +- [How charts reach the frontend](#how-charts-reach-the-frontend) +- [Testing](#testing) +- [File reference](#file-reference) + +## Mental model + +A **dataset** answers one question over time ("uptime," "cve"). On a schedule, a +collection cron asks each dataset to **collect** the current state: it produces a +set of host IDs (optionally grouped by an *entity*, e.g. one host-set per CVE) and +hands them to the datastore as a [roaring bitmap](https://roaringbitmap.org/). The +datastore folds that observation into a slowly-changing-dimension (SCD) history in +the `host_scd_data` table. + +On read, a chart request resolves which hosts the caller may see (a "filter mask" +bitmap), walks the SCD history bucket by bucket, ANDs each bucket's host-set +against the mask, and returns the population count per bucket. Everything is set +math on bitmaps; a "value" on the chart is always a distinct-host count. + +``` +collection cron (cmd/fleet/cron.go) + └─ Service.CollectDatasets(scope) + └─ Dataset.Collect() ──reads hosts──▶ DatasetStore (FindOnlineHostIDs, AffectedHostIDsByCVE, …) + └─ store.RecordBucketData(strategy, map[entity]*bitmap) ──writes──▶ host_scd_data + +HTTP GET /api/_version_/fleet/charts/{metric} + └─ getChartDataEndpoint ─▶ Service.GetChartData() + ├─ ViewerProvider.ViewerScope() → team scope for authz + data + ├─ authz.Authorize() → fail closed + ├─ GetHostIDsForFilter() → mask → cached per filter (hostFilterCache) + └─ GetSCDData(...) walks buckets, AND mask, popcount ─▶ []DataPoint +``` + +## Package layout + +| Path | Role | May depend on | +|------|------|---------------| +| `server/chart` (root) | Bitmap helpers (`blob.go`), `Dataset` implementations (`datasets.go`), shared constants | `api` only | +| `server/chart/api` | **Public surface.** `Service`, `Dataset`, `DatasetStore`, `ViewerProvider` interfaces; request/response types | nothing in Fleet | +| `server/chart/api/http` | HTTP request/response DTOs (wire tags, e.g. `fleet_id`) | `api` only | +| `server/chart/internal/types` | Internal `Datastore` interface, `HostFilter` | `api` only | +| `server/chart/internal/mysql` | MySQL `Datastore` implementation, SQL, SCD read/write | chart, types, platform | +| `server/chart/internal/service` | `Service` implementation, authz, host-filter cache, bucket math | chart, platform | +| `server/chart/bootstrap` | `New(...)` wires datastore + service + routes; entry point for `serve.go` | mysql, service, api, platform | +| `server/chart/internal/testutils` | Test helpers | — | + +External code only ever touches `bootstrap.New` and the `api` package. The +anti-corruption layer that bridges to legacy Fleet types lives **outside** this +tree in `server/acl/chartacl` (the only place that imports both chart `api` and +`server/contexts/viewer`). + +## Dependency rules + +`arch_test.go` runs `archtest` assertions on every package. The rules in plain +English: + +- `api` has **zero** Fleet dependencies (it's the contract). +- `api/http`, `internal/types`, and the root `chart` package depend on **`api` + only**. +- `internal/mysql` and `internal/service` may additionally use `server/platform/...` + and `server/contexts/...`, plus other chart packages. +- **No chart package may import `server/fleet` or `server/contexts/viewer`.** + +If you need something from legacy Fleet (the current user, a fleet config value), +do not import it — add a narrow interface to `api` (see `ViewerProvider`) and +implement it in `server/acl/chartacl`. Then wire it through `bootstrap.New`. + +## Data model: SCD type-2 + bitmaps + +Everything lives in one table, `host_scd_data`: + +```sql +CREATE TABLE host_scd_data ( + id bigint unsigned AUTO_INCREMENT, + dataset varchar(50) NOT NULL, -- "uptime", "cve", … + entity_id varchar(100) NOT NULL DEFAULT '', -- "" for single-dimension; CVE id for cve + host_bitmap mediumblob NOT NULL, -- serialized host-id set + valid_from datetime NOT NULL, + valid_to datetime NOT NULL DEFAULT '9999-12-31 00:00:00', -- sentinel = "still open" + encoding_type tinyint NOT NULL DEFAULT 0, -- 0 = dense, 1 = roaring + PRIMARY KEY (id), + UNIQUE KEY uniq_entity_bucket (dataset, entity_id, valid_from), + KEY idx_dataset_range (dataset, valid_from, valid_to), + KEY idx_valid_to_dataset (valid_to, dataset, entity_id) +); +``` + +A **row is a host-set that was valid over `[valid_from, valid_to)`**. The +`9999-12-31` sentinel means "currently open." This is a textbook +[slowly-changing-dimension type-2](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) +table: state changes append a new row and close the old one rather than mutating +in place, so history is preserved. + +`entity_id` is the sub-dimension. Single-dimension datasets (uptime) use the empty +string. Multi-dimension datasets (cve) write one row per entity (per CVE) and the +read path ORs across entities to get a distinct-host union. + +### Bitmap encoding (`blob.go`) + +`host_bitmap` stores a set of host IDs. Two on-disk formats, discriminated by +`encoding_type`: + +- **`EncodingDense` (0)** — legacy raw bit-array, `bit n set ⇔ host n in set`. Only + read, never written anymore. Old rows decode transparently and age out via + retention. +- **`EncodingRoaring` (1)** — portable [RoaringBitmap](https://github.com/RoaringBitmap/roaring) + serialization. **All new writes use this.** + +Two in-memory representations, and the distinction matters: + +- **`Blob{Bytes, Encoding}` — storage form.** Only at the DB I/O boundary. Built by + `HostIDsToBlob` / `BitmapToBlob`; consumed by INSERT/UPDATE. +- **`*roaring.Bitmap` — op form.** Everything else: all set math + (`BlobAND/OR/ANDNOT`, `BlobPopcount`) and change detection. Built by `NewBitmap` + or `DecodeBitmap`. + +Encoding-awareness is confined to `DecodeBitmap` (storage→op) and `BitmapToBlob` +(op→storage). The rest of the code works in op form and never thinks about bytes. +Change detection compares op-form bitmaps with `roaring.Equals` — never bytes, +because a dense row and a roaring row can represent the same set with different +bytes. + +## Sample strategies + +A dataset declares a `SampleStrategy` that governs how observations combine within +a write-bucket and how rows collapse across buckets. **All collectors write at 1h +granularity** regardless of the *display* resolution requested at read time. + +### `SampleStrategyAccumulate` + +*"hosts observed doing the thing at any point during the bucket."* Used by +**uptime**. + +- **Write:** every row is born *closed* (`valid_to = bucketStart + bucketSize` at + insert). Repeated samples within the same bucket OR-merge into the existing row + (ODKU on `uniq_entity_bucket`). A sample in a new bucket starts a fresh row. No + explicit close step, no cross-bucket collapse. +- **Read:** a bucket's value = OR of every row whose interval overlaps the bucket. + +### `SampleStrategySnapshot` + +*"state as of the end of the bucket."* Used by **cve**. + +- **Write:** rows align to 1h boundaries. The latest sample in a write-bucket + overwrites via ODKU (last-write-wins). Across buckets, **unchanged** state keeps + the row open (`valid_to` stays sentinel); a **changed** sample closes the prior + row at the new boundary and opens a new one. An entity that disappears from the + input has its open row closed. +- **Read:** for each entity, pick the row active at `bucketEnd`, then OR across + entities. + +> **Snapshot collectors must call `RecordBucketData` even with an empty map.** An +> empty input is meaningful — it means "no entities are in the tracked state right +> now," which must close any still-open rows. Accumulate short-circuits on empty +> (nothing to merge, no state to reconcile). See the comments in `datasets.go` +> (`CVEDataset.Collect`) and `data.go` (`RecordBucketData`). + +The read-side aggregation for both strategies lives in `aggregateBucket` +(`internal/mysql/data.go`). + +## The write path (collection) + +1. A cron (`newChartDataCollectionSchedule` in `cmd/fleet/cron.go`, default 1h) + calls `Service.CollectDatasets(ctx, now, scope)`. +2. `scope` is a `CollectScopeFn` built fresh each tick from AppConfig + team + configs (`buildChartScopeResolver`). For each dataset it returns `(skip, + disabledFleetIDs)` — whether the dataset is globally off, and which fleets opted + out. +3. For each registered dataset, `Collect(ctx, store, now, disabledFleetIDs)` runs. + A failure is logged and the loop continues — one dataset can't block the others. +4. `Collect` reads host state through the narrow `DatasetStore` interface + (`FindOnlineHostIDs`, `AffectedHostIDsByCVE`, `TrackedCriticalCVEs`), builds + `map[entityID]*roaring.Bitmap`, and calls `store.RecordBucketData(...)` with its + strategy. +5. `RecordBucketData` dispatches to `recordAccumulate` or `recordSnapshot`, which + serialize via `BitmapToBlob` and upsert. + +Retention: a separate cleanup cron calls `CleanupData(days)` → +`CleanupSCDData`, which deletes *closed* rows older than the cutoff in batches. +Open rows (sentinel `valid_to`) are never deleted. + +## The read path (`GetChartData`) + +`internal/service/service.go::GetChartData` is the heart of the read side: + +1. **Resolve scope.** `ViewerProvider.ViewerScope(ctx)` returns `(isGlobal, + teamIDs)`. Fails closed if there's no viewer (requests sit behind authenticated + middleware; absence means misconfiguration). +2. **Authorize.** Explicit `team_id` → `Host{TeamID}` + `ActionRead` (Rego enforces + team-role match). No `team_id` → `Host{}` + `ActionList` (global users pass; + team users are scoped by data below). +3. **Validate** metric exists, `1 ≤ days ≤ 31`, resolution is 0 or a positive + divisor of 24. +4. **Build the filter mask.** `effectiveTeamIDs` collapses the team scope, then + `GetHostIDsForFilter` resolves team/label/platform/include/exclude into a host-id + list → `NewBitmap`. This is memoized per canonicalized filter by `hostFilterCache` + (60s TTL, singleflight-collapsed). The mask encodes "currently visible hosts," + which incidentally drops hosts deleted since the SCD rows were written. +5. **Walk buckets.** `GetSCDData` selects every row overlapping the range, + decodes each once, then for each bucket: `aggregateBucket` (per strategy) → AND + the mask → `popcount` → one `DataPoint`. Zero buckets are emitted as `0`, not + omitted. +6. **Respond** with metric, visualization, `TotalHosts` (popcount of the mask), + resolution label, applied filters, and the data points. + +Bucket boundaries are aligned to the client's local time via `tz_offset` +(`computeBucketRange`), so an "hourly" or "daily" chart lines up with the user's +day. + +### Key invariant: nil vs empty + +Several layers depend on the difference between a `nil` slice and an empty non-nil +slice. **Do not "normalize" one to the other.** + +- `HostFilter.TeamIDs`: `nil` = no team filter (all hosts); `[]uint{}` = team user + with zero teams → SQL emits `1=0` (see nothing); `[]uint{0}` = no-team hosts + (`team_id IS NULL`). +- `GetSCDData` `entityIDs`: `nil` = match every entity; `[]uint{}` non-nil = match + nothing (zero-valued buckets), avoiding an `IN ()` syntax error. +- `TrackedCriticalCVEs` returns a non-nil empty slice when nothing matches so the + caller can tell "filter resolved to empty" from "no filter." + +## Scoping, config gating, and scrubbing + +Whether a dataset collects at all is gated by `HistoricalDataSettings` in AppConfig +(global) and per-team config (`Features.HistoricalData`). The cron's scope resolver +translates these into `skip` / `disabledFleetIDs`. + +When an admin **disables** a dataset, already-collected data must be removed. That +flip is handled by `fleet.OnHistoricalDataChanged` (in `server/fleet/historical_data.go`), +which enqueues a worker job *after* the config commit: + +- **Global disable** → `chart_scrub_dataset_global` job → + `Service.ScrubDatasetGlobal` → `DeleteAllForDataset` (batched delete of all rows + for the dataset). +- **Per-fleet disable** → `chart_scrub_dataset_fleet` job → + `Service.ScrubDatasetFleet` → resolve the fleets' host IDs into a mask, then + `ApplyScrubMaskToDataset` walks every row and `BlobANDNOT`s the mask out. Both are + idempotent. + +The worker jobs live in `server/worker/chart_scrub.go`. Note the dataset-name +strings (`"uptime"`, `"cve"`) are mirrored in three places — the `Dataset.Name()` +return, the scrub job payloads, and `OnHistoricalDataChanged`'s change list. They +must stay in sync, since `host_scd_data.dataset` is the join key for all of it. + +## How to add a new dataset + +Worked example: a "battery health" dataset showing how many hosts had a healthy +battery. + +1. **Implement `api.Dataset`** in `server/chart/datasets.go`: + + ```go + type BatteryDataset struct{} + + func (b *BatteryDataset) Name() string { return "battery" } + func (b *BatteryDataset) DefaultResolutionHours() int { return 24 } + func (b *BatteryDataset) SampleStrategy() api.SampleStrategy { return api.SampleStrategySnapshot } + func (b *BatteryDataset) DefaultVisualization() string { return "line" } + + func (b *BatteryDataset) Collect(ctx context.Context, store api.DatasetStore, now time.Time, disabledFleetIDs []uint) error { + hostIDs, err := store.FindHealthyBatteryHostIDs(ctx, disabledFleetIDs) // new store method + if err != nil { + return err + } + bucketStart := now.UTC().Truncate(time.Hour) + // Snapshot: always record, even when empty, so open rows close. + return store.RecordBucketData(ctx, b.Name(), bucketStart, time.Hour, b.SampleStrategy(), + map[string]*roaring.Bitmap{"": chart.NewBitmap(hostIDs)}) + } + ``` + + Pick the strategy deliberately: *accumulate* for "seen doing X at any point" + (uptime-like), *snapshot* for "in state X as of now" (inventory-like). Use a + non-empty `entity_id` only if you need a sub-dimension you'll OR across (like + per-CVE). + +2. **Add the collection query** to the store. Define the method on **both** + `api.DatasetStore` (`api/chart.go`) and `internal/types.Datastore` + (`internal/types/chart.go`) — `Collect` only sees `DatasetStore`, but the + concrete MySQL type must satisfy `types.Datastore` — then implement it in + `internal/mysql/charts.go`. Keep these read-only and bounded; stream large joins + (see `streamCVEHostPairs`). + +3. **Register the dataset** in `cmd/fleet/serve.go::createChartBoundedContext`: + + ```go + chartSvc.RegisterDataset(&chart.BatteryDataset{}) + ``` + + An unregistered metric returns a 400 from `GetChartData`. + +4. **Wire config gating** (if the dataset is opt-in/out): add a sub-key to + `HistoricalDataSettings`, teach `Enabled(name)` about it, and add a row to the + change list in `fleet.OnHistoricalDataChanged` so disabling it triggers a scrub. + The scrub workers are dataset-agnostic and need no changes. + +5. **Test** the collector and the read aggregation. MySQL-backed tests live in + `internal/mysql/*_test.go`; service-level behavior in `internal/service/*_test.go`. + +No migration is needed — new datasets reuse `host_scd_data`, keyed by the new +`dataset` string. A migration is only needed if you change the table shape (e.g. a +new column or index). + +## How charts reach the frontend + +- Route: `GET /api/_version_/fleet/charts/{metric}` (registered in + `internal/service/handler.go`). +- Query params: `days`, `resolution` (hours), `tz_offset` (minutes, from JS + `Date.getTimezoneOffset()`), `fleet_id` (note the teams→fleets rename — the wire + name is `fleet_id`, the Go field stays `TeamID`), `label_ids`, `platforms`, + `include_host_ids`, `exclude_host_ids` (comma lists). +- The response carries `visualization` (from `DefaultVisualization()`), so the + frontend learns how to render each metric from the backend rather than hardcoding + it. + +## Testing + +```bash +# Fast, no external deps (bitmap helpers, arch test): +go test ./server/chart/... + +# MySQL-backed datastore + service tests: +MYSQL_TEST=1 go test ./server/chart/... + +# A single test: +MYSQL_TEST=1 go test -run TestName ./server/chart/internal/mysql/... +``` + +`arch_test.go` is part of `go test ./server/chart/...` and will fail the build if a +package grows a forbidden dependency (e.g. an accidental `server/fleet` import). +When you add a store method, run `go test ./server/service/` too — uninitialized +mocks elsewhere can crash if an interface method is missing. + +## File reference + +| File | What's in it | +|------|--------------| +| `blob.go` | Bitmap encode/decode, storage-form vs op-form, set ops | +| `datasets.go` | `UptimeDataset`, `CVEDataset` — the `Dataset` implementations | +| `api/service.go` | `Service`, `ViewerProvider`, `CollectScopeFn` | +| `api/chart.go` | `Dataset`, `DatasetStore`, `SampleStrategy`, request/response types | +| `api/http/types.go` | HTTP wire DTOs | +| `internal/types/chart.go` | `Datastore` interface, `HostFilter` (nil/empty semantics) | +| `internal/service/service.go` | `GetChartData`, scope/authz, scrub, bucket range math | +| `internal/service/host_cache.go` | Per-filter mask cache (TTL + singleflight) | +| `internal/service/handler.go` | Route registration + endpoint decode | +| `internal/mysql/data.go` | SCD read/write: `RecordBucketData`, `GetSCDData`, cleanup, scrub | +| `internal/mysql/charts.go` | Host-filter SQL, online-host query, CVE collection + tracked-CVE filter | +| `bootstrap/bootstrap.go` | `New(...)` — wires the context together | +| `arch_test.go` | Enforces the dependency rules above | + +Related code outside this tree: + +- `server/acl/chartacl/` — anti-corruption layer (viewer adapter). +- `cmd/fleet/serve.go` — `createChartBoundedContext`, dataset registration. +- `cmd/fleet/cron.go` — collection schedule, scope resolver, cleanup. +- `server/worker/chart_scrub.go` — scrub worker jobs. +- `server/fleet/historical_data.go` — config-flip → scrub/activity orchestration. +- `server/datastore/mysql/migrations/tables/20260423161823_AddHostSCDData.go` — the table migration. +- `tools/charts-backfill/`, `tools/charts-collect/` — dev tools (see below). + +## Dev tools: populating chart data + +In dev you usually don't have hours of real collection history to chart. Two +standalone tools under `tools/` write directly to `host_scd_data` so you have +something to render. Each has its own README with the full flag reference; this is +the orientation. + +### `tools/charts-backfill` — synthetic history + +Generates **fake but realistically-shaped** history for a dataset. This is the one +you want for frontend work or eyeballing a chart — point it at your local DB and it +fabricates days of data in seconds. Safe to re-run (ODKU merge), always writes +roaring encoding. + +Crucially, it backfills in the mode that **matches the dataset's +`SampleStrategy`** so the data looks like what production would eventually produce: + +- **Accumulate datasets** (uptime) → 24 independent hourly rows per day, + each a fresh random sample, each bounded to its single hour. +- **Snapshot datasets** (cve) → per-entity state-segment rows: most entities get one + long open row, a small fraction "flip" on day boundaries to produce closed + segments. CVE cardinality follows a long-tail distribution (most CVEs touch a + handful of hosts, a few are browser/kernel-wide) so unioning hundreds of entities + doesn't saturate at fleet size. The final segment per entity stays open + (`valid_to` = sentinel) so the live collector compares against it on its next + tick rather than stacking a row on top. + +```bash +# 30 days of uptime for all hosts in the local DB: +go run ./tools/charts-backfill --dataset uptime --days 30 + +# CVE data using the same CVE set production would track: +go run ./tools/charts-backfill --dataset cve --days 30 --use-tracked-cves + +# Scope to specific hosts / entities / a custom DSN: +go run ./tools/charts-backfill --dataset uptime --days 7 --host-ids 1,2,3 +go run ./tools/charts-backfill --dataset cve --days 30 --entity-ids CVE-2024-1,CVE-2024-2 +go run ./tools/charts-backfill --mysql-dsn "fleet:fleet@tcp(localhost:3306)/fleet" +``` + +Key flags: `--dataset` (default `uptime`), `--days` (30), `--start-date` +(`YYYY-MM-DD`, defaults to `now - days`), `--host-ids` (default: all hosts), +`--entity-ids`, `--use-tracked-cves` (cve only — auto-discovers entity IDs via the +production tracked-CVE query; needs vuln data populated), `--mysql-dsn`. Full table +in `tools/charts-backfill/README.md`. + +If you add a snapshot-strategy dataset, add its name to `snapshotDatasets` in +`tools/charts-backfill/main.go` (and a density range in `densityRange`) so the tool +generates it in the right shape; otherwise it defaults to the accumulate/hourly +model. + +### `tools/charts-collect` — real data from a live Fleet + +The other tool pulls **real** state from a running Fleet instance over the REST API +(currently uptime + CVE) and writes it into a local DB. It's the out-of-process +stand-in for the in-server collection cron — designed to run hourly against e.g. +dogfood. Use this when you want a chart backed by real fleet data rather than +synthetic noise. + +```bash +go run ./tools/charts-collect --fleet-url https://dogfood.fleetdm.com --fleet-token +``` + +Targets are also configurable via `FLEET_URL` / `FLEET_TOKEN` / `MYSQL_DSN`, and +the DSN falls back to the standard `FLEET_MYSQL_*` env vars. See +`tools/charts-collect/README.md`. + +> Both tools duplicate a few storage constants (the `9999-12-31` open sentinel, +> upsert batch size, roaring encoding) because they write `host_scd_data` +> out-of-process and can't import the internal mysql package. If you change the +> storage format or those constants in `internal/mysql/data.go`, update the tools to +> match. diff --git a/tools/charts-collect/README.md b/tools/charts-collect/README.md index ad6c46f62b..234cb99e3c 100644 --- a/tools/charts-collect/README.md +++ b/tools/charts-collect/README.md @@ -6,7 +6,7 @@ into a local database. Designed to run hourly via cron. ## What it collects - **Uptime** — fetches currently online hosts and ORs them into the current - hour's `host_hourly_data_blobs` row (`dataset='uptime'`). + hour's `host_scd_data` row (`dataset='uptime'`). - **CVE** — fetches per-host vulnerabilities, inverts into per-CVE host bitmaps, and reconciles into `host_scd_data` (`dataset='cve'`). Unchanged CVEs keep their open row; changed bitmaps close the prior-day row and open @@ -45,5 +45,5 @@ Render `fromService`): ## Notes - SCD encoding constants (`9999-12-31` open sentinel, batch size) mirror - `server/chart/internal/mysql/scd.go`. Keep in sync when either side changes. + `server/chart/internal/mysql/data.go`. Keep in sync when either side changes. - Errors in one collector (uptime/cve) are logged but do not block the other.