Files
fleet/server/datastore/mysqlredis/metrics.go
T
Victor Lyuboslavsky de86536f42 Redis-backed cache for host-by-key lookups (#43936)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #43928 

This PR adds a Redis-backed cache in front of the two host-by-key
lookups on the agent auth paths.

Docs: https://github.com/fleetdm/fleet/pull/44504

## What changes

**Read path (osquery/orbit auth):**

- `LoadHostByNodeKey` and `LoadHostByOrbitNodeKey` now check Redis
before falling through to MySQL.
- Successful lookups are cached for 60s ± 10% jitter (configurable via
`FLEET_REDIS_HOST_CACHE_TTL`).
- `NotFound` results are cached for 5s as a negative entry, dampening
repeated probes for keys that
do not exist (deleted hosts whose agents are still polling, attacker
scans, retry storms).
- Concurrent lookups for the same key collapse into one DB query via
`singleflight`. The shared
query runs under a context detached from any one caller's deadline so
the leader giving up does
not abort the work for joiners. The shared query is itself bounded by a
30s timeout so a wedged
  DB call cannot pin the singleflight slot indefinitely.

**Write path (invalidations):**

- These methods now invalidate the cache after a successful inner call:
`UpdateHost`, `SerialUpdateHost`, `UpdateHostOsqueryIntervals`,
`UpdateHostRefetchRequested`,
`UpdateHostRefetchCriticalQueriesUntil`,
`UpdateHostIdentityCertHostIDBySerial`, `EnrollOsquery`,
`EnrollOrbit`, `NewHost`, `DeleteHost`, `DeleteHosts`,
`CleanupExpiredHosts`,
  `CleanupIncomingHosts`, `AddHostsToTeam`.
- `AddHostsToTeam`, `DeleteHosts`, `CleanupExpiredHosts`, and
`CleanupIncomingHosts` use a pipelined
batch invalidator so 10k-host operations stay in the millisecond range
instead of taking minutes
  of sequential round-trips.
- Inner-call errors are not invalidations: a failing write leaves cached
state intact.

**Configuration:**

- New flags `FLEET_REDIS_HOST_CACHE_ENABLED` (default `true`) and
`FLEET_REDIS_HOST_CACHE_TTL`
  (default `60s`).
- Server refuses to start if the cache is enabled with `TTL <= 0`.

**Observability:**

- Three new OTEL counters under the `fleet` meter:
  - `fleet.host_cache.lookups{result=hit|negative_hit|miss}`
  - `fleet.host_cache.errors{op=get|set|del}`
-
`fleet.host_cache.invalidations{reason=update|enroll|team|delete|cert}`
- A pre-built SigNoz dashboard ships in
`tools/signoz/host_cache_dashboard.json`.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional Redis-backed host lookup cache for osquery and orbit auth,
with automatic invalidation and metrics/monitoring dashboard.

* **Bug Fixes**
* Fixed host-removal batching so cache-related removals use correct
chunks.

* **Tests**
* Added comprehensive host-cache unit tests covering hits, negative
cache, invalidation, concurrency, and JSON round-trips.

* **Chores**
* New config flags to enable the cache and set TTL (default 60s ±10%
jitter).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-01 12:06:16 -05:00

73 lines
2.2 KiB
Go

package mysqlredis
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
)
// OpenTelemetry instruments for the host lookup cache (covers both
// LoadHostByNodeKey and LoadHostByOrbitNodeKey). These are unused until the
// cache is wired up via WithHostCache. When the global MeterProvider is not
// configured (tests, services started without OTEL), otel.Meter returns a
// no-op meter and all operations on these counters silently succeed.
var (
meter = otel.Meter("fleet")
// hostCacheLookups counts cache read attempts, labeled by result.
// Attribute `result` is one of: hit, negative_hit, miss.
hostCacheLookups metric.Int64Counter
// hostCacheErrors counts Redis/JSON errors encountered in the cache path,
// labeled by operation. Attribute `op` is one of: get, set, del.
hostCacheErrors metric.Int64Counter
// hostCacheInvalidations counts cache invalidation operations, labeled by
// the write path that triggered the invalidation. Attribute `reason` is
// one of: update, enroll, team, delete, cert.
hostCacheInvalidations metric.Int64Counter
)
func init() {
var err error
hostCacheLookups, err = meter.Int64Counter(
"fleet.host_cache.lookups",
metric.WithDescription("Host lookup cache reads, labeled by result"),
metric.WithUnit("{event}"),
)
if err != nil {
panic(err)
}
hostCacheErrors, err = meter.Int64Counter(
"fleet.host_cache.errors",
metric.WithDescription("Host lookup cache Redis/serialization errors, labeled by operation"),
metric.WithUnit("{event}"),
)
if err != nil {
panic(err)
}
hostCacheInvalidations, err = meter.Int64Counter(
"fleet.host_cache.invalidations",
metric.WithDescription("Host lookup cache invalidations, labeled by the write path reason"),
metric.WithUnit("{event}"),
)
if err != nil {
panic(err)
}
}
func hostCacheLookupAttrs(result string) metric.AddOption {
return metric.WithAttributes(attribute.String("result", result))
}
func hostCacheErrorAttrs(op string) metric.AddOption {
return metric.WithAttributes(attribute.String("op", op))
}
func hostCacheInvalidationAttrs(reason string) metric.AddOption {
return metric.WithAttributes(attribute.String("reason", reason))
}