<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43928 This PR adds a Redis-backed cache in front of the two host-by-key lookups on the agent auth paths. Docs: https://github.com/fleetdm/fleet/pull/44504 ## What changes **Read path (osquery/orbit auth):** - `LoadHostByNodeKey` and `LoadHostByOrbitNodeKey` now check Redis before falling through to MySQL. - Successful lookups are cached for 60s ± 10% jitter (configurable via `FLEET_REDIS_HOST_CACHE_TTL`). - `NotFound` results are cached for 5s as a negative entry, dampening repeated probes for keys that do not exist (deleted hosts whose agents are still polling, attacker scans, retry storms). - Concurrent lookups for the same key collapse into one DB query via `singleflight`. The shared query runs under a context detached from any one caller's deadline so the leader giving up does not abort the work for joiners. The shared query is itself bounded by a 30s timeout so a wedged DB call cannot pin the singleflight slot indefinitely. **Write path (invalidations):** - These methods now invalidate the cache after a successful inner call: `UpdateHost`, `SerialUpdateHost`, `UpdateHostOsqueryIntervals`, `UpdateHostRefetchRequested`, `UpdateHostRefetchCriticalQueriesUntil`, `UpdateHostIdentityCertHostIDBySerial`, `EnrollOsquery`, `EnrollOrbit`, `NewHost`, `DeleteHost`, `DeleteHosts`, `CleanupExpiredHosts`, `CleanupIncomingHosts`, `AddHostsToTeam`. - `AddHostsToTeam`, `DeleteHosts`, `CleanupExpiredHosts`, and `CleanupIncomingHosts` use a pipelined batch invalidator so 10k-host operations stay in the millisecond range instead of taking minutes of sequential round-trips. - Inner-call errors are not invalidations: a failing write leaves cached state intact. **Configuration:** - New flags `FLEET_REDIS_HOST_CACHE_ENABLED` (default `true`) and `FLEET_REDIS_HOST_CACHE_TTL` (default `60s`). - Server refuses to start if the cache is enabled with `TTL <= 0`. **Observability:** - Three new OTEL counters under the `fleet` meter: - `fleet.host_cache.lookups{result=hit|negative_hit|miss}` - `fleet.host_cache.errors{op=get|set|del}` - `fleet.host_cache.invalidations{reason=update|enroll|team|delete|cert}` - A pre-built SigNoz dashboard ships in `tools/signoz/host_cache_dashboard.json`. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Optional Redis-backed host lookup cache for osquery and orbit auth, with automatic invalidation and metrics/monitoring dashboard. * **Bug Fixes** * Fixed host-removal batching so cache-related removals use correct chunks. * **Tests** * Added comprehensive host-cache unit tests covering hits, negative cache, invalidation, concurrency, and JSON round-trips. * **Chores** * New config flags to enable the cache and set TTL (default 60s ±10% jitter). <!-- end of auto-generated comment: release notes by coderabbit.ai -->
73 lines
2.2 KiB
Go
73 lines
2.2 KiB
Go
package mysqlredis
|
|
|
|
import (
|
|
"go.opentelemetry.io/otel"
|
|
"go.opentelemetry.io/otel/attribute"
|
|
"go.opentelemetry.io/otel/metric"
|
|
)
|
|
|
|
// OpenTelemetry instruments for the host lookup cache (covers both
|
|
// LoadHostByNodeKey and LoadHostByOrbitNodeKey). These are unused until the
|
|
// cache is wired up via WithHostCache. When the global MeterProvider is not
|
|
// configured (tests, services started without OTEL), otel.Meter returns a
|
|
// no-op meter and all operations on these counters silently succeed.
|
|
|
|
var (
|
|
meter = otel.Meter("fleet")
|
|
|
|
// hostCacheLookups counts cache read attempts, labeled by result.
|
|
// Attribute `result` is one of: hit, negative_hit, miss.
|
|
hostCacheLookups metric.Int64Counter
|
|
|
|
// hostCacheErrors counts Redis/JSON errors encountered in the cache path,
|
|
// labeled by operation. Attribute `op` is one of: get, set, del.
|
|
hostCacheErrors metric.Int64Counter
|
|
|
|
// hostCacheInvalidations counts cache invalidation operations, labeled by
|
|
// the write path that triggered the invalidation. Attribute `reason` is
|
|
// one of: update, enroll, team, delete, cert.
|
|
hostCacheInvalidations metric.Int64Counter
|
|
)
|
|
|
|
func init() {
|
|
var err error
|
|
hostCacheLookups, err = meter.Int64Counter(
|
|
"fleet.host_cache.lookups",
|
|
metric.WithDescription("Host lookup cache reads, labeled by result"),
|
|
metric.WithUnit("{event}"),
|
|
)
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
|
|
hostCacheErrors, err = meter.Int64Counter(
|
|
"fleet.host_cache.errors",
|
|
metric.WithDescription("Host lookup cache Redis/serialization errors, labeled by operation"),
|
|
metric.WithUnit("{event}"),
|
|
)
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
|
|
hostCacheInvalidations, err = meter.Int64Counter(
|
|
"fleet.host_cache.invalidations",
|
|
metric.WithDescription("Host lookup cache invalidations, labeled by the write path reason"),
|
|
metric.WithUnit("{event}"),
|
|
)
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
}
|
|
|
|
func hostCacheLookupAttrs(result string) metric.AddOption {
|
|
return metric.WithAttributes(attribute.String("result", result))
|
|
}
|
|
|
|
func hostCacheErrorAttrs(op string) metric.AddOption {
|
|
return metric.WithAttributes(attribute.String("op", op))
|
|
}
|
|
|
|
func hostCacheInvalidationAttrs(reason string) metric.AddOption {
|
|
return metric.WithAttributes(attribute.String("reason", reason))
|
|
}
|