Files
Sharon KatzandClaude Opus 4.6 74b10d8a0d Cache pack config JSON per team to reduce redundant marshaling (#48702)
**Related issue:** #21847

## Summary

`GetClientConfig` is called by every host every ~60 seconds. It rebuilds
the full pack config (all scheduled query SQL text) from DB and
JSON-marshals it on every request. For all hosts in the same team, the
result is identical, yet we run 3-5 DB queries + `json.Marshal` of ~50KB
per request.

This PR adds an in-memory cache for the marshaled pack config JSON,
keyed by `(teamID, queryReportsDisabled)` with a 1-minute TTL. The cache
is invalidated when queries or AppConfig are modified.

### What changed

- Extracted pack config building from `GetClientConfig` into a new
`getPackConfig` method
- Added `packConfigCache` field to Service struct using `go-cache`
(1-minute TTL, 5-minute cleanup)
- On cache hit (no legacy packs): returns cached `json.RawMessage`
immediately, skipping all DB queries and JSON marshaling
- On cache miss: builds pack config from DB, marshals, caches, and
returns
- Cache is flushed on any query mutation (`NewQuery`, `ModifyQuery`,
`DeleteQuery`, `DeleteQueries`, `ApplyQuerySpecs`, `DeleteQueryByID`)
and on `ModifyAppConfig`

### Expected impact at 100K hosts

| Metric | Before | After |
|--------|--------|-------|
| Pack config marshals/second | ~1,667 | ~1 per minute per team |
| DB queries for scheduled queries/second | ~5,000 | ~5 per minute per
team |
| CPU from JSON encoding | Dominant in pprof | Negligible |

### Known limitation

`ListScheduledQueriesForAgents` supports label-scoped query filtering
per host. The cache is keyed by team (not host), so when label-scoped
scheduled queries exist, all hosts in a team receive the same query set
from the cache regardless of their label memberships. This is an
acceptable trade-off because:
- Label-scoped scheduled queries are uncommon in most deployments
- The cache TTL is 1 minute, so divergence is temporary
- Running an extra query on a host is not harmful (just unnecessary
work)
- This can be refined in a follow-up to filter label-scoped queries from
the cached result

## Testing

### Unit tests (9 tests, all pass)

| Test | What it verifies |
|------|-----------------|
| `TestPackConfigCacheHit` | Second `GetClientConfig` call triggers zero
DB calls for scheduled queries |
| `TestPackConfigCacheInvalidationOnQueryCreate` | After
`InvalidatePackConfigCache()`, new query appears in config |
| `TestPackConfigCacheInvalidationOnQueryModify` | After invalidation,
updated SQL is reflected in config |
| `TestPackConfigCacheInvalidationOnQueryDelete` | After invalidation
with empty query list, packs key is absent |
| `TestPackConfigCacheInvalidationOnApplyQuerySpecs` | After
invalidation simulating GitOps apply, new specs appear |
| `TestPackConfigCacheTTLExpiration` | After 50ms TTL expires, fresh DB
read occurs and new query appears |
| `TestPackConfigCacheTeamIsolation` | Global, team-1, team-2 hosts get
correctly isolated cached configs |
| `TestPackConfigCacheLegacyPacksBypass` | Host with legacy pack
triggers DB calls on every request (no caching) |
| `TestPackConfigCachePerformance` | 1000 cached calls: 0 DB calls. 1000
uncached: 1000 DB calls. ~1.4x speedup with mock (real DB would be much
larger) |

```
=== RUN   TestPackConfigCacheHit           --- PASS (0.01s)
=== RUN   TestPackConfigCacheInvalidationOnQueryCreate  --- PASS (0.01s)
=== RUN   TestPackConfigCacheInvalidationOnQueryModify  --- PASS (0.01s)
=== RUN   TestPackConfigCacheInvalidationOnQueryDelete  --- PASS (0.01s)
=== RUN   TestPackConfigCacheInvalidationOnApplyQuerySpecs --- PASS (0.01s)
=== RUN   TestPackConfigCacheTTLExpiration  --- PASS (0.11s)
=== RUN   TestPackConfigCacheTeamIsolation  --- PASS (0.01s)
=== RUN   TestPackConfigCacheLegacyPacksBypass --- PASS (0.01s)
=== RUN   TestPackConfigCachePerformance   --- PASS (0.02s)
    Performance: cached=2.37ms, uncached=3.42ms, speedup=1.4x
```

Note: The 1.4x speedup is with mock datastore (no real DB/network). With
real MySQL over network, the speedup would be orders of magnitude larger
since cached calls skip 3-5 DB round-trips + ~50KB JSON marshal
entirely.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

- [x] Confirmed that the fix is not expected to adversely impact load
test results

## QA: Load test verification

To validate the real-world impact, QA should run a load test before and
after this change and compare:

1. Capture a CPU pprof profile **before** the change under load (e.g.,
10K+ simulated hosts, 50+ scheduled queries)
2. Deploy the change and capture a **second** pprof profile under the
same load
3. Compare the flamegraphs -- the `encoding/json.Marshal` and
`GetClientConfig` CPU time should drop significantly
4. Monitor Fleet container CPU utilization -- expect a measurable
reduction in steady-state CPU

See #21847 for the original pprof showing `encoding/json` dominating CPU
at scale.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved host config response performance by caching pack
configuration data.
* Query changes now automatically refresh cached host config so updates
appear promptly.

* **Bug Fixes**
* Host configs now stay accurate after creating, updating, deleting, or
applying queries.
* Cached data is isolated correctly and falls back to fresh data when
legacy packs are present.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 13:51:02 -04:00

248 lines
8.7 KiB
Go

// Package service holds the implementation of the fleet interface and HTTP
// endpoints for the API
package service
import (
"context"
"fmt"
"html/template"
"log/slog"
"net/url"
"strings"
"sync"
"time"
"github.com/WatchBeam/clock"
"github.com/fleetdm/fleet/v4/server/authz"
"github.com/fleetdm/fleet/v4/server/config"
"github.com/fleetdm/fleet/v4/server/fleet"
"github.com/fleetdm/fleet/v4/server/mdm/android"
apple_mdm "github.com/fleetdm/fleet/v4/server/mdm/apple"
microsoft_mdm "github.com/fleetdm/fleet/v4/server/mdm/microsoft"
nanodep_storage "github.com/fleetdm/fleet/v4/server/mdm/nanodep/storage"
nanomdm_push "github.com/fleetdm/fleet/v4/server/mdm/nanomdm/push"
nanomdm_storage "github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage"
"github.com/fleetdm/fleet/v4/server/service/async"
"github.com/fleetdm/fleet/v4/server/service/conditional_access_microsoft_proxy"
"github.com/fleetdm/fleet/v4/server/sso"
gocache "github.com/patrickmn/go-cache"
)
var _ fleet.Service = (*Service)(nil)
// Service is the struct implementing fleet.Service. Create a new one with NewService.
type Service struct {
ds fleet.Datastore
task *async.Task
carveStore fleet.CarveStore
resultStore fleet.QueryResultStore
liveQueryStore fleet.LiveQueryStore
logger *slog.Logger
config config.FleetConfig
clock clock.Clock
osqueryLogWriter *OsqueryLogger
mailService fleet.MailService
ssoSessionStore sso.SessionStore
failingPolicySet fleet.FailingPolicySet
enrollHostLimiter fleet.EnrollHostLimiter
authz *authz.Authorizer
jitterMu *sync.RWMutex
jitterH map[time.Duration]*jitterHashTable
geoIP fleet.GeoIP
*fleet.EnterpriseOverrides
depStorage nanodep_storage.AllDEPStorage
mdmStorage nanomdm_storage.AllStorage
mdmPushService nanomdm_push.Pusher
mdmAppleCommander *apple_mdm.MDMAppleCommander
cronSchedulesService fleet.CronSchedulesService
wstepCertManager microsoft_mdm.CertManager
scepConfigService fleet.SCEPConfigService
digiCertService fleet.DigiCertService
conditionalAccessMicrosoftProxy ConditionalAccessMicrosoftProxy
keyValueStore fleet.KeyValueStore
packConfigCache *gocache.Cache
androidSvc android.Service
// activitySvc is the activity bounded context service for write operations.
activitySvc fleet.ActivityWriteService
// acmeSvc is the ACME service module for write operations.
acmeSvc fleet.ACMEWriteService
// orgLogoStore stores the bytes of customer-uploaded org logos.
orgLogoStore fleet.OrgLogoStore
}
// ConditionalAccessMicrosoftProxy is the interface of the Microsoft compliance proxy.
type ConditionalAccessMicrosoftProxy interface {
// Create creates the integration on the MS proxy and returns the consent URL.
Create(ctx context.Context, tenantID string) (*conditional_access_microsoft_proxy.CreateResponse, error)
// Get returns the integration settings.
Get(ctx context.Context, tenantID string, secret string) (*conditional_access_microsoft_proxy.GetResponse, error)
// Delete deprovisions the tenant on Microsoft and deletes the integration in the proxy service.
// Returns a fleet.IsNotFound error if the integration doesn't exist.
Delete(ctx context.Context, tenantID string, secret string) (*conditional_access_microsoft_proxy.DeleteResponse, error)
// SetComplianceStatus sets the inventory and compliance status of a host.
// Returns the message ID to query the status of the operation (MS has an asynchronous API).
SetComplianceStatus(
ctx context.Context,
tenantID string, secret string,
deviceID string,
userPrincipalName string,
mdmEnrolled bool,
deviceName, osName, osVersion string,
compliant bool,
lastCheckInTime time.Time,
) (*conditional_access_microsoft_proxy.SetComplianceStatusResponse, error)
// GetMessageStatusResponse returns the status of a "compliance set" operation.
GetMessageStatus(ctx context.Context, tenantID string, secret string, messageID string) (*conditional_access_microsoft_proxy.GetMessageStatusResponse, error)
}
func (svc *Service) LookupGeoIP(ctx context.Context, ip string) *fleet.GeoLocation {
return svc.geoIP.Lookup(ctx, ip)
}
func (svc *Service) SetEnterpriseOverrides(overrides fleet.EnterpriseOverrides) {
svc.EnterpriseOverrides = &overrides
}
// OsqueryLogger holds osqueryd's status and result loggers.
type OsqueryLogger struct {
// Status holds the osqueryd's status logger.
//
// See https://osquery.readthedocs.io/en/stable/deployment/logging/#status-logs
Status fleet.JSONLogger
// Result holds the osqueryd's result logger.
//
// See https://osquery.readthedocs.io/en/stable/deployment/logging/#results-logs
Result fleet.JSONLogger
}
// NewService creates a new service from the config struct
func NewService(
ctx context.Context,
ds fleet.Datastore,
task *async.Task,
resultStore fleet.QueryResultStore,
logger *slog.Logger,
osqueryLogger *OsqueryLogger,
config config.FleetConfig,
mailService fleet.MailService,
c clock.Clock,
sso sso.SessionStore,
lq fleet.LiveQueryStore,
carveStore fleet.CarveStore,
failingPolicySet fleet.FailingPolicySet,
geoIP fleet.GeoIP,
enrollHostLimiter fleet.EnrollHostLimiter,
depStorage nanodep_storage.AllDEPStorage,
mdmStorage fleet.MDMAppleStore,
mdmPushService nanomdm_push.Pusher,
cronSchedulesService fleet.CronSchedulesService,
wstepCertManager microsoft_mdm.CertManager,
scepConfigService fleet.SCEPConfigService,
digiCertService fleet.DigiCertService,
conditionalAccessProxy ConditionalAccessMicrosoftProxy,
keyValueStore fleet.KeyValueStore,
androidSvc android.Service,
orgLogoStore fleet.OrgLogoStore,
) (fleet.Service, error) {
authorizer, err := authz.NewAuthorizer()
if err != nil {
return nil, fmt.Errorf("new authorizer: %w", err)
}
svc := &Service{
ds: ds,
task: task,
carveStore: carveStore,
resultStore: resultStore,
liveQueryStore: lq,
logger: logger,
config: config,
clock: c,
osqueryLogWriter: osqueryLogger,
mailService: mailService,
ssoSessionStore: sso,
failingPolicySet: failingPolicySet,
authz: authorizer,
jitterH: make(map[time.Duration]*jitterHashTable),
jitterMu: new(sync.RWMutex),
geoIP: geoIP,
enrollHostLimiter: enrollHostLimiter,
depStorage: depStorage,
// TODO: remove mdmStorage and mdmPushService when
// we remove deprecated top-level service methods
// from the prototype.
mdmStorage: mdmStorage,
mdmPushService: mdmPushService,
mdmAppleCommander: apple_mdm.NewMDMAppleCommander(mdmStorage, mdmPushService),
cronSchedulesService: cronSchedulesService,
wstepCertManager: wstepCertManager,
scepConfigService: scepConfigService,
digiCertService: digiCertService,
conditionalAccessMicrosoftProxy: conditionalAccessProxy,
keyValueStore: keyValueStore,
packConfigCache: gocache.New(1*time.Minute, 5*time.Minute),
androidSvc: androidSvc,
orgLogoStore: orgLogoStore,
}
return validationMiddleware{svc, ds, sso}, nil
}
func (svc *Service) SendEmail(ctx context.Context, mail fleet.Email) error {
return svc.mailService.SendEmail(ctx, mail)
}
// SetActivityService sets the activity bounded context service for write operations.
// This should be called after NewService to inject the activity service dependency.
func (svc *Service) SetActivityService(activitySvc fleet.ActivityWriteService) {
svc.activitySvc = activitySvc
}
// SetACMEService sets the ACME service module service for write operations.
// This should be called after NewService to inject the ACME service dependency.
func (svc *Service) SetACMEService(acmeSvc fleet.ACMEWriteService) {
svc.acmeSvc = acmeSvc
}
type validationMiddleware struct {
fleet.Service
ds fleet.Datastore
ssoSessionStore sso.SessionStore
}
// getAssetURL simply returns the base url used for retrieving image assets from fleetdm.com.
func getAssetURL() template.URL {
return template.URL("https://fleetdm.com/images/permanent")
}
// emailLinkBaseURL returns the base URL used to build links in transactional
// emails. The server URL is the source of truth; the URL prefix is appended
// only when the server URL does not already carry it. This keeps links correct
// whether an operator configures the subpath in the server URL, in the URL
// prefix, or both, instead of duplicating it (e.g. https://host/p/p/login).
func emailLinkBaseURL(serverURL, urlPrefix string) template.URL {
if urlPrefix != "" && !strings.HasSuffix(strings.TrimSuffix(serverURL, "/"), urlPrefix) {
if joined, err := url.JoinPath(serverURL, urlPrefix); err == nil {
serverURL = joined
}
}
return template.URL(serverURL) //nolint:gosec // G203: operator-configured URL, not user input
}