**Related issue:** #21847 ## Summary `GetClientConfig` is called by every host every ~60 seconds. It rebuilds the full pack config (all scheduled query SQL text) from DB and JSON-marshals it on every request. For all hosts in the same team, the result is identical, yet we run 3-5 DB queries + `json.Marshal` of ~50KB per request. This PR adds an in-memory cache for the marshaled pack config JSON, keyed by `(teamID, queryReportsDisabled)` with a 1-minute TTL. The cache is invalidated when queries or AppConfig are modified. ### What changed - Extracted pack config building from `GetClientConfig` into a new `getPackConfig` method - Added `packConfigCache` field to Service struct using `go-cache` (1-minute TTL, 5-minute cleanup) - On cache hit (no legacy packs): returns cached `json.RawMessage` immediately, skipping all DB queries and JSON marshaling - On cache miss: builds pack config from DB, marshals, caches, and returns - Cache is flushed on any query mutation (`NewQuery`, `ModifyQuery`, `DeleteQuery`, `DeleteQueries`, `ApplyQuerySpecs`, `DeleteQueryByID`) and on `ModifyAppConfig` ### Expected impact at 100K hosts | Metric | Before | After | |--------|--------|-------| | Pack config marshals/second | ~1,667 | ~1 per minute per team | | DB queries for scheduled queries/second | ~5,000 | ~5 per minute per team | | CPU from JSON encoding | Dominant in pprof | Negligible | ### Known limitation `ListScheduledQueriesForAgents` supports label-scoped query filtering per host. The cache is keyed by team (not host), so when label-scoped scheduled queries exist, all hosts in a team receive the same query set from the cache regardless of their label memberships. This is an acceptable trade-off because: - Label-scoped scheduled queries are uncommon in most deployments - The cache TTL is 1 minute, so divergence is temporary - Running an extra query on a host is not harmful (just unnecessary work) - This can be refined in a follow-up to filter label-scoped queries from the cached result ## Testing ### Unit tests (9 tests, all pass) | Test | What it verifies | |------|-----------------| | `TestPackConfigCacheHit` | Second `GetClientConfig` call triggers zero DB calls for scheduled queries | | `TestPackConfigCacheInvalidationOnQueryCreate` | After `InvalidatePackConfigCache()`, new query appears in config | | `TestPackConfigCacheInvalidationOnQueryModify` | After invalidation, updated SQL is reflected in config | | `TestPackConfigCacheInvalidationOnQueryDelete` | After invalidation with empty query list, packs key is absent | | `TestPackConfigCacheInvalidationOnApplyQuerySpecs` | After invalidation simulating GitOps apply, new specs appear | | `TestPackConfigCacheTTLExpiration` | After 50ms TTL expires, fresh DB read occurs and new query appears | | `TestPackConfigCacheTeamIsolation` | Global, team-1, team-2 hosts get correctly isolated cached configs | | `TestPackConfigCacheLegacyPacksBypass` | Host with legacy pack triggers DB calls on every request (no caching) | | `TestPackConfigCachePerformance` | 1000 cached calls: 0 DB calls. 1000 uncached: 1000 DB calls. ~1.4x speedup with mock (real DB would be much larger) | ``` === RUN TestPackConfigCacheHit --- PASS (0.01s) === RUN TestPackConfigCacheInvalidationOnQueryCreate --- PASS (0.01s) === RUN TestPackConfigCacheInvalidationOnQueryModify --- PASS (0.01s) === RUN TestPackConfigCacheInvalidationOnQueryDelete --- PASS (0.01s) === RUN TestPackConfigCacheInvalidationOnApplyQuerySpecs --- PASS (0.01s) === RUN TestPackConfigCacheTTLExpiration --- PASS (0.11s) === RUN TestPackConfigCacheTeamIsolation --- PASS (0.01s) === RUN TestPackConfigCacheLegacyPacksBypass --- PASS (0.01s) === RUN TestPackConfigCachePerformance --- PASS (0.02s) Performance: cached=2.37ms, uncached=3.42ms, speedup=1.4x ``` Note: The 1.4x speedup is with mock datastore (no real DB/network). With real MySQL over network, the speedup would be orders of magnitude larger since cached calls skip 3-5 DB round-trips + ~50KB JSON marshal entirely. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) ## Testing - [x] Added/updated automated tests - [x] QA'd all new/changed functionality manually - [x] Confirmed that the fix is not expected to adversely impact load test results ## QA: Load test verification To validate the real-world impact, QA should run a load test before and after this change and compare: 1. Capture a CPU pprof profile **before** the change under load (e.g., 10K+ simulated hosts, 50+ scheduled queries) 2. Deploy the change and capture a **second** pprof profile under the same load 3. Compare the flamegraphs -- the `encoding/json.Marshal` and `GetClientConfig` CPU time should drop significantly 4. Monitor Fleet container CPU utilization -- expect a measurable reduction in steady-state CPU See #21847 for the original pprof showing `encoding/json` dominating CPU at scale. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved host config response performance by caching pack configuration data. * Query changes now automatically refresh cached host config so updates appear promptly. * **Bug Fixes** * Host configs now stay accurate after creating, updating, deleting, or applying queries. * Cached data is isolated correctly and falls back to fresh data when legacy packs are present. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
248 lines
8.7 KiB
Go
248 lines
8.7 KiB
Go
// Package service holds the implementation of the fleet interface and HTTP
|
|
// endpoints for the API
|
|
package service
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"html/template"
|
|
"log/slog"
|
|
"net/url"
|
|
"strings"
|
|
"sync"
|
|
"time"
|
|
|
|
"github.com/WatchBeam/clock"
|
|
"github.com/fleetdm/fleet/v4/server/authz"
|
|
"github.com/fleetdm/fleet/v4/server/config"
|
|
"github.com/fleetdm/fleet/v4/server/fleet"
|
|
"github.com/fleetdm/fleet/v4/server/mdm/android"
|
|
apple_mdm "github.com/fleetdm/fleet/v4/server/mdm/apple"
|
|
microsoft_mdm "github.com/fleetdm/fleet/v4/server/mdm/microsoft"
|
|
nanodep_storage "github.com/fleetdm/fleet/v4/server/mdm/nanodep/storage"
|
|
nanomdm_push "github.com/fleetdm/fleet/v4/server/mdm/nanomdm/push"
|
|
nanomdm_storage "github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage"
|
|
"github.com/fleetdm/fleet/v4/server/service/async"
|
|
"github.com/fleetdm/fleet/v4/server/service/conditional_access_microsoft_proxy"
|
|
"github.com/fleetdm/fleet/v4/server/sso"
|
|
gocache "github.com/patrickmn/go-cache"
|
|
)
|
|
|
|
var _ fleet.Service = (*Service)(nil)
|
|
|
|
// Service is the struct implementing fleet.Service. Create a new one with NewService.
|
|
type Service struct {
|
|
ds fleet.Datastore
|
|
task *async.Task
|
|
carveStore fleet.CarveStore
|
|
resultStore fleet.QueryResultStore
|
|
liveQueryStore fleet.LiveQueryStore
|
|
logger *slog.Logger
|
|
config config.FleetConfig
|
|
clock clock.Clock
|
|
|
|
osqueryLogWriter *OsqueryLogger
|
|
|
|
mailService fleet.MailService
|
|
ssoSessionStore sso.SessionStore
|
|
|
|
failingPolicySet fleet.FailingPolicySet
|
|
enrollHostLimiter fleet.EnrollHostLimiter
|
|
|
|
authz *authz.Authorizer
|
|
|
|
jitterMu *sync.RWMutex
|
|
jitterH map[time.Duration]*jitterHashTable
|
|
|
|
geoIP fleet.GeoIP
|
|
|
|
*fleet.EnterpriseOverrides
|
|
|
|
depStorage nanodep_storage.AllDEPStorage
|
|
mdmStorage nanomdm_storage.AllStorage
|
|
mdmPushService nanomdm_push.Pusher
|
|
mdmAppleCommander *apple_mdm.MDMAppleCommander
|
|
|
|
cronSchedulesService fleet.CronSchedulesService
|
|
|
|
wstepCertManager microsoft_mdm.CertManager
|
|
scepConfigService fleet.SCEPConfigService
|
|
digiCertService fleet.DigiCertService
|
|
|
|
conditionalAccessMicrosoftProxy ConditionalAccessMicrosoftProxy
|
|
|
|
keyValueStore fleet.KeyValueStore
|
|
|
|
packConfigCache *gocache.Cache
|
|
|
|
androidSvc android.Service
|
|
|
|
// activitySvc is the activity bounded context service for write operations.
|
|
activitySvc fleet.ActivityWriteService
|
|
|
|
// acmeSvc is the ACME service module for write operations.
|
|
acmeSvc fleet.ACMEWriteService
|
|
|
|
// orgLogoStore stores the bytes of customer-uploaded org logos.
|
|
orgLogoStore fleet.OrgLogoStore
|
|
}
|
|
|
|
// ConditionalAccessMicrosoftProxy is the interface of the Microsoft compliance proxy.
|
|
type ConditionalAccessMicrosoftProxy interface {
|
|
// Create creates the integration on the MS proxy and returns the consent URL.
|
|
Create(ctx context.Context, tenantID string) (*conditional_access_microsoft_proxy.CreateResponse, error)
|
|
// Get returns the integration settings.
|
|
Get(ctx context.Context, tenantID string, secret string) (*conditional_access_microsoft_proxy.GetResponse, error)
|
|
// Delete deprovisions the tenant on Microsoft and deletes the integration in the proxy service.
|
|
// Returns a fleet.IsNotFound error if the integration doesn't exist.
|
|
Delete(ctx context.Context, tenantID string, secret string) (*conditional_access_microsoft_proxy.DeleteResponse, error)
|
|
// SetComplianceStatus sets the inventory and compliance status of a host.
|
|
// Returns the message ID to query the status of the operation (MS has an asynchronous API).
|
|
SetComplianceStatus(
|
|
ctx context.Context,
|
|
tenantID string, secret string,
|
|
deviceID string,
|
|
userPrincipalName string,
|
|
mdmEnrolled bool,
|
|
deviceName, osName, osVersion string,
|
|
compliant bool,
|
|
lastCheckInTime time.Time,
|
|
) (*conditional_access_microsoft_proxy.SetComplianceStatusResponse, error)
|
|
// GetMessageStatusResponse returns the status of a "compliance set" operation.
|
|
GetMessageStatus(ctx context.Context, tenantID string, secret string, messageID string) (*conditional_access_microsoft_proxy.GetMessageStatusResponse, error)
|
|
}
|
|
|
|
func (svc *Service) LookupGeoIP(ctx context.Context, ip string) *fleet.GeoLocation {
|
|
return svc.geoIP.Lookup(ctx, ip)
|
|
}
|
|
|
|
func (svc *Service) SetEnterpriseOverrides(overrides fleet.EnterpriseOverrides) {
|
|
svc.EnterpriseOverrides = &overrides
|
|
}
|
|
|
|
// OsqueryLogger holds osqueryd's status and result loggers.
|
|
type OsqueryLogger struct {
|
|
// Status holds the osqueryd's status logger.
|
|
//
|
|
// See https://osquery.readthedocs.io/en/stable/deployment/logging/#status-logs
|
|
Status fleet.JSONLogger
|
|
// Result holds the osqueryd's result logger.
|
|
//
|
|
// See https://osquery.readthedocs.io/en/stable/deployment/logging/#results-logs
|
|
Result fleet.JSONLogger
|
|
}
|
|
|
|
// NewService creates a new service from the config struct
|
|
func NewService(
|
|
ctx context.Context,
|
|
ds fleet.Datastore,
|
|
task *async.Task,
|
|
resultStore fleet.QueryResultStore,
|
|
logger *slog.Logger,
|
|
osqueryLogger *OsqueryLogger,
|
|
config config.FleetConfig,
|
|
mailService fleet.MailService,
|
|
c clock.Clock,
|
|
sso sso.SessionStore,
|
|
lq fleet.LiveQueryStore,
|
|
carveStore fleet.CarveStore,
|
|
failingPolicySet fleet.FailingPolicySet,
|
|
geoIP fleet.GeoIP,
|
|
enrollHostLimiter fleet.EnrollHostLimiter,
|
|
depStorage nanodep_storage.AllDEPStorage,
|
|
mdmStorage fleet.MDMAppleStore,
|
|
mdmPushService nanomdm_push.Pusher,
|
|
cronSchedulesService fleet.CronSchedulesService,
|
|
wstepCertManager microsoft_mdm.CertManager,
|
|
scepConfigService fleet.SCEPConfigService,
|
|
digiCertService fleet.DigiCertService,
|
|
conditionalAccessProxy ConditionalAccessMicrosoftProxy,
|
|
keyValueStore fleet.KeyValueStore,
|
|
androidSvc android.Service,
|
|
orgLogoStore fleet.OrgLogoStore,
|
|
) (fleet.Service, error) {
|
|
authorizer, err := authz.NewAuthorizer()
|
|
if err != nil {
|
|
return nil, fmt.Errorf("new authorizer: %w", err)
|
|
}
|
|
|
|
svc := &Service{
|
|
ds: ds,
|
|
task: task,
|
|
carveStore: carveStore,
|
|
resultStore: resultStore,
|
|
liveQueryStore: lq,
|
|
logger: logger,
|
|
config: config,
|
|
clock: c,
|
|
osqueryLogWriter: osqueryLogger,
|
|
mailService: mailService,
|
|
ssoSessionStore: sso,
|
|
failingPolicySet: failingPolicySet,
|
|
authz: authorizer,
|
|
jitterH: make(map[time.Duration]*jitterHashTable),
|
|
jitterMu: new(sync.RWMutex),
|
|
geoIP: geoIP,
|
|
enrollHostLimiter: enrollHostLimiter,
|
|
depStorage: depStorage,
|
|
// TODO: remove mdmStorage and mdmPushService when
|
|
// we remove deprecated top-level service methods
|
|
// from the prototype.
|
|
mdmStorage: mdmStorage,
|
|
mdmPushService: mdmPushService,
|
|
mdmAppleCommander: apple_mdm.NewMDMAppleCommander(mdmStorage, mdmPushService),
|
|
cronSchedulesService: cronSchedulesService,
|
|
wstepCertManager: wstepCertManager,
|
|
scepConfigService: scepConfigService,
|
|
digiCertService: digiCertService,
|
|
|
|
conditionalAccessMicrosoftProxy: conditionalAccessProxy,
|
|
keyValueStore: keyValueStore,
|
|
packConfigCache: gocache.New(1*time.Minute, 5*time.Minute),
|
|
androidSvc: androidSvc,
|
|
orgLogoStore: orgLogoStore,
|
|
}
|
|
return validationMiddleware{svc, ds, sso}, nil
|
|
}
|
|
|
|
func (svc *Service) SendEmail(ctx context.Context, mail fleet.Email) error {
|
|
return svc.mailService.SendEmail(ctx, mail)
|
|
}
|
|
|
|
// SetActivityService sets the activity bounded context service for write operations.
|
|
// This should be called after NewService to inject the activity service dependency.
|
|
func (svc *Service) SetActivityService(activitySvc fleet.ActivityWriteService) {
|
|
svc.activitySvc = activitySvc
|
|
}
|
|
|
|
// SetACMEService sets the ACME service module service for write operations.
|
|
// This should be called after NewService to inject the ACME service dependency.
|
|
func (svc *Service) SetACMEService(acmeSvc fleet.ACMEWriteService) {
|
|
svc.acmeSvc = acmeSvc
|
|
}
|
|
|
|
type validationMiddleware struct {
|
|
fleet.Service
|
|
ds fleet.Datastore
|
|
ssoSessionStore sso.SessionStore
|
|
}
|
|
|
|
// getAssetURL simply returns the base url used for retrieving image assets from fleetdm.com.
|
|
func getAssetURL() template.URL {
|
|
return template.URL("https://fleetdm.com/images/permanent")
|
|
}
|
|
|
|
// emailLinkBaseURL returns the base URL used to build links in transactional
|
|
// emails. The server URL is the source of truth; the URL prefix is appended
|
|
// only when the server URL does not already carry it. This keeps links correct
|
|
// whether an operator configures the subpath in the server URL, in the URL
|
|
// prefix, or both, instead of duplicating it (e.g. https://host/p/p/login).
|
|
func emailLinkBaseURL(serverURL, urlPrefix string) template.URL {
|
|
if urlPrefix != "" && !strings.HasSuffix(strings.TrimSuffix(serverURL, "/"), urlPrefix) {
|
|
if joined, err := url.JoinPath(serverURL, urlPrefix); err == nil {
|
|
serverURL = joined
|
|
}
|
|
}
|
|
return template.URL(serverURL) //nolint:gosec // G203: operator-configured URL, not user input
|
|
}
|