Magnus JensenandClaude
b42a154cf6
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Closes #46153 This PR is big, but I found it worth it to include in the same PR to keep the mental change context in one place. This PR moves away from our previous version of a big SQL computing the desired state and label membership with big union branches. It does so by switching the model up completely, first: - We batch read hosts (current hardcoded is 5k), and we always iterate 5k hosts and then decide if they have changes, so that means a tick (30s) could read 5k hosts that DOES NOT require changes, but that is computed in code after, rather than relying on a big SQL to do it (twice). - We then for those hosts, bulk fetch label memberships, their related team profiles and current rows. This performs much better as we can lookup everything we need by primary key or super fast indexed columns, simple fetch all these calls. - Then once gathered the information we move to the code to determine if the operation is install, remove, NO-OP (Desired state calculation), then we check the label membership to further determine it's final action. - We then move to what we did before, which is queue the correct command etc. It comes with some slight caveats, which is we now load a lot more data into memory (but before we could spike worse), so when loadtesting we watched CPU/Memory utilization, which never seemed to spike as the datasets are kept as small as possible. _Cleanup will come in a follow-up PR where we remove all the old code._ # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Optimized Apple profile and DDM (Declarations) reconciliation engine with batched processing for significantly improved performance in environments with large numbers of Apple-enrolled hosts. * Implemented cursor-based pagination for more efficient reconciliation across large fleets. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45573?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude <noreply@anthropic.com>
Platform packages
This directory contains infrastructure and cross-cutting technical concerns that are independent of Fleet's business domain. These packages provide foundational capabilities used across the codebase.
Platform vs domain
Following separation of concerns, we distinguish:
- Platform (infrastructure): Technical concerns like database connectivity, HTTP utilities, middleware, and transport-level error handling. These packages have no knowledge of Fleet's business domain.
- Domain (business logic): Feature-specific code organized into bounded contexts. Domain packages depend on platform packages, not the reverse.
Guidelines
- Platform packages must not import domain packages
- Platform packages should be general-purpose and reusable
- Architectural boundaries are enforced by
arch_test.go