<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Closes#46153
This PR is big, but I found it worth it to include in the same PR to
keep the mental change context in one place.
This PR moves away from our previous version of a big SQL computing the
desired state and label membership with big union branches. It does so
by switching the model up completely, first:
- We batch read hosts (current hardcoded is 5k), and we always iterate
5k hosts and then decide if they have changes, so that means a tick
(30s) could read 5k hosts that DOES NOT require changes, but that is
computed in code after, rather than relying on a big SQL to do it
(twice).
- We then for those hosts, bulk fetch label memberships, their related
team profiles and current rows. This performs much better as we can
lookup everything we need by primary key or super fast indexed columns,
simple fetch all these calls.
- Then once gathered the information we move to the code to determine if
the operation is install, remove, NO-OP (Desired state calculation),
then we check the label membership to further determine it's final
action.
- We then move to what we did before, which is queue the correct command
etc.
It comes with some slight caveats, which is we now load a lot more data
into memory (but before we could spike worse), so when loadtesting we
watched CPU/Memory utilization, which never seemed to spike as the
datasets are kept as small as possible.
_Cleanup will come in a follow-up PR where we remove all the old code._
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance**
* Optimized Apple profile and DDM (Declarations) reconciliation engine
with batched processing for significantly improved performance in
environments with large numbers of Apple-enrolled hosts.
* Implemented cursor-based pagination for more efficient reconciliation
across large fleets.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45573?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
