<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#46390
Migration to add a non nullable team_id column. I chose this approach so
that server code doesn't have to deal with a potentially null team id
for categories. It first adds team_id=0 to all the existing categories,
then duplicates that for every fleet so that default categories can be
edited and deleted by admins.
It also renames the default categories to include the emojis in their
name, which required updating some test expectations, and also mapping
default names to the new ones for Fleet maintained apps. If we don't do
that, FMA's manifests would have to all be updated right after 4.87
releases and every user would have to migrate immediately. This would
also break existing gitops files if the names aren't mapped. The
alternative would be to keep the names unchanged, and add custom logic
in various places to insert the emojis in the backend and frontend.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Closes#46153
This PR is big, but I found it worth it to include in the same PR to
keep the mental change context in one place.
This PR moves away from our previous version of a big SQL computing the
desired state and label membership with big union branches. It does so
by switching the model up completely, first:
- We batch read hosts (current hardcoded is 5k), and we always iterate
5k hosts and then decide if they have changes, so that means a tick
(30s) could read 5k hosts that DOES NOT require changes, but that is
computed in code after, rather than relying on a big SQL to do it
(twice).
- We then for those hosts, bulk fetch label memberships, their related
team profiles and current rows. This performs much better as we can
lookup everything we need by primary key or super fast indexed columns,
simple fetch all these calls.
- Then once gathered the information we move to the code to determine if
the operation is install, remove, NO-OP (Desired state calculation),
then we check the label membership to further determine it's final
action.
- We then move to what we did before, which is queue the correct command
etc.
It comes with some slight caveats, which is we now load a lot more data
into memory (but before we could spike worse), so when loadtesting we
watched CPU/Memory utilization, which never seemed to spike as the
datasets are kept as small as possible.
_Cleanup will come in a follow-up PR where we remove all the old code._
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance**
* Optimized Apple profile and DDM (Declarations) reconciliation engine
with batched processing for significantly improved performance in
environments with large numbers of Apple-enrolled hosts.
* Implemented cursor-based pagination for more efficient reconciliation
across large fleets.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45573?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#41848
Docs updates: https://github.com/fleetdm/fleet/pull/41868/changes
# Checklist for submitter
- changes not needed since this is a dev environment and test issue
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Enhanced test infrastructure to support environment-variable-based
configuration for SAML, mail, database, and S3 services, enabling more
flexible and dynamic test setups.
* **Chores**
* Updated Docker Compose configuration to use environment variables for
service ports, allowing runtime customization while maintaining backward
compatibility with default values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- already included in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Standardized logging infrastructure across the database and storage
layers for improved consistency and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38234
Minor developer experience improvement for tests using MySQL.
This is a code review follow up.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#37244
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Internal MySQL utility package reorganized and all internal imports
updated to the new platform location; no changes to end-user
functionality or behavior.
* **Documentation**
* Added platform package documentation describing infrastructure
responsibilities and architectural boundaries to guide maintainers.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Jonathan Katz
