<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#36682
- Added similar "not found" handling than the one we have when calling
`getHostScriptExecutionResultDB` within the `GetHostLockWipeStatus`
function for consistency => now return a 404 instead of a 500 when no
rows are returned.
- Applied similar changes to
https://github.com/fleetdm/fleet/issues/33090 (PR:
https://github.com/fleetdm/fleet/pull/40009) => logging an error instead
of returning a 404, so that the Host details page does not crash.
The actual fix would be to remove the `host_mdm_actions` record that
contains the orphan `lock_ref` mentioned in the issue.
I'm not sure what the root cause is, but I think setting up some
monitoring on the logs I added should help determining that.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #41031
# Details
* Updates server-side error message about software installers to use
"fleet" instead of "team".
* Update front-end code that rewrites that error text 🤦
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Saw correct error banner when trying to add a VPP app that
conflicted with an FMA
<img width="741" height="67" alt="image"
src="https://github.com/user-attachments/assets/d171097c-b165-45f8-bafb-fd6337c94cb9"
/>
- [X] Saw correct error banner when trying to add a script with the same
contents as a another script
<img width="765" height="60" alt="image"
src="https://github.com/user-attachments/assets/db02b92a-942d-448d-9062-3fca49132a94"
/>
I haven't tested all the other cases but I think these two cover them;
one uses the `CantAddSoftwareConflictMessage` constant on the server and
one uses a hard-coded message. Everything else uses the constant.
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
Dismisses some gosec rules in test code where they do not apply, since
they show up when running `golangci-lint run` locally and make it harder
to spot newly introduced errors.
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
FYI @allenhouchins: We will want to remove any of the changes made to
the ee folder in this PR before merging since these files were added for
testing
---
**Related issue:** Resolves
https://github.com/fleetdm/fleet/issues/38254
This pull request updates how Fleet-maintained apps handle default
categories, improving compatibility with older Fleet builds that may not
recognize newer categories. It introduces a more flexible approach for
category assignment, ensuring apps can still be added even if some
categories do not exist in the current database.
**Category compatibility improvements:**
* The service now maps only existing categories to IDs when adding
Fleet-maintained apps, allowing apps to be added even if some default
categories (like "Security" or "Utilities") are not present in older
Fleet builds. This prevents errors and improves backward compatibility.
[[1]](diffhunk://#diff-9e807526199f81f987717f1f3c0ec60260510dc79c12cbeb6b20190dcf39caa7L167-R186)
[[2]](diffhunk://#diff-8384a65651f44fd1a1b78da9bad5794155a76b3849d1b846f8cc5e06018c1365R1)
* Introduced a new `GetSoftwareCategoryNameToIDMap` method in the
datastore to return a mapping of known category names to their IDs, only
including those that exist. This is used in both production and test
code.
[[1]](diffhunk://#diff-4fe70646343c5f7a1b61dd6906889113f0e0ae63f684ab74a617acecb39fe6d3R6170-R6197)
[[2]](diffhunk://#diff-2fa819f19cb7709a65ded13aa51ac75cacbeb5764c88e4a64a02f3ab5b8933b2R700-R702)
[[3]](diffhunk://#diff-f4666ccc907f67a670871fd5600a726858ae613f67aeca8f2537b8315f9b1bf0R544-R545)
[[4]](diffhunk://#diff-f4666ccc907f67a670871fd5600a726858ae613f67aeca8f2537b8315f9b1bf0R2526-R2528)
[[5]](diffhunk://#diff-f4666ccc907f67a670871fd5600a726858ae613f67aeca8f2537b8315f9b1bf0R6148-R6154)
**App category updates:**
* Changed the default category for `appcleaner` from "Productivity" to
"Utilities" in both input and output JSON files.
[[1]](diffhunk://#diff-a6ff089976e7e51f336ab7fc67670375498323de9ec3b87d9aa6c6308ab8fb30L7-R7)
[[2]](diffhunk://#diff-a90f40ad8bd457b4977a1d53984999f89e014fa7442851defc559db252bcdf24L13-R13)
* Changed the default category for `nordvpn` from "Productivity" to
"Security" in both input and output JSON files.
[[1]](diffhunk://#diff-f3ea48ab7da0ffc182e6ec342ee426c0280aa105c79282a7284d0fa35e9a1cf5L7-R7)
[[2]](diffhunk://#diff-83f09709778de1c5a20cbfe88125d58c85563bd1afbc76dcf9f4e8731d6884b3L13-R13)
**Test and codebase adjustments:**
* Updated mocks and tests to use the new
`GetSoftwareCategoryNameToIDMap` method instead of the previous category
ID list approach.
[[1]](diffhunk://#diff-d29fdd0261044f92f761a0707ff8417c9478054ca7d9e4ab5b9d04ed95e15573L285-R286)
[[2]](diffhunk://#diff-d29fdd0261044f92f761a0707ff8417c9478054ca7d9e4ab5b9d04ed95e15573L371-R372)
* Removed an unused import (`fmt`) from `maintained_apps.go` for
cleanup.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
---------
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39150
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40284
Following existing codebase pattern of passing the NOW time to the
method.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
- Via manipulating DB.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enhanced Android certificate renewal system with improved
time-handling capabilities and better testability infrastructure for
certificate template operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: jacobshandling <61553566+jacobshandling@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40183 and #40182
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38536
Split the activities cleanup job from the queries cleanup job.
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added automated cleanup job for expired live queries based on activity
expiration settings.
* **Improvements**
* Refactored activity data cleanup to use a dedicated service for better
reliability and maintainability.
* Enhanced scheduled cleanup operations with improved separation of
concerns for activity and live query management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35799
Loadtest results for 100K hosts and 300K software titles.
```
=== Performance Test Results: No team_id (all teams) ===
Description Average Worst Results
----------- ------- ----- -------
Page 0, hosts_count DESC 229ms 241ms 20 items
Page 0, hosts_count ASC 203ms 211ms 20 items
Page 1, hosts_count DESC 339ms 423ms 20 items
Page 1000, hosts_count DESC 202ms 219ms 20 items
100 per_page, hosts_count DESC 620ms 708ms 100 items
Default sort (no order params) 229ms 245ms 20 items
Order by name ASC, page 0 4.642s 4.785s 20 items
Order by name ASC, page 1000 6.418s 6.771s 20 items
Vulnerable only 3.431s 3.496s 20 items
Search 'chrome' 9.6s 10.111s 20 items
Known exploit filter 9.792s 10.102s 20 items
Min CVSS score 7.0 12.368s 12.665s 20 items
CVSS range 7.0-9.0 12.221s 12.523s 20 items
Available for install 87ms 93ms NO RESULTS
Self-service only 4.46s 4.757s 20 items
=== Performance Test Results: team_id=0 (no team / unassigned) ===
Description Average Worst Results
----------- ------- ----- -------
Page 0, hosts_count DESC 378ms 404ms 20 items
Page 0, hosts_count ASC 339ms 345ms 20 items
Page 1, hosts_count DESC 478ms 513ms 20 items
Page 1000, hosts_count DESC 398ms 417ms 20 items
100 per_page, hosts_count DESC 864ms 1.025s 100 items
Default sort (no order params) 399ms 411ms 20 items
Order by name ASC, page 0 5.346s 5.41s 20 items
Order by name ASC, page 1000 7.444s 7.615s 20 items
Search 'chrome' 9.051s 9.245s 20 items
Known exploit filter 10.511s 10.884s 20 items
Min CVSS score 7.0 16.589s 16.701s 20 items
CVSS range 7.0-9.0 15.878s 15.999s 20 items
Available for install 1.394s 1.429s 1 items
Self-service only 1.4s 1.456s 1 items
```
Documented in the issue:
The fix includes a small behavior change. The default primary sort of
/software/titles remains host_counts, but the secondary sort is now
software_title_id and not name. This was necessary to optimize the
endpoint. This means that if you have 1 host in your fleet, the software
page will not show the software titles ordered by name anymore. For
large fleets, this does not matter since all titles generally have
different host counts.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Optimized the software titles endpoint for improved query performance
and faster data retrieval.
* Enhanced database efficiency when retrieving software information,
resulting in better overall system responsiveness and reduced query
times.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40540
Almost done with slog migration.
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated internal logging infrastructure to use Go's standard logging
library, modernizing the logging system while maintaining existing
functionality and error handling behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This pull request updates the logic for selecting which user receives
maintenance window calendar events on hosts with multiple users. The
changes clarify and enforce a priority system for choosing the recipient
email, ensuring that IdP-sourced emails are preferred, followed by
Google Chrome profile emails. This affects both user-facing
documentation and backend implementation.
**User-facing behavior and documentation:**
* The end-user documentation now explicitly describes the email
selection priority for calendar event recipients: IdP Username email is
chosen first, then Google Chrome profile email, and if multiple Chrome
emails exist, the first alphabetically is selected.
**Backend logic and data selection:**
* The comment in `calendar_cron.go` is updated to match the new email
selection logic, explaining the prioritization of email sources for
host-user assignment.
* The SQL query in `policies.go` is refactored to implement the new
priority system for selecting user emails per host:
- IdP sources (`mdm_idp_accounts`, `idp`) are considered first,
- then Google Chrome profiles,
- then other sources.
- If multiple emails exist at the same priority, the first
alphabetically is chosen.
---------
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Juan Fernandez <juan-fdz-hawa@users.noreply.github.com>
Co-authored-by: Juan Fernandez <juan@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39673
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38536
- Moved PostJSONWithTimeout to platform/http
- Created platform/errors package with only types needed by ctxerr. This
way, ctxerr did not need to import fleethttp.
- Made activity bounded context use PostJSONWithTimeout directly
- Removed some activity types from legacy code that were no longer
needed
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes file `38536-new-activity-bc` already present, and this is just
cleanup from that work.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Refactor**
* Reorganized error handling utilities for improved clarity and
decoupling.
* Consolidated HTTP utilities to centralize JSON posting functionality
with timeout support.
* Simplified activity service initialization by removing unused internal
parameters.
* Cleaned up test utilities and removed webhook-related test
scaffolding.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40549
Fix some settings like setup experience, self service, scripts, not
being updated in BatchSetSoftwareInstallers
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
- Tested that unlocked version FMA setup experience, self service, or
script changes appropriately
- Tested with version locked FMA
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
**Related issue:** Resolves#37802
Attempt to fix Windows FMAs not showing up as available when the
software titles they match to have upgrade codes. Since we don't surface
upgrade codes in the `fleet_maintained_apps` table and matching exactly
by name could miss some cases, this fix uses `team_titles.name LIKE
CONCAT(LEAST(fma.name, fma.unique_identifier), '%')`. Note the LEAST
there is only for the "Box Drive" app which has a longer name than
unique_identifier, and just compares the strings and not their length.
This isn't optimal for performance or correctness, but it only checks
with titles already available to the team as installers so it shouldn't
be terrible. Until upgrade_code is surfaced in the
`fleet_maintained_apps` table this should be sufficient.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [ ] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39672
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information. (Will add in the next task, the bigger piece of
work)
## Testing
- [ ] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38536
This PR moves all logic to create new activities to activity bounded
context.
The old service and ActivityModule methods are not facades that route to
the new activity bounded context. The facades will be removed in a
subsequent PR.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added webhook support for activity events with configurable endpoint
and enable/disable settings.
* Enhanced automation-initiated activity creation without requiring a
user context.
* Improved activity service architecture with centralized creation and
management.
* **Improvements**
* Refactored activity creation to use a dedicated service layer for
better separation of concerns.
* Added support for host-specific and automation-originated activities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #40348
# Details
This PR updates a number of error message on the server to use `fleet`
and `report` instead of `team` or `query` where applicable.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
this is all internal, i don't think it warrants a changelog
## Testing
- [X] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
I did not go trying to trigger all these errors. It's text changes.
Extended the statistics payload to include arrays of Fleet-maintained
app slugs in use on macOS and Windows. Updated the datastore to query
and populate these fields, and documented the new fields in the usage
statistics article.
---------
Co-authored-by: Juan Fernandez <juan@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Already present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Migrated logging to a structured, context-aware backend for clearer,
richer diagnostics and consistent log formatting.
* Introduced broader context propagation and adjusted internal
interfaces to support the new logging approach (no end-user behavior
changes).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35805
Fixed to make sure software host counts tables never have host counts of
0.
Planning to loadtest this fix along with the follow up fix for
https://github.com/fleetdm/fleet/issues/35799
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed accumulation of dead rows in software host count tracking,
improving data accuracy and system performance.
* Enhanced validation to ensure consistent and reliable software
availability records.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#15744
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
Manually inserted 10k hosts and random emails associated to them.
Response times were always below 200ms locally.
We already have covering indexes so we don't need a migration.
https://github.com/user-attachments/assets/721db4f1-f3c9-4ede-ba62-499ac30c4a02
SQL used:
```sql
-- Seed 10k hosts for performance testing email search.
-- Requires: SET cte_max_recursion_depth = 10000;
--
-- Distribution (by osquery_host_id suffix):
-- 1-2500: Chrome profile email only
-- 2501-5000: IdP email only
-- 5001-7500: Both Chrome profile and IdP emails
-- 7501-10000: No email association
-- Generate 10k hosts using a recursive CTE
INSERT INTO hosts (osquery_host_id, node_key, hostname, uuid, platform, detail_updated_at, label_updated_at, policy_updated_at)
WITH RECURSIVE seq AS (
SELECT 1 AS n
UNION ALL
SELECT n + 1 FROM seq WHERE n < 10000
)
SELECT
CONCAT('perf-osq-', n),
CONCAT('perf-nk-', n),
CONCAT('perf-host-', n),
CONCAT('perf-uuid-', n),
'darwin',
NOW(),
'2000-01-01 00:00:00',
'2000-01-01 00:00:00'
FROM seq;
-- Populate display names so that hosts are visible in the UI.
INSERT INTO host_display_names (host_id, display_name)
SELECT id, hostname
FROM hosts
WHERE osquery_host_id LIKE 'perf-osq-%';
-- 1-2500: Chrome profile email only
INSERT INTO host_emails (host_id, email, source)
SELECT id, CONCAT('chrome-', id, '@example.com'), 'google_chrome_profiles'
FROM hosts
WHERE osquery_host_id LIKE 'perf-osq-%'
AND CAST(SUBSTRING(osquery_host_id, 10) AS UNSIGNED) BETWEEN 1 AND 2500;
-- 2501-5000: IdP email only
INSERT INTO host_emails (host_id, email, source)
SELECT id, CONCAT('idp-', id, '@example.com'), 'mdm_idp_accounts'
FROM hosts
WHERE osquery_host_id LIKE 'perf-osq-%'
AND CAST(SUBSTRING(osquery_host_id, 10) AS UNSIGNED) BETWEEN 2501 AND 5000;
-- 5001-7500: Both Chrome profile and IdP emails
INSERT INTO host_emails (host_id, email, source)
SELECT id, CONCAT('chrome-', id, '@example.com'), 'google_chrome_profiles'
FROM hosts
WHERE osquery_host_id LIKE 'perf-osq-%'
AND CAST(SUBSTRING(osquery_host_id, 10) AS UNSIGNED) BETWEEN 5001 AND 7500;
INSERT INTO host_emails (host_id, email, source)
SELECT id, CONCAT('idp-', id, '@example.com'), 'mdm_idp_accounts'
FROM hosts
WHERE osquery_host_id LIKE 'perf-osq-%'
AND CAST(SUBSTRING(osquery_host_id, 10) AS UNSIGNED) BETWEEN 5001 AND 7500;
-- 7501-10000: no emails (nothing to insert)
-- Cleanup:
-- DELETE FROM host_emails WHERE host_id IN (SELECT id FROM hosts WHERE osquery_host_id LIKE 'perf-osq-%');
-- DELETE FROM host_display_names WHERE host_id IN (SELECT id FROM hosts WHERE osquery_host_id LIKE 'perf-osq-%');
-- DELETE FROM hosts WHERE osquery_host_id LIKE 'perf-osq-%';
```
Fixes#34068 Adds automatic retries (up to 3 attempts) for failed software installs from host details, self-service, and setup experience across all installer types.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39858
~~Also implements the idea from this comment:
https://github.com/fleetdm/fleet/issues/37802#issuecomment-3715729822~~
Decided to move the FMA name overriding idea to another PR.
## Changes
- `getOrGenerateSoftwareInstallerTitleID` now attempts to find existing
titles by name or by upgrade code if possible. It will update the
upgrade code if possible.
- also updated `addSoftwareTitleToMatchingSoftware` to match _only_ by
upgrade code if the installer has an upgrade code
These are the assumptions (and tests mostly) I made:
| installer | existing title | result |
|------------------|-----------------------------------|----------------------------------------------------------------------------|
| no upgrade code | same name, no upgrade code | uses existing |
| no upgrade code | same name, has upgrade code | uses existing,
existing upgrade code stays |
| has upgrade code | same name, no upgrade code | uses existing,
existing title is updated with the incoming upgrade code |
| has upgrade code | same name, different upgrade code | new title is
created with same name |
| has upgrade code | same name, same upgrade code | uses existing |
| has upgrade code | different name, same upgrade code | uses existing,
~~existing title's name is updated~~ |
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
- Tested having the title `7-Zip 23.01 (x64)` with an upgrade code in
the db, added 7-Zip FMA, title became the existing `7-Zip 23.01 (x64)`
title with the same upgrade code.
- Trying to add it again fails with the correct error message
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:**
Refactoring these complex desired state queries for readability. No
functional changes
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40028
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Bug Fixes**
* Fixed telemetry misclassification where client-side errors were
incorrectly reported as server errors. Client-side errors and request
cancellations are now properly categorized for improved error tracking
and observability.
* **Tests**
* Added test coverage for client error detection and context
cancellation handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#39921
Summary of fixes:
- Removed unnecessary calls to `BulkSetPendingMDMHostProfiles` when
settings are unchanged.
- Removed 3 unnecessary calls to `BulkSetPendingMDMHostProfiles` when
the Apple OS updates are unset.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves: #35528 (followup)
This migration should fix `.pkg` installers that have a mismatched
software title, as well as software (versions) that are using the wrong
title. It's a little bit risky, as it could modify `software`,
`software_installers`, and `software_titles` data.
For software versions it just checks if the title's source is not the
same as the software. For installers it uses installers with
platform=`darwin` where the title's source isn't `apps`, which should
probably be the only case after the previous migration to convert titles
with source `pkg_packages` to `apps`.
Changefile was already added for 35528 in a previous PR.
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- already included in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Standardized logging infrastructure across the database and storage
layers for improved consistency and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#31820 and #39898
Vulnerability processing performance improvements, and added OTEL spans
to the vulnerabilities cron job.
Optimized the two main bottlenecks in the vulnerability cron job: CPE
matching and CVE insertion. In my loadtest testing (10K hosts), the
overall initial vulnerabilities job went from over 2 hours down to 53
minutes, and the number of spans (DB accesses) went from ~2 million to
~90K.
1. CPE matching (TranslateSoftwareToCPE): replaced the goqu query
builder with hand-written SQL using raw database/sql queries. Replaced
UNION with separate queries because case number 3 was an expensive full
text match operation and in most cases we did not need to do that.
2. CVE insertion (TranslateCPEToCVE and other places): replaced
individual INSERT INTO software_cve ... VALUES (?,?,?,?) calls with
batch inserts of 500 rows each, using the existing BatchProcessSimple
helper. Same pattern applied to OS vulnerability inserts using the
existing InsertOSVulnerabilities batch method.
Functional equivalence verified using osquery perf dataset locally. Both
changes produce identical output (22,366 CPEs, 131,233 CVEs) when
compared against the old code using a before/after comparison tool.
- CPE caveats: bugs #39898 and
https://github.com/fleetdm/fleet/issues/39899 found
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded tracing for automated vulnerability workflows to improve
observability.
* **Performance**
* Bulk/batched processing for software and OS vulnerability inserts to
speed ingestion and downstream tasks.
* More efficient CPE lookup and read-optimized database access for
faster translations.
* **Bug Fixes**
* Improved error recording and read-after-write consistency to reduce
missed or duplicate vulnerability notifications.
* **Tests**
* Test suite updated to support batch insertion semantics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38351
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
- Including Okta conditional access
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Device authentication tokens now remain valid after automatic refresh,
keeping your Device page URL accessible
* Device tokens are automatically generated when needed during
conditional access authentication
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#35528
Potentially #38983
I avoided adding a migration to fix existing bad platform data. However
a small migration is needed to convert software titles with the source
"pkg_packages" that have a bundle id, to the source "apps".
### Changes
To get an installer with the wrong platform fixed:
- Remove then add back the .pkg installer that showed the wrong platform
to see it create a new title with the correct platform
To get hosts that report the wrong platform in inventory/library fixed:
- Devices would still show the app with the wrong platform in
inventory/library. This happens because there are still entries in
`software` that match the incorrect previous title. All of these entries
need to be deleted for this bug to be fixed. One way for this to happen
is for all hosts with that app to remove it then wait for refetch and
the `vulnerabilities` cron, or be deleted directly in the database where
they should get restored after new refetches.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#33090
All the calls to getHostScriptExecutionResultDB within
GetHostLockWipeStatus now log an error with the hostID and execID if no
rows are found. (As agreed with @lucasmrod instead of modifying the DB
as I proposed in the issue, at least for now).
## Testing
- [x] QA'd all new/changed functionality manually
- Set the unlock_reference to a dummy value for a host_mdm_action =>
reproduced 404.
- Restarted the server with the fix => host details page loaded.
https://github.com/user-attachments/assets/4742d2ab-2af1-4837-9b34-895ea7e4463chttps://github.com/user-attachments/assets/50a20523-6ad7-41f3-8546-c6f9bc72f0be
**Related issue:** Resolves#38955
The query that was causing the lock contention was identified as `DELETE
FROM host_mdm_apple_profiles WHERE ( profile_identifier , host_uuid ) IN
( (...) /* , ... */ )` which corresponds to the query we use to delete
the Apple profiles in Bulk. The query was refactored to utilize the
primary key: (profile_uuid, host_uuid), which removes any kind of table
scanning.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38889
This is preparatory work before incrementally converting datastore/mysql
files to directly use *slog.Logger.
This will be done by using `logger.SlogLogger()` to get the underlying
`*slog.Logger`
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes file already exists from previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated internal logging infrastructure to use a standardized platform
logging package across database and utility components. This
consolidates logging dependencies and improves system consistency
without affecting user-facing functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35239
Docs PR: #39770
## Remote trigger approach
When FLEET_VULNERABILITIES_DISABLE_SCHEDULE=true, the main Fleet server
registers a RemoteTriggerSchedule instead of the real vulnerability
schedule. When a user runs fleetctl trigger --name=vulnerabilities:
1. Main server: RemoteTriggerSchedule.Trigger() inserts a cron_stats
record with status=queued.
2. Worker server: The vulnerability schedule runs with
WithTriggerPollInterval(60s), which starts a poll goroutine that checks
the DB every 60s for queued records.
3. Pickup: When the poll goroutine finds a queued record, it sends the
stats ID on the trigger channel (non-blocking).
4. Execution: The trigger handler acquires the lock, claims the record
via ClaimCronStats (updating status to pending and instance to the
actual worker ID), runs all jobs, and marks it completed.
Key details:
- The trigger channel carries an int: 0 for in-process triggers, >0 for
DB-polled stats IDs. This lets runWithStats reuse the existing record
instead of inserting a new one.
- Both Schedule.Trigger() and RemoteTriggerSchedule.Trigger() treat
pending and queued as conflicts to prevent duplicate runs.
- Queued records expire after 2 hours via CleanupCronStats, same as
pending records.
- The poll goroutine only signals; it doesn't modify DB state. The
handler claims when ready.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for remote trigger execution in vulnerability scheduling
workflows.
* Implemented periodic polling mechanism to detect and process
externally triggered vulnerability scans.
* **Bug Fixes**
* Enhanced trigger status tracking to properly handle queued scan jobs.
* **Improvements**
* Strengthened scheduling system with improved timeout and cancellation
management capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39228
Manually tested by triggering a failover on loadtest.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Health checks now detect a primary DB becoming read-only and report
failure so the service restarts and reconnects to a writable primary.
* Write failures due to DB read-only state now trigger immediate fatal
handling to prompt graceful shutdown and recovery.
* Improved detection and handling of read-only DB conditions to increase
stability during failovers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Nico