Fixes#32295
The issue was identified/fixed using this performance test:
https://gist.github.com/getvictor/b289b7b14981fb7bf77e57c80af117d1
With the fix:
- 100 records: 2.6ms (similar)
- 1,000 records: ~7ms (32x faster)
- 5,000 records: ~10ms (530x faster)
- 10,000 records: 13ms (1,430x faster)
- 20,000 records: ~25ms (2,960x faster)
- 40,000 records: 50ms (6,000x faster)
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Script details now surface the most relevant “latest” status per
script, prioritizing upcoming executions when present.
- Performance Improvements
- Significantly faster loading of host script results and script
details, especially at large scale.
- Improved responsiveness when filtering/sorting script results.
- Documentation
- Added changelog entry describing the optimization to script results
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
# Details
Applying the patches from
https://github.com/fleetdm/fleet/pull/32516#discussion_r2316985807 and
https://github.com/fleetdm/fleet/pull/32563
onto `main`.
This fixes:
* Batches where all remaining hosts are deleted will be correctly marked
as "finished"
* Batches scheduled for the future, and then canceled, will have al
hosts marked as "canceled" rather than pending
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
- [X] Started a new batch script run with one host, deleted that host,
and triggered the batch_activity_completion_checker schedule, and
verified that the batch was moved to "finished".
- [X] Scheduled script
- Created a new batch script run scheduled for a future date
- Canceled that batch run
- Clicked on the batch run in the "finished" tab of the batch scripts
list
- Verified that the number of canceled hosts = the number of targeted
hosts for that batch, and all other numbers were 0.
- Verified that clicking the canceled hosts number navigated to the
correct list of canceled hosts
- [X] "Run now" script
- Created a new batch script run with "run now"
- Waited for at least one host to run.
- Canceled that batch
- Clicked on the batch run in the "finished" tab of the batch scripts
list
- Verified that the number of canceled hosts = the number that were
still pending when I canceled the script, and that the # of pending
hosts was 0
- Verified that clicking the canceled hosts number navigated to the
correct list of canceled hosts
- [X] Multiple batches with the same hosts don't bleed into each other
- Created another batch script with the same hosts, scheduled for the
future
- Verified that the "pending" host list is correct
There are still some TODOs particularly within Gitops test code which
will be worked on in a followup PR
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
- [x] Alerted the release DRI if additional load testing is needed
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
For #32040.
---
Backend changes to unblock the development of the orbit and frontend
changes.
New GET and PUT APIs for setting/getting software for Linux Setup
Experience:
```
curl -k -X GET -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software?team_id=8&per_page=3000
curl -k -X PUT -H "Authorization: Bearer $TEST_TOKEN" https://localhost:8080/api/latest/fleet/setup_experience/linux/software -d '{"team_id":8,"software_title_ids":[3000, 3001, 3007]}'
```
New setup_experience/init API called by orbit to trigger the Linux setup
experience on the device:
```
curl -v -k -X POST -H "Content-Type: application/json" "https://localhost:8080/api/fleet/orbit/setup_experience/init" -d '{"orbit_node_key": "ynYEtFsvv9xZ7rX619UE8of1I28H+GCj"}'
```
Get status API to call on "My device":
```
curl -v -k -X POST "https://localhost:8080/api/latest/fleet/device/7d940b6e-130a-493b-b58a-2b6e9f9f8bfc/setup_experience/status"
```
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Added Linux support for Setup Experience alongside macOS.
- Introduced platform-specific admin APIs to configure and retrieve
Setup Experience software (macOS/Linux).
- Added device API to report Setup Experience status and an Orbit API to
initialize Setup Experience on non-macOS devices.
- Setup Experience now gates policy queries on Linux until setup is
complete.
- New activity log entry when Setup Experience software is edited
(includes platform and team).
- Documentation
- Updated audit logs reference to include the new “edited setup
experience software” event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32160
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Dedicated automations for “No team,” including failing policy webhooks
and Jira/Zendesk ticketing, separate from global settings.
- Policies without a team now use the default team automation
configuration instead of global.
- Chores
- Database migration splits global vs “No team” policy IDs and copies
applicable automation settings to the default team.
- To enable this behavior, set FLEET_PARTNERSHIPS_ENABLE_PRIMO=1 before
running migrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
for #1817
# Details
This PR gives Fleet servers the ability to connect to RDS MySQL and
Elasticache Redis via AWS [Identity and Access Management
(IAM)](https://aws.amazon.com/iam/). It is based almost entirely on the
work of @titanous, branched from his [original pull
request](https://github.com/fleetdm/fleet/pull/31075). The main
differences between his branch and this are:
1. Removal of auto-detection of AWS region (and cache name for
Elasticache) in favor of specifying these values in configuration. The
auto-detection is admittedly handy but parsing AWS host URLs is not
considered a best practice.
2. Relying on the existence of these new configs to determine whether or
not to connect via IAM. This sidesteps a thorny issue of whether to try
an IAM-based Elasticache connection when a password is not supplied,
since this is technically a valid setup.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually - besides using
@titanous's excellent test tool, I verified the following end-to-end:
- [X] regular (non RDS) MySQL connection
- [X] RDS MySQL connection using username/password
- [X] RDS MySQL connection using IAM (no role)
- [X] RDS MySQL connection using IAM (assuming role)
- [X] regular (non Elasticache) Redis connection
- [X] Elasticache Redis connection using username/password
- [X] Elasticache Redis connection using NO password (without IAM)
- [X] Elasticache Redis connection using IAM (no role)
- [X] Elasticache Redis connection using IAM (assuming role)
---------
Co-authored-by: Jonathan Rudenberg <jonathan@titanous.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
For #30095.
#32482 is additional cleanup. Merging this to unblock orchestration
Linux setup experience work. Code has already been reviewed prior to
merging into the feature branch.
---------
Co-authored-by: Konstantin Sykulev <konst@sykulev.com>
Co-authored-by: Anthony Maxwell <133805840+Illbjorn@users.noreply.github.com>
# Overview
This PR addresses a missing product name bump when the parse reaches the
table-formatted section of HTML.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
Signed-off-by: Illbjorn <am@hades.so>
- Added Jira and Zendesk integrations for "No team". (These are not
supported by GitOps for teams)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [x] Setting(s) is/are explicitly excluded from GitOps
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Default (No Team) responses now include limited integrations (Jira,
Zendesk).
- You can configure or clear Jira/Zendesk integrations for the Default
(No Team) settings.
- Bug Fixes
- More consistent handling of the Default (No Team) when fetching team
details.
- Improved validation to prevent conflicting automation settings between
webhooks and integrations.
- Documentation
- Clarified that Jira/Zendesk integrations aren’t supported via GitOps
or at the team level (including No Team).
- Noted that certain options (e.g., Google Calendar, Conditional Access)
aren’t supported for the Default (No Team).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Fixes: #30403
Keys for deletedTitles map were generated differently, causing the same
software title to be marked removed even when a new version of the same
title was inserted.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
For #28713
Refactored the PATH fleet/config end-point to use the primary DB node
for both persisting changes and fetching modified App Config to avoid
stale UI due to read replica delay.
Fixes#31580
Fixes issues
- When updating a script to exactly match the content of another script,
we fail
- When updating one script which happens to match content of another
script, both get updated and not just the one being edited
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Resolved error when updating a script to exactly match another
script’s contents.
* Improved handling of script content updates: identical contents are
deduplicated and unused versions are cleaned up.
* Scheduled/pending runs are canceled on content updates with clearer
cancellation messaging.
* **Documentation**
* Added changelog entry describing the fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32313
OpenTelemetry Tracing
- Added tracing to async task collectors: FlushHostsLastSeen,
collectHostsLastSeen, collectLabelQueryExecutions,
collectPolicyQueryExecutions, collectScheduledQueryStats
- Updated HTTP middleware to use OTEL semantic convention for span names
({method} {route})
- Added OTELEnabled() helper to FleetConfig
Optimizations
- Reduced OTEL batch size from 512 to 256 spans to prevent gRPC message
size errors
- Enabled gzip compression for trace exports
NOTE: I tried to improve OTEL instrumentation for cron jobs, but it got
too complicated due to goroutines in `schedule.go` so that effort should
be separate. We do have SQL instrumentation for cron jobs, but we are
missing root spans for cron jobs as a whole.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Expanded OpenTelemetry tracing for async tasks (host last seen, label
membership, policy membership, scheduled query stats) to provide richer
observability.
* More descriptive HTTP span names using “METHOD /route” for clearer
trace analysis.
* **Bug Fixes**
* Improved OTLP gRPC exporter reliability by enabling gzip compression
and reducing export batch size, mitigating intermittent gRPC errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32060
This PR adds:
- new default_team_config_json table
- caching of config from that table, including deep copy methods -- all
of this is not absolutely needed for this change since we are only using
`webhook_settings.failing_policies_webhook` here but added for
completeness/future
- teams/0 API updates
- GitOps updates
- generate gitops updates
Future PRs will add:
- ticket automation
- primo mode migration
- frontend changes
- documentation
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
## New Fleet configuration settings
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Configure failing-policy webhooks for “No team” via GitOps
(no-team.yml) and API, including enable/disable, destination URL, policy
IDs, and batch size; settings clear when omitted.
- GitOps and CLI now read/apply the real “No team” settings with dry-run
support.
- Policy automation evaluates hosts without a team and triggers “No
team” webhooks when applicable.
- GET/PATCH team 0 returns/accepts a minimal, webhook-focused config.
- Chores
- Added persistence and caching for the default “No team” configuration.
- Introduced a database table to store the default configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Fixes#31432
- Added campaign target cleanup: Deletes targets from campaigns
completed >24h ago. Uses 10% or 50k min per run, processes in 10k
batches. Added DB index, integrated into hourly cron, includes tests.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Automatic cleanup of live query campaign targets 24 hours after
campaign completion to reduce clutter and storage usage.
- Chores
- Added a database index to speed up live query target operations for
improved performance at scale.
- Enhanced scheduled maintenance to log cleanup counts and execution
time for better observability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes: #31581
Undo changes to defFilter, they weren't necessary
Add unit test for team_id=0
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
for #32231
# Details
This PR adjusts the queries for listing batch scripts slightly to count
_every_ row in `batch_activities` matching the filters, regardless of
whether any `batch_activity_host_results` rows exist for it. This
handles the edge case of a batch script where all the hosts have been
deleted.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [ ] Added/updated automated tests
I didn't add tests for this because these tests have already changed
quite a bit in https://github.com/fleetdm/fleet/pull/32174. I can add
tests in there when this merges.
- [X] QA'd all new/changed functionality manually
* Select a host in Manage Hosts, click Run Script, select a script and
do Run Now
* Delete that host
* Go to the batch scripts list (Controls -> Scripts -> Batch Progress)
* Verify that the batch script is still listed.
We don't have clear expectations for what numbers should be displayed
for the progress of a batch like this, but this PR at least ensures the
batch doesn't disappear.
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
Closes#31581
Note:
- When no team id is provided it lists all installers, but they don't
have software_package fielded. I don't know if this is the intended
behavior or not.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
fixes: #30888
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked table schema to confirm autoupdate
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* New Features
* Added support for an IdP full name variable in Apple configuration
profiles, enabling substitution of a user’s full name.
* Profiles automatically re-deploy when a user’s IdP full name changes,
is removed, or the user is assigned to a host.
* Bug Fixes
* Added clearer failure handling: profiles that require an IdP full name
now report a delivery failure with a specific message if the value is
missing.
* Tests
* Expanded test coverage for full name handling and failure scenarios.
* Chores
* Seeded the new variable in the database.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
For #31167
Adds API Changes for showing user-scoped profile scoeps and managed
local user accounts
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
Closes https://github.com/fleetdm/fleet/issues/32257
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
Fixes#32274
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
Used this query for QA. Got denylisted eventually.
```sql
SELECT * FROM time WHERE unix_time = unix_time AND sleep(300) = 0;
```
Fixes: #31283Fixes: #32197
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Test Plan
I manually tested with these powershell versions
Note more CVEs show up than before
CVEs [40733](https://nvd.nist.gov/vuln/detail/CVE-2022-40733) and
[40732](https://nvd.nist.gov/vuln/detail/CVE-2022-40732) are weird but
are expected to show up with this fix.
<google-sheets-html-origin><style type="text/css"><!--td {border: 1px
solid #cccccc;}br {mso-data-placement:same-cell;}--></style>
OS | Name | Version | expected CVE
-- | -- | -- | --
macos | PowerShell | 7.5.0-rc.1 | 30399, 21171
macos | PowerShell | 7.5.0 | 30399
windows | PowerShell 7-preview-x86 | 7.5.0.101 | 30399, 21171, 40733,
40732
windows | PowerShell 7-x64 | 7.5.0.0 | 30399, 40733, 40732
windows | PowerShell 7.5.2.0-x64 | 7.5.2.0 | 40733, 40732
ubuntu | powershell | 7.5.0-1.deb | -
ubuntu | powershell-preview | 7.5.0-rc.1-1.deb | -
For #28133.
When generating the payload for the BitLocker CSP used for setting the
TPM PIN policy, make sure the UUID used in the Atomic enclousure matches
the UUID used for the whole command.
Fixes#31173
Reproduced and fixed in loadtest environment. Uncovered another source
of deadlocks, filed as a separate:
https://github.com/fleetdm/fleet/issues/32201
- Also, still seeing some deadlocks (a lot fewer) in DB, and they are
hidden from the API results by retries. They may still be happening
because locks happen row by row and not all at once. A potential fix
would be to lock the whole policy_membership table.
Additional frontend fix, which is needed to prevent potential timeouts:
https://github.com/fleetdm/fleet/pull/32212
Backend + frontend fix should be a sufficient fix for this issue
(ignoring the issue with the long software transaction).
Also, this PR contains some refactoring to split out the 1-host use
case.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* Bug Fixes
* Resolved rare deadlocks during concurrent policy updates and bulk
automations.
* Correctly clears stale MDM data and actions on host re-enrollment and
platform changes.
* Performance Improvements
* Optimized policy issue recalculation with per-host updates to reduce
contention.
* Improved concurrency handling for bulk policy updates to avoid lock
contention.
* Reliability
* More robust host enrollment: updates seen time, display name, and
label membership consistently.
* Ensures accurate policy-issue counts after membership changes and
re-enrollment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#31477
Docs PR: https://github.com/fleetdm/fleet/pull/32116
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- GitOps now supports FLEET_SECRET_ placeholders in macOS
(.mobileconfig/.xml) profiles. Secrets are expanded only for validation,
while remaining unexpanded in uploaded content.
- Improved environment variable handling: non-secret vars expand as
before; server-side secrets are preserved.
- Validation enforces that profile display names cannot contain
FLEET_SECRET_ values.
- Bug Fixes
- Resolves validation issues when FLEET_SECRET_ appears in <data> tags
by performing safe client-side expansion for validation.
- More accurate error reporting during profile parsing and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
#31977
Tested with:
224 scripts, 102 Apple configuration profiles, 105 Apple declarations,
37 Windows configuration profiles. Scattered in 7 teams.
No variables on profiles/scripts:
- scan scripts: 22.079416ms
- scan apple profiles: 1.865083ms
- scan apple declarations: 683.75µs
- scan windows profiles: 531.167µs
A variable "X" on with profiles/scripts that have variable "Y":
- scan scripts: 26.305459ms
- scan apple profiles: 1.841708ms
- scan apple declarations: 756.917µs
- scan windows profiles: 571.875µs
---
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [X] Confirmed that the fix is not expected to adversely impact load
test results
- [X] Alerted the release DRI if additional load testing is needed
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Prevents deleting a secret that’s still in use. The system now checks
scripts, Apple/Windows MDM profiles, and Apple declarations across teams
before allowing deletion.
- The delete dialog now shows a clear message indicating which item (and
team) is using the secret, replacing the previous generic error.
- Deletion succeeds once all references are removed, ensuring safer
secret management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Overview
This PR implements the S3 upload manager under-the-hood of our
`datastore/s3` client's `Put()` method.
# Description
As surfaced by #31667, the current S3 implementation utilizes the `PUT`
operation which means services, such as MinIO, take issue with attempted
uploads that are [too
large](https://github.com/minio/minio/blob/master/cmd/streaming-signature-v4.go#L260).
The `PUT` operation can also present challenges in memory-constrained
environments as the entire upload target is read into memory before it's
shipped.
# Notes
- See the `TODO` comment section, there's more cool stuff we can and
should do with this in the future!
# Standard Pull Request Details
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
---------
Signed-off-by: Illbjorn <am@hades.so>
For #29795
Refactored the way we compute the number of failing policies to avoid
discrepancies due to either read replica delays or due to async nature
of the failing policy computation stored in host issues.
Fixes: #4498
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- Added unit test
- Changed existing unit tests to accept empty array instead of null
- [x] QA'd all new/changed functionality manually
- Tested that Fleet UI > host details, returns `software: []` instead of
nothing.
- Tested that with exclude_software=false, software returns the full
array for host.
---------
Co-authored-by: Anthony Maxwell <133805840+Illbjorn@users.noreply.github.com>
Fixes#29894 and probably #31980.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
fixes: #29222
This is a feature branch that was completed last week, but did not get
merged in time.
All pr's going in was approved, and reviewed.
I will after this is merged, do a cherry pick onto the RC 4.73 branch,
and initiate the FR merge process.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
fixes#31932
The problem here was that `hostInstalledSoftware` returns a slice of
pointers (`[]*hostSoftware`), so when iterating through and assigning
`LastOpenedAt` the original records were getting modified. This code
duplicates the records being put into `bySoftwareTitleID` so that the
records being stored in `bySoftwareID` are the original records.
As a side benefit to this I modified the logic to store the most recent
`LastOpenedAt` for the software title. I think we may be doing something
similar to this on the front end to show the "last used" column when we
have multiple version of a software installed. But this can potentially
be fetched from the API now.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
