Commit Graph
4010 Commits
Author SHA1 Message Date
Juan Fernandez 82fe6c8560 Better error messages when using built-in labels (#34739)
Resolves #32776 

Show better error messages when using built-in labels.
2025-11-04 12:52:54 -04:00
Jahziel Villasana-Espinoza 1cf16f0539 software display names: DB changes (#35066)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33776 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
2025-11-04 10:04:42 -05:00
Sarah Gillespie 475614f19d Update manual MDM enroll flow for macOS to use OTA with SSO (#34650) 2025-11-04 08:24:50 -06:00
Carlo bf3800b862 CP: Fix activity logs for failed script package installations (#35124)
Cherry-pick of #34951
2025-11-04 08:24:34 -05:00
Victor Lyuboslavsky b3b667a40c Re-timestamped migrations to match 4.76.0 RC branch. (#35136)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34500
2025-11-04 07:21:53 -06:00
Jahziel Villasana-EspinozaandMartin Angers 9259b56da0 update copy, add one-time cron to update default policy (#35127) (#35140)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34899

Cherry pick for #35127

Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
2025-11-03 18:47:39 -05:00
Tim Lee 81831fdd35 SCIM user reconcile migration (#35017) 2025-11-03 16:00:20 -07:00
Scott Gress b482e07605 End-user authentication for Window/Linux setup experience: agent (#34847)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34528 

# Details

This PR implements the agent changes for allowing Fleet admins to
require that users authenticate with an IdP prior to having their
devices set up. I'll comment on changes inline but the high-level is:

1. Orbit calls the enroll endpoint as usual. This is triggered lazily by
any one of a number of subsystems like device token rotation or
requesting Fleet config
2. If the enroll endpoint returns the new `ErrEndUserAuthRequired`
response, then it opens a window to the `/mdm/sso` Fleet page and
retries the enroll endpoint every 30 seconds indefinitely.
3. Any other non-200 response to the enroll request is treated as before
(limited # of retries, with backoff)

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-
changes.md#changes-files) for more information.
Will add changelog when story is one.

## Testing

- [X] Added/updated automated tests
Added test for new retry logic

- [X] QA'd all new/changed functionality manually

This is kinda hard to test without the associated backend PR:
https://github.com/fleetdm/fleet/pull/34835

## fleetd/orbit/Fleet Desktop

- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
This is compatible with all Fleet versions, since older ones won't send
the new error.
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
This is compatible with all platforms, although it currently should only
ever run on Windows and Linux since macOS devices will have end-user
auth taken care of before they even download Orbit.
- [ ] Verified that fleetd runs on macOS, Linux and Windows
Testing this now.
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added SSO (Single Sign-On) enrollment support for end-user
authentication
  * Enhanced error messaging for authentication-required scenarios

* **Bug Fixes**
  * Improved error handling and retry logic for enrollment failures

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-03 16:41:57 -06:00
Jonathan Katz 59a73b1e47 32084 Cisco Secure Client installer fix (#35077)
**Related issue:** Resolves #32084 
This PR modifies `isValidAppFilePath` to allow subdirectors in
`Applications/`, like in this case `Applications/Cisco/Cisco Secure
Client.app`.
This also changes the metadata extraction from packageinfo to trim
`.app` from the name in all cases.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

### Test plan:
---
I ran this on my local environment and it seemed fine
- Have environment with the bug recreated, it has two software titles
for "Cisco Secure Client", and the one with the bundle id
`com.cisco.pkg.anyconnect.vpn` is used by the installer.
- URL to pkg:
https://fndtnfleetmsp.blob.core.windows.net/fndtnpkgs/cisco-secure-client-macos-5.1.3.62-core-vpn-webdeploy-k9.pkg
- Cisco Secure Client doesn't show as installed in UI even after
installing.
- Run the new migration.
- Cisco Secure Client shows as installed now in ui, software title with
bundle id `com.cisco.pkg.anyconnect.vpn` is gone from the database, and
the software installer references the correct title
(`com.cisco.secureclient.gui`).
- Check that deleting and reuploading the installer doesn't recreate the
bad software title.

### QA Note:
---
There are some problems with the install script, but that is probably a
different scope than this ticket.
`Reinstall` wont work, it says Cisco Secure Client is already installed.
Uninstalling through Fleet then Installing again works fine though.
2025-11-03 15:22:39 -05:00
Victor Lyuboslavsky ba5f02f9ca os_versions endpoint performance improvements (#34897)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34500 and Resolves #33758

Video demo: https://www.youtube.com/watch?v=4HZlKG0G1B0

- Added a new aggregation table
`operating_system_version_vulnerabilities` for faster queries. The table
is currently used only for Linux vulnerabilities, but could be used for
other OS vulnerabilities.
- Added `max_vulnerabilities` parameter per [API
doc](https://github.com/fleetdm/fleet/pull/33533)
- Also added `max_vulnerabilities` parameter to `os_versions/{id}`
endpoint, but not making it public since that endpoint is still slow and
needs other API changes. bug #34974
- Removed `"kernels": []` from `os_versions` endpoint result

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added ability to limit the number of vulnerabilities displayed for
operating system versions via an optional parameter.
* Introduced vulnerability count tracking for operating system versions,
now visible in API responses and UI displays.
* Enhanced operating system vulnerability visualization with improved
count-based rendering.

* **Tests**
* Added comprehensive test coverage for vulnerability limiting behavior
across multiple operating system versions and architectures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-03 13:07:44 -06:00
Zach WassermanandVictor Lyuboslavsky fdfd050d87 Fix JSON marshal for software entries (#35011)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #35005 

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Software API responses now properly include installed paths and path
signature information when populated.

* **Tests**
* Added test coverage for software data serialization and installed
paths functionality in API responses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
2025-11-03 11:47:21 -06:00
Zach Wasserman 0cdde239b9 Add activity feed entries for host deletion and expiration (#34720)
**Related issue:** Resolves #33513 

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-31 09:37:31 -07:00
Scott Gress d4271986e0 End-user authentication for Window/Linux setup experience: backend (#34835)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34529 

# Details

This PR implements the backend (and some related front-end screens) for
allowing Fleet admins to require that users authenticate with an IdP
prior to having their devices set up. I'll comment on changes inline but
the high-level for the device enrollment flow is:

1. The handler for the `/orbit/enroll` endpoint now checks whether the
end-user authentication is required for the team (or globally, if using
the global enroll secret).
2. If so, it checks whether a `host_mdm_idp_accounts` row exists with a
`host_uuid` matching the identifier sent with the request
3. If a row exists, enroll. If not, return back a new flavor of
`OrbitError` with a `401` status code and a message
(`END_USER_AUTH_REQUIRED`) that Orbit can interpret and act accordingly.

Additionally some changes were made to the MDM SSO flow. Namely, adding
more data to the session we store for correlating requests we make to
the IdP to initiate SSO to responses aimed at our callback. We now store
a `RequestData` struct which contains the UUID of the device making the
request, as well as the "initiator" (in this case, "setup_experience").
When our SSO callback detects that the initiator was the setup
experience, it attempts to add all of the relevant records to our
database to associate the host with an IdP account. This removes the
enrollment gate in the `/orbit/enroll` endpoint.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
Will put the changelog in the last ticket for the story

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [X] Added/updated automated tests
will see if there's any more to update

- [X] QA'd all new/changed functionality manually
To test w/ SimpleSAML

1. Log in to your local Fleet dashboard with MDM and IdP set up for
SimpleSAML
   1. Go to Settings -> Integrations -> Identity provider
   2. Use "SimpleSAML" for the provider name
   3. Use `mdm.test.com` for the entity ID
4. Use `http://127.0.0.1:9080/simplesaml/saml2/idp/metadata.php` for the
metadata URL
1. Set up a team (or "no team") to have End User Authentication required
(Controls -> Setup experience)
1. Get the enroll secret of that team
1. In the browser console, do:
```
fetch("https://localhost:8080/api/fleet/orbit/enroll", {
  "headers": {
    "accept": "application/json, text/plain, */*",
    "cache-control": "no-cache",
    "content-type": "application/json",
    "pragma": "no-cache",
  },
  "body": "{\"enroll_secret\":\"<enroll secret>", \"hardware_uuid\":\"abc123\" }",
  "method": "POST",
});
``` 
replacing `<enroll secret>` with your team's enroll secret.

8. Verify in the network tab that you get a 401 error with message
`END_USER_AUTH_REQUIRED`
1. Go to
https://localhost:8080/mdm/sso?initiator=setup_experience&host_uuid=abc123
1. Verify that a new screen appears asking you to log in to your IdP
1. Log in to SimpleSAML with `sso_user / user123#`
1. Verify that you're taken to a success screen
1. In your database, verify that records exist in the `mdm_idp_accounts`
and `host_mdm_idp_accounts` tables with uuid `abc123`
1. Try the `fetch` command in the browser console again, verify that it
succeeds.

## fleetd/orbit/Fleet Desktop

- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
This is _not_ compatible with the current version of fleetd or the
soon-to-be-released 1.49.x. Until #34847 changes are released in fleetd,
this will need to be put behind a feature flag or withheld from Fleet
releases.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added support for device UUID linkage during MDM enrollment to enable
host-initiated enrollment tracking
* Introduced setup experience flow for device authentication during
enrollment
* Added end-user authentication requirement configuration for macOS MDM
enrollment

* **Improvements**
* Enhanced MDM enrollment process to maintain device context through
authentication
* Updated authentication UI to display completion status for device
setup flows
  * Refined form layout styling for improved visual consistency

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-31 11:16:42 -05:00
Magnus Jensen 8ccdbe05f1 DCSW: Add managed certificate entry for Windows SCEP profiles. (#34964)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34251 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-31 12:53:09 -03:00
Magnus Jensen d2d918dd49 DCSW: Verify Windows SCEP profiles without validation (#34852)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34249

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-30 14:50:03 -03:00
RachelElysiaandCarlo DiCelico c7ce54c267 Fleet UI: Show run/running for setup experience software scripts (#34938)
---------

Co-authored-by: Carlo DiCelico <carlo@fleetdm.com>
2025-10-30 09:41:36 -04:00
Jordan Montgomery a884e5472c Add pkiclient.exe suffix support to SCEP proxy handler (#34709)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34252 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually
2025-10-29 19:59:06 -03:00
Martin Angers b776398e67 Cherrypick of bugfix for 4.76.0: make software title status counts consistent (#34944)
Cherry-pick PR into `main` from
https://github.com/fleetdm/fleet/pull/34932 that targeted 4.76.0
2025-10-29 14:56:56 -04:00
Jonathan Katz b3df8b7940 Fix software_package info not provided (#34888)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34882 
Also fix uploaded_time not being returned in metadata

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually
2025-10-28 16:21:51 -04:00
Jonathan Katz f4fd1c106e Fix in house app package name and title name (#34868)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34823 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually
## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
2025-10-28 13:19:13 -04:00
Robert Fairburn 1fedabe7a8 Update alpine base image to latest (#34864)
Resolves openssl:3.3.3/CVE-2025-9230 in base images.
2025-10-28 11:24:05 -05:00
Martin Angers cab7cc15be Initial support for in-house apps on iOS/iPadOS (#34802) 2025-10-28 08:33:58 -04:00
jacobshandling 3aad722924 (releases on merge to main) Fix vuln false positives for "Logi Bolt.app" (#33920)
**_QA on-branch before merge_**
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #31082 

Before:
<img width="1258" height="989" alt="Screenshot 2025-10-06 at 9 31 46 PM"
src="https://github.com/user-attachments/assets/b027b3b6-6201-468d-9141-76b80daa35c8"
/>

After
<img width="1258" height="989" alt="Screenshot 2025-10-06 at 9 26 03 PM"
src="https://github.com/user-attachments/assets/147bdd41-5ebc-4d23-bd85-c1cb963a429d"
/>


- [x] Changes file added for user-visible changes in `changes/`
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-27 16:55:30 -07:00
jacobshandling cb2d42de78 (releases on merge to main) Fix vuln false positives for vscode golang extension (#33839)
**_QA on-branch before merge_**

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
## Resolves #33235 

<img width="2556" height="1419" alt="Screenshot 2025-10-03 at 5 55
40 PM"
src="https://github.com/user-attachments/assets/49078de7-699a-4a64-86ab-f435065f91ed"
/>



- [x] Changes file added for user-visible changes in `changes/`
2025-10-27 14:48:29 -07:00
Magnus Jensen f2341497c8 DCSW: Allow uploading and validating Windows SCEP profiles (#34691)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34247 

Allows for uploading and validating SCEP profiles for Windows.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.
## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-27 10:48:37 -03:00
Victor Lyuboslavsky 27b8d1aa4b Fixed issue searching software versions (#34770)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34713

Fix for unreleased but. Needs to be cherry picked into 4.76.0

# Checklist for submitter

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-25 16:02:02 -05:00
jacobshandling ab0065ab39 (releases on merge to main) Detect JetBrains IDE plugin vulnerabilities (#34331)
**Related issue:** Resolves #32266


[Demo](https://drive.google.com/file/d/1ZDYJkWkxZ519le8v9qGmcrL8YkP-uivW/view?usp=sharing)

- [x] Changes file added for user-visible changes in `changes/`,
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-24 12:06:35 -07:00
Tim Lee c5d7c9f626 31970 NPM vuln support (#33100) 2025-10-24 12:54:57 -06:00
Jordan Montgomery 79b886455a Update host expiry logic to account for Apple MDM checkin times (#34698)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32499

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved host expiry logic to correctly identify and preserve Apple
MDM-enrolled hosts that don't check in through Orbit, preventing
unintended host deletions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-24 13:12:03 -04:00
Victor Lyuboslavsky 0db1b472a1 Okta conditional access configs (#34566)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34533

This is the first sub-task out of several. Changes file will be added in
a subsequent PR.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## New Fleet configuration settings

- [x] Setting(s) is/are explicitly **excluded** from GitOps

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Okta Conditional Access support (IDP, ACS URL, audience,
certificate) and exposed conditional access in AppConfig/API
  * App activity logging for adding/removing Okta conditional access

* **Bug Fixes**
  * Fixed typo in conditional access validation messaging

* **Tests**
* Added tests for Okta Conditional Access lifecycle, license gating, and
GitOps export exclusion

* **Documentation**
  * Added audit-log entries for Okta conditional access add/delete
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-24 10:11:14 -05:00
Magnus Jensen ba15654498 DCSW: Support all IDP variables in Windows config profiles (#34707)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves
https://fleetdm.slack.com/archives/C03C41L5YEL/p1761232938314509, but
also https://github.com/fleetdm/fleet/issues/34363, as it would have
been stupid to only add IDP_USERNAME (to allow for an email) when they
are so close together and no extra effort to support the rest.

Also does another fix to ensure fleet variables are correctly ordered by
longest name to shortest, to ensure variables that are used in longer
format (USERNAME, USERNAME_LOCAL_PART) that the LOCAL_PART one gets
processed first.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.


## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
2025-10-24 10:10:58 -03:00
Carlo 855ead6338 Fix script package unsupported fields in GitOps (#34703)
Fixes #34643. Fixes script packages (`.sh` and `.ps1`) incorrectly outputting unsupported fields in GitOps YAML and accepting invalid configuration during upload.
2025-10-23 17:45:08 -04:00
Carlo 821ed9d8b9 Add .sh packages to Linux setup experience (#34710)
Fixes #34654. Updates the SQL query to recognize `.sh` as a valid distribution-agnostic installer extension.
2025-10-23 17:20:47 -04:00
Jahziel Villasana-Espinoza 5c7416711f enable app reports on default profile (#34693)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** No ticket, just a bug noticed while testing

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

## Testing

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-23 13:33:35 -04:00
Magnus Jensen 5105e1914b Windows Var Replacement: use updated profile contents instead of incoming (#34681)
This fixes a bug, but also fixes the flaky test runs seen in
[Slack](https://fleetdm.slack.com/archives/C019WG4GH0A/p1761192670589249).
2025-10-23 10:04:24 -03:00
Magnus Jensen f682f063bd Check if IsMDMLostModeEnabled is present before taking value (#34653)
Resolves the dogfood down, issue.
https://fleetdm.slack.com/archives/C071NNMSP2R/p1761162242092529
2025-10-22 17:19:32 -03:00
Magnus Jensen d6a23a79ee Modify Windows replacement code to allow Custom SCEP variables (#34633)
and refactor to share with apple mdm

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34246 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
2025-10-22 15:46:48 -03:00
7593d102fb Experimental fleet server config for custom updates & disk encryption settings (#34598)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33316

Merges in changes made in this community PR:
https://github.com/fleetdm/fleet/pull/33665

Adds support for Windows and tests, also blocks the feature on fleet
free

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

---------

Co-authored-by: Wesley Whetstone <wesw@stripe.com>
Co-authored-by: Wesley Whetstone <jckwhet@gmail.com>
2025-10-22 13:51:10 -04:00
Tim Lee d4004a4f8e IDP user update API (#34332) 2025-10-21 12:02:25 -06:00
Scott Gress 7fa8793b1e Add macos web setup experience capability check (#34582)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34596 

# Details

This PR adds back the "legacy" macOS setup experience, and a new
`CapabilityMacOSWebSetupExperience` capability check. The legacy
experience will be shown whenever the
`CapabilityMacOSWebSetupExperience` capability is _not_ detected in the
Fleet server response.

The majority of the code is just copying the `Run` and
`startSwiftDialog` from the [4.75.1 patch
release](https://github.com/fleetdm/fleet/blob/rc-patch-fleet-v4.75.1/orbit/pkg/setup_experience/setup_experience.go)
and adding them back as `RunLegacy` and `startSwiftDialogLegacy`, then
adding the little bit of plumbing to have `Run()` call `RunLegacy()`
when the new capability is not detected.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [X] QA'd all new/changed functionality manually
Tested on a VM that the setup experience starts in "legacy" mode if
`CapabilityMacOSWebSetupExperience` is not set in the server, and that
it starts in the new web view if the capability _is_ set.


For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed

## fleetd/orbit/Fleet Desktop

- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
I have not re-verified this, but the changes will only run on macOS
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
This should not be affected.
2025-10-21 12:32:16 -05:00
Konstantin Sykulev c42a732ef7 Unbatching checksum migration (#34586)
**Related issue:** https://github.com/fleetdm/fleet/issues/34559
PR for 4.75.1: https://github.com/fleetdm/fleet/pull/34485

needs to be added in main to be consistent.
2025-10-21 11:21:33 -05:00
Ian Littman e995891359 Revise 4.76 migrations to land after all 4.75 migrations (#34479)
- [x] QA'd locally blank -> 4.75 -> this branch
2025-10-17 17:55:57 -05:00
Ian LittmanandScott Gress 52410a3328 Add CPE translations to map various software where name/bundle ID based matching doesn't work (#34233)
Fixes #34282. Must be QA'd on-branch so split from the software name
cleanup PR for #34159.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

---------

Co-authored-by: Scott Gress <scott@fleetdm.com>
2025-10-17 16:09:14 -05:00
Jordan Montgomery 23a4b781fd ISII: Make setup_experience gitops flag on app_store_apps enable setup for all platforms an app supports (#34461)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #30890 

Unreleased bug(or requirements miss) in iOS/iPadOS setup experience. Per
@noahtalerman we want setup_experience on VPP apps to apply to all
platforms an app is available for:
https://github.com/fleetdm/fleet/issues/30890#issuecomment-3408445511

This will change with the upcoming inclusion of the `platform` field and
it will be possible to specify settings for each individual OS

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-17 15:16:34 -04:00
jacobshandling 1c0b004676 Server changes for matching JetBrains IDE vulnerabilities (#34459)
## Addresses #32266 

These are the same changes present in
https://github.com/fleetdm/fleet/pull/34331, excluding changes that
would instantly be released. This separation will allow these
already-approved server changes to be merged before the cut while
allowing more time for QA to get to the QA-on-branch parts of the issue
2025-10-17 11:50:29 -07:00
Sarah Gillespie e11ddc9866 Support $FLEET_VAR_SCEP_RENEWAL_ID in OU field (#34403) 2025-10-17 12:47:04 -05:00
jacobshandlingandJacob Shandling 5f626e2a8c Add gigs_all_disk_space vital collection, storage, service, and UI rendering for Linux hosts (#34077)
## Addresses #31671 

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added total disk space metrics for all partitions on Linux hosts. The
disk space indicator now displays comprehensive storage information
including root partition and all other partitions, improving visibility
into host storage capacity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
2025-10-17 08:24:23 -07:00
Victor Lyuboslavsky 9295a82e83 Improved MySQL query performance software versions and vulnerabilities endpoints (#34262)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32178

Software optimization: skipping an unnecessary software_cve join when
vulnerability details are not needed. Vulnerabilities are still
returned, so functionality remains unchanged.

Vulnerabilities optimization: Query vulnerability_host_counts directly
and LEFT JOIN for metadata. This eliminates the expensive UNION of all
CVE rows that was causing performance issues.

Previous approach: UNION all CVEs (many rows) → JOIN
vulnerability_host_counts → filter
New approach: Start with filtered vulnerability_host_counts → LEFT JOIN
for metadata
This reduces the working set before any expensive operations

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests (see below for the test)
- [x] QA'd all new/changed functionality manually
- Planning to test in loadtest after also improving the software
endpoint

Performance test for replicating the problem and testing the fix:
```go
package mysql

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/ptr"
	"github.com/fleetdm/fleet/v4/server/test"
	"github.com/stretchr/testify/require"
)

// TestListVulnerabilitiesPerformance is a performance test that replicates
// the production performance problem with ListVulnerabilities.
//
// This test creates a realistic dataset with thousands of CVEs and measures
// query performance under various conditions. Run with:
//
//	go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
//
// To see detailed timing output, set the environment variable:
//
//	VERBOSE=1 go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
func TestListVulnerabilitiesPerformance(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping performance test in short mode")
	}

	ds := CreateMySQLDS(t)
	defer TruncateTables(t, ds)

	ctx := context.Background()

	// Create a realistic dataset
	t.Log("Setting up test data...")
	setupPerformanceTestData(t, ds)
	t.Log("Test data setup complete")

	// Test cases covering common query patterns
	testCases := []struct {
		name string
		opts fleet.VulnListOptions
	}{
		{
			name: "Global list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE:   true,
				TeamID: ptr.Uint(1),
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - with exploit filter",
			opts: fleet.VulnListOptions{
				IsEE:         true,
				TeamID:       ptr.Uint(1),
				KnownExploit: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - with CVE search",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					MatchQuery:     "2023",
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - second page",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           1,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Free version - global list",
			opts: fleet.VulnListOptions{
				IsEE: false,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
	}

	// Run performance tests
	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			// Warm up the query cache
			_, _, err := ds.ListVulnerabilities(ctx, tc.opts)
			require.NoError(t, err)

			// Measure query performance
			const iterations = 5
			var totalDuration time.Duration

			for i := 0; i < iterations; i++ {
				start := time.Now()
				vulns, meta, err := ds.ListVulnerabilities(ctx, tc.opts)
				duration := time.Since(start)
				totalDuration += duration

				require.NoError(t, err)
				require.NotNil(t, meta)
				require.NotEmpty(t, vulns, "expected vulnerabilities to be returned")

				if i == 0 {
					t.Logf("  First run: %v (returned %d results)", duration, len(vulns))
				}
			}

			avgDuration := totalDuration / iterations
			t.Logf("  Average of %d runs: %v", iterations, avgDuration)

			// Performance assertions
			// These thresholds represent the current performance problem
			// After optimization, these should be reduced significantly
			if avgDuration > 2*time.Second {
				t.Logf("  ⚠️  WARNING: Query took %v (>2s) - performance issue detected", avgDuration)
			} else if avgDuration > 500*time.Millisecond {
				t.Logf("  ⚠️  SLOW: Query took %v (>500ms)", avgDuration)
			} else {
				t.Logf("  ✓ GOOD: Query took %v (<500ms)", avgDuration)
			}
		})
	}

	// Test count query performance
	t.Run("Count vulnerabilities performance", func(t *testing.T) {
		opts := fleet.VulnListOptions{
			IsEE: true,
		}

		// Warm up
		_, err := ds.CountVulnerabilities(ctx, opts)
		require.NoError(t, err)

		// Measure
		const iterations = 5
		var totalDuration time.Duration

		for i := 0; i < iterations; i++ {
			start := time.Now()
			count, err := ds.CountVulnerabilities(ctx, opts)
			duration := time.Since(start)
			totalDuration += duration

			require.NoError(t, err)
			require.Greater(t, count, uint(0))

			if i == 0 {
				t.Logf("  First run: %v (count=%d)", duration, count)
			}
		}

		avgDuration := totalDuration / iterations
		t.Logf("  Average of %d runs: %v", iterations, avgDuration)

		if avgDuration > 2*time.Second {
			t.Logf("  ⚠️  WARNING: Count query took %v (>2s)", avgDuration)
		} else if avgDuration > 500*time.Millisecond {
			t.Logf("  ⚠️  SLOW: Count query took %v (>500ms)", avgDuration)
		} else {
			t.Logf("  ✓ GOOD: Count query took %v (<500ms)", avgDuration)
		}
	})
}

// BenchmarkListVulnerabilities provides benchmark results for ListVulnerabilities.
// Run with:
//
//	go test -bench=BenchmarkListVulnerabilities -benchmem -run=^$ ./server/datastore/mysql
func BenchmarkListVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()

	// Setup test data
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	// Benchmark the most common query pattern
	opts := fleet.VulnListOptions{
		IsEE: true,
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkListVulnerabilitiesWithTeam benchmarks team-specific queries
func BenchmarkListVulnerabilitiesWithTeam(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE:   true,
		TeamID: ptr.Uint(1),
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkCountVulnerabilities benchmarks the count query
func BenchmarkCountVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE: true,
	}

	for i := 0; i < b.N; i++ {
		_, err := ds.CountVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// setupPerformanceTestData creates a realistic dataset that mimics production
// This creates:
// - ~80,000+ unique CVEs (matching production scale)
// - ~73,000 software_cve entries
// - ~35,000 operating_system_vulnerabilities entries
// - Multiple teams
// - Various host counts per vulnerability
//
// Note: This will take several minutes to run but will replicate production performance issues
func setupPerformanceTestData(t testing.TB, ds *Datastore) {
	ctx := context.Background()

	// Create 100 hosts across different teams and OS types (doubled from 50)
	// More hosts = more realistic host count distributions
	hosts := make([]*fleet.Host, 100)
	for i := 0; i < 100; i++ {
		hosts[i] = test.NewHost(t, ds, fmt.Sprintf("host%d", i),
			fmt.Sprintf("192.168.1.%d", i%255+1), // Handle more than 255 hosts
			fmt.Sprintf("key%d", i),
			fmt.Sprintf("uuid%d", i),
			time.Now())
	}

	// Create 3 teams
	team1, err := ds.NewTeam(ctx, &fleet.Team{Name: "Engineering"})
	require.NoError(t, err)

	team2, err := ds.NewTeam(ctx, &fleet.Team{Name: "Sales"})
	require.NoError(t, err)

	team3, err := ds.NewTeam(ctx, &fleet.Team{Name: "Support"})
	require.NoError(t, err)

	// Distribute hosts across teams
	// 40 hosts in team1, 30 in team2, 20 in team3, 10 with no team
	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team1.ID, getHostIDs(hosts[0:40])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team2.ID, getHostIDs(hosts[40:70])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team3.ID, getHostIDs(hosts[70:90])))
	require.NoError(t, err)

	// Set up OS versions (Windows, macOS, Ubuntu)
	windowsOS := fleet.OperatingSystem{
		Name:     "Microsoft Windows 11 Enterprise",
		Version:  "10.0.22621.2715",
		Arch:     "x86_64",
		Platform: "windows",
	}

	macOS := fleet.OperatingSystem{
		Name:     "macOS",
		Version:  "14.1.2",
		Arch:     "arm64",
		Platform: "darwin",
	}

	ubuntuOS := fleet.OperatingSystem{
		Name:     "Ubuntu",
		Version:  "22.04",
		Arch:     "x86_64",
		Platform: "ubuntu",
	}

	// Assign OS to hosts: 50 Windows, 30 macOS, 20 Ubuntu
	for i := 0; i < 50; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, windowsOS)
		require.NoError(t, err)
	}
	for i := 50; i < 80; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, macOS)
		require.NoError(t, err)
	}
	for i := 80; i < 100; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, ubuntuOS)
		require.NoError(t, err)
	}

	err = ds.UpdateOSVersions(ctx)
	require.NoError(t, err)

	// Create realistic CVE distribution matching production scale
	// In production, we see:
	// - ~73,000 software CVEs
	// - ~35,000 OS CVEs
	// - Many CVEs overlap between software and OS
	// - ~80,000 unique CVEs after deduplication

	// We now create production-scale data:
	// - 50,000 software CVEs (70% of production)
	// - 30,000 OS CVEs (85% of production)
	// - Some overlap to create ~80,000 total CVEs after UNION
	// This should replicate the 500-1000ms+ query times seen in production

	t.Log("Creating software vulnerabilities... (this will take a few minutes)")
	createSoftwareVulnerabilities(t, ds, hosts[:60], 50000)

	t.Log("Creating OS vulnerabilities...")
	createOSVulnerabilities(t, ds, 30000)

	t.Log("Creating CVE metadata...")
	createCVEMetadata(t, ds, 80000)

	t.Log("Updating vulnerability host counts...")
	err = ds.UpdateVulnerabilityHostCounts(ctx, 10)
	require.NoError(t, err)

	t.Log("Setup complete - ready for performance testing")
}

// createSoftwareVulnerabilities creates software entries and their CVEs
func createSoftwareVulnerabilities(t testing.TB, ds *Datastore, hosts []*fleet.Host, numCVEs int) {
	ctx := context.Background()

	// Create more software packages to better distribute CVEs
	softwarePackages := []fleet.Software{
		{Name: "Chrome", Version: "120.0.1", Source: "programs"},
		{Name: "Firefox", Version: "121.0", Source: "programs"},
		{Name: "Node.js", Version: "18.19.0", Source: "programs"},
		{Name: "Python", Version: "3.11.7", Source: "programs"},
		{Name: "Docker", Version: "24.0.7", Source: "programs"},
		{Name: "nginx", Version: "1.24.0", Source: "deb_packages"},
		{Name: "postgresql", Version: "15.5", Source: "deb_packages"},
		{Name: "redis", Version: "7.2.3", Source: "deb_packages"},
		{Name: "mysql", Version: "8.0.35", Source: "deb_packages"},
		{Name: "git", Version: "2.43.0", Source: "deb_packages"},
		{Name: "openssl", Version: "3.0.12", Source: "deb_packages"},
		{Name: "curl", Version: "8.5.0", Source: "deb_packages"},
		{Name: "vim", Version: "9.0", Source: "deb_packages"},
		{Name: "apache2", Version: "2.4.58", Source: "deb_packages"},
		{Name: "php", Version: "8.2.14", Source: "deb_packages"},
	}

	// Install software on hosts
	for i, host := range hosts {
		// Each host gets 5-8 software packages
		numPackages := 5 + (i % 4)
		if numPackages > len(softwarePackages) {
			numPackages = len(softwarePackages)
		}
		hostSoftware := softwarePackages[:numPackages]
		_, err := ds.UpdateHostSoftware(ctx, host.ID, hostSoftware)
		require.NoError(t, err)
	}

	err := ds.SyncHostsSoftware(ctx, time.Now())
	require.NoError(t, err)

	// Create CVEs for software (distributed across 15 software IDs)
	// Each software gets many CVEs to simulate real-world vulnerability distribution
	cvesPerSoftware := numCVEs / 15
	t.Logf("  Creating %d CVEs per software package (15 packages)", cvesPerSoftware)

	for softwareID := uint(1); softwareID <= 15; softwareID++ {
		// Insert CVEs in batches for better performance
		batchSize := 1000
		for batchStart := 0; batchStart < cvesPerSoftware; batchStart += batchSize {
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerSoftware {
				batchEnd = cvesPerSoftware
			}

			for i := batchStart; i < batchEnd; i++ {
				cveNum := int(softwareID-1)*cvesPerSoftware + i
				// Use wider CVE number range to avoid duplicates
				cve := fmt.Sprintf("CVE-2023-%05d", cveNum)

				_, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
					SoftwareID: softwareID,
					CVE:        cve,
				}, fleet.NVDSource)
				require.NoError(t, err)
			}
		}

		if softwareID%5 == 0 {
			t.Logf("  Progress: %d/%d software packages completed", softwareID, 15)
		}
	}
}

// createOSVulnerabilities creates OS vulnerabilities
func createOSVulnerabilities(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()

	// Create CVEs for each OS type
	// OS ID 1 = Windows, 2 = macOS, 3 = Ubuntu
	osIDs := []uint{1, 2, 3}
	cvesPerOS := numCVEs / len(osIDs)

	t.Logf("  Creating %d CVEs per OS type (3 OS types)", cvesPerOS)

	for _, osID := range osIDs {
		// Insert in batches to avoid memory issues with large slices
		batchSize := 5000
		totalBatches := (cvesPerOS + batchSize - 1) / batchSize

		for batchNum := 0; batchNum < totalBatches; batchNum++ {
			batchStart := batchNum * batchSize
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerOS {
				batchEnd = cvesPerOS
			}

			vulns := make([]fleet.OSVulnerability, batchEnd-batchStart)
			for i := 0; i < len(vulns); i++ {
				actualIndex := batchStart + i
				// Use different year to avoid overlap with software CVEs
				// (but still create some overlap)
				cveNum := int(osID-1)*cvesPerOS + actualIndex
				cve := fmt.Sprintf("CVE-2022-%05d", cveNum)

				// 10% overlap with software CVEs for realism
				if actualIndex%10 == 0 {
					cve = fmt.Sprintf("CVE-2023-%05d", cveNum)
				}

				vulns[i] = fleet.OSVulnerability{
					OSID: osID,
					CVE:  cve,
				}
			}

			_, err := ds.InsertOSVulnerabilities(ctx, vulns, fleet.MSRCSource)
			require.NoError(t, err)

			if (batchNum+1)%2 == 0 || batchNum == totalBatches-1 {
				t.Logf("  Progress: OS %d - batch %d/%d completed", osID, batchNum+1, totalBatches)
			}
		}
	}
}

// createCVEMetadata creates CVE metadata entries
func createCVEMetadata(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()
	mockTime := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)

	// Create metadata in batches of 500 for better performance
	batchSize := 500
	totalBatches := (numCVEs + batchSize - 1) / batchSize
	t.Logf("  Creating CVE metadata in %d batches", totalBatches)

	for start := 0; start < numCVEs; start += batchSize {
		end := start + batchSize
		if end > numCVEs {
			end = numCVEs
		}

		batch := make([]fleet.CVEMeta, end-start)
		for i := start; i < end; i++ {
			// Alternate between 2022 and 2023 CVEs to match our created vulnerabilities
			year := 2023
			if i >= 50000 {
				year = 2022
			}

			// Use 5-digit format to match our CVE creation
			cve := fmt.Sprintf("CVE-%d-%05d", year, i%100000)

			// 30% have CISA known exploit
			cisaExploit := (i % 10) < 3

			batch[i-start] = fleet.CVEMeta{
				CVE:              cve,
				CVSSScore:        ptr.Float64(5.0 + float64(i%50)/10.0),
				EPSSProbability:  ptr.Float64(float64(i%100) / 100.0),
				CISAKnownExploit: ptr.Bool(cisaExploit),
				Published:        ptr.Time(mockTime.Add(time.Duration(i) * time.Hour)),
				Description:      fmt.Sprintf("Test vulnerability %s", cve),
			}
		}

		err := ds.InsertCVEMeta(ctx, batch)
		require.NoError(t, err)

		// Report progress every 10 batches
		batchNum := (start / batchSize) + 1
		if batchNum%10 == 0 || batchNum == totalBatches {
			t.Logf("  Progress: %d/%d batches completed (%d CVEs)", batchNum, totalBatches, end)
		}
	}
}

// getHostIDs extracts host IDs from a slice of hosts
func getHostIDs(hosts []*fleet.Host) []uint {
	ids := make([]uint, len(hosts))
	for i, h := range hosts {
		ids[i] = h.ID
	}
	return ids
}

// CreateMySQLDSForBenchmark creates a datastore for benchmarking
func CreateMySQLDSForBenchmark(b *testing.B) *Datastore {
	return CreateMySQLDS(b)
}
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance Improvements**
* Faster loading of the vulnerabilities list via optimized database
queries for the vulnerabilities API endpoint.

* **Bug Fixes**
* More accurate “created at” timestamps for vulnerabilities, improving
sorting and consistency.
* More consistent source attribution for vulnerabilities when multiple
sources are available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-17 09:57:47 -05:00
RachelElysiaandCarlo DiCelico 1ef91fe4e3 Feature: Script only package e2e followup (#34271)
Co-authored-by: Carlo DiCelico <carlo@fleetdm.com>
2025-10-17 10:54:00 -04:00
Jordan Montgomery 927fd1d240 Fix android policy non-compliance errors (#34430)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34422 

Unreleased bug in 4.75.0 Android Config Profile feature. No changes file
as such

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-17 10:27:24 -04:00