Commit Graph
19039 Commits
Author SHA1 Message Date
Noah Talerman 3b2717f4fd YAML reference (#34604)
Use serial numbers for labels example. Serial numbers is best practice.
2025-10-21 14:15:57 -05:00
Zach Wasserman 6726bb196f Add mcp_listening_servers table (#34286)
**Related issue:** Resolves #34330

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually (so far just macOS)

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] Verified that fleetd runs on macOS, Linux and Windows


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a built-in mcp_listening_servers table to discover MCP servers
by inspecting listening ports and probing endpoints; returns process
info, server metadata, capabilities, tools, prompts, and resources
(supports macOS, Windows, Linux).

* **Tests**
* Added comprehensive unit tests covering detection, IPv6 handling, SSE
responses, and session lifecycle.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-21 11:25:11 -07:00
Savannah Friend caa57019f0 Include CEO's preparation steps for board meetings (#34593) 2025-10-21 13:22:07 -05:00
Tim Lee d4004a4f8e IDP user update API (#34332) 2025-10-21 12:02:25 -06:00
Scott Gress 7fa8793b1e Add macos web setup experience capability check (#34582)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34596 

# Details

This PR adds back the "legacy" macOS setup experience, and a new
`CapabilityMacOSWebSetupExperience` capability check. The legacy
experience will be shown whenever the
`CapabilityMacOSWebSetupExperience` capability is _not_ detected in the
Fleet server response.

The majority of the code is just copying the `Run` and
`startSwiftDialog` from the [4.75.1 patch
release](https://github.com/fleetdm/fleet/blob/rc-patch-fleet-v4.75.1/orbit/pkg/setup_experience/setup_experience.go)
and adding them back as `RunLegacy` and `startSwiftDialogLegacy`, then
adding the little bit of plumbing to have `Run()` call `RunLegacy()`
when the new capability is not detected.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [X] QA'd all new/changed functionality manually
Tested on a VM that the setup experience starts in "legacy" mode if
`CapabilityMacOSWebSetupExperience` is not set in the server, and that
it starts in the new web view if the capability _is_ set.


For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed

## fleetd/orbit/Fleet Desktop

- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
I have not re-verified this, but the changes will only run on macOS
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
This should not be affected.
2025-10-21 12:32:16 -05:00
Konstantin Sykulev c42a732ef7 Unbatching checksum migration (#34586)
**Related issue:** https://github.com/fleetdm/fleet/issues/34559
PR for 4.75.1: https://github.com/fleetdm/fleet/pull/34485

needs to be added in main to be consistent.
2025-10-21 11:21:33 -05:00
Harrison Ravazzolo fdc184fe58 Windows CSP - Spotlight config + Okta scep (#34589) 2025-10-21 08:47:50 -07:00
Noah Talerman 140d1dd92f Fix wording and formatting in product design README (#34592) 2025-10-21 11:42:52 -04:00
Gabe Lopez df176aa047 Update release-qa.md MDM tests (#34470)
added the following to the MDM smoke tests:

- account-driven user enrollments for iOS/iPadOS
- Windows Autopilot enrollments
- Android tests
- Certificate Authorities
- IdP provisioning SCIM
2025-10-21 08:17:36 -07:00
fleet-releaseandmostlikelee 39931ccdec Update Fleet-maintained apps (#34580)
Automated ingestion of latest Fleet-maintained app data.

Co-authored-by: mostlikelee <16102903+mostlikelee@users.noreply.github.com>
2025-10-21 10:36:17 -04:00
Tim Lee 77df7e011c bugfix: correct santa json status based on santa docs (#34561) 2025-10-21 08:26:12 -06:00
Luke Heath 5ad1b1295a Add open positions for QA and Software Specialist (#34547) 2025-10-20 20:03:51 -05:00
Scott Gress b44a9ab6e4 Update script upload UI: updates from QA (#34564)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32632
**Related issue:** Resolves #34454 

# Details

Addresses the following QA-reported issues with #32632:

* Adding a .sh script file does not show it is valid for macOS
<img width="617" height="183" alt="image"
src="https://github.com/user-attachments/assets/39dcf884-ffbc-43b3-a06b-ad88520d6a16"
/>

* With Gitops mode enabled the 'add scripts' button is still clickable
<img width="693" height="151" alt="image"
src="https://github.com/user-attachments/assets/84c6e77d-ed1c-4e8a-8ebd-099d9da5405a"
/>

* Controls > Scripts: "Add script" is missing padding (#34454)
<img width="173" height="79" alt="image"
src="https://github.com/user-attachments/assets/d89cb728-f4ba-427b-868f-f072196777ae"
/>

# Checklist for submitter

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-20 19:19:40 -05:00
Rachael ShawandNoah Talerman 16478197dc Update ritual for reviewing upcoming release reference docs (#34558)
Missed the step about reverting changes that were already merged into
the docs release branch.

---------

Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
2025-10-20 18:02:10 -04:00
jacobshandling f700d1029e Hide software host count, versions table when no hosts have the software installed (#34484)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32635
Hide host count and version table when no hosts have the software
installed

<img width="1518" height="568" alt="Screenshot 2025-10-17 at 4 36 13 PM"
src="https://github.com/user-attachments/assets/4f77d039-d9d0-427e-a3f4-a8774a3f6ff7"
/>



- [x] Changes file added for user-visible changes in `changes/`
- [x] QA'd all new/changed functionality manually
2025-10-20 14:46:35 -07:00
Ian Littman 75a3af5b43 Don't try to pull config profiles when MDM isn't enabled for any platform when generating GitOps YAML (#34549)
Fixes #33677.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-20 15:47:44 -05:00
George Karr 33650644c8 Update sprint retrospective guidelines with tool recommendations (#34543) 2025-10-20 15:22:43 -05:00
dependabot[bot] 5b57843922 Bump playwright from 1.39.0 to 1.56.1 (#34548) 2025-10-20 15:21:33 -05:00
malomar124 14f6697c6d Update README.md (#34538) 2025-10-20 15:03:19 -05:00
Scott GressandIan Littman 6e0235cb31 Update test file to allow Go to zip module (#34554)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #6994

This is a non-functional change to the `generate-gitops` tests that
renames a testfile to not contain an emoji, because Go can't zip and
publish modules that have filenames with emojis in them.

The actual output from the `generate-gitops` command _can_ contain a
file that has an emoji in its name, representing a team with an emoji in
its name. I updated the test here to translate the test filename (now
without emoji) to the actual expected filename (with the emoji).

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2025-10-20 14:50:13 -05:00
Rachael Shaw 688f50b619 Update product design rituals (#34482)
+ Remove "🦢🚀🤝 Product Design + Frontend Engineering" ritual
+ Formalize the rituals for release docs: add post-sprint-kickoff,
pre-release, and post-release documentation tasks
2025-10-20 15:47:58 -04:00
Sam Pfluger b49e7826d4 Fix links in communications.md for writing section (#34546) 2025-10-20 14:01:30 -05:00
Noah Talerman 49a458a8ca Add Fleet UI refresh details to release notes (#34536) 2025-10-20 12:36:12 -05:00
fleet-releaseandmostlikelee 81d963cc4a Update Fleet-maintained apps (#34498)
Automated ingestion of latest Fleet-maintained app data.

Co-authored-by: mostlikelee <16102903+mostlikelee@users.noreply.github.com>
2025-10-20 09:53:22 -04:00
Luke Heath 2c8ae8cc78 Adding changes for Fleet v4.75.0 (#33583) (#34483) 2025-10-17 21:51:17 -05:00
jacobshandling f80d8a5eb8 Remove unused mock endpoint handler (#34301) 2025-10-17 16:10:21 -07:00
Ian Littman e995891359 Revise 4.76 migrations to land after all 4.75 migrations (#34479)
- [x] QA'd locally blank -> 4.75 -> this branch
2025-10-17 17:55:57 -05:00
Noah Talerman 9318d26e90 Release article: 4.75 (#34390) 2025-10-17 17:46:50 -05:00
+3 7ed4aac2b8 Docs v4.75.0 (#34443)
Documentation changes for the 4.75 release

---------

Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
Co-authored-by: Jordan Montgomery <elijah.jordan.montgomery@gmail.com>
Co-authored-by: Tim Lee <timlee@fleetdm.com>
Co-authored-by: Janis Watts <184028114+jmwatts@users.noreply.github.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
2025-10-17 17:45:52 -05:00
jacobshandling c9a14d5038 Update vuln processing guide, add change file for JetBrains IDE extension vulns (#34478)
Addresses #32266 

Update docs, add change file
2025-10-17 17:39:21 -05:00
Noah Talerman 93acdb8e8c Product Design handbook: Stories that don't require product changes (#34463) 2025-10-17 17:11:01 -05:00
Noah Talerman 19899d48d1 Product Design rituals: Revise sprint kickoff review and add sprint retro task (#34477)
- Separate sprint kickoff reviews for @marko-lisica (#g-mdm +
#g-software) and @rachaelshaw (#g-orchestration +
#g-security-compliance)
  - Up to the Product Designer to reschedule as needed
- Sprint retro is its own call
2025-10-17 18:01:38 -04:00
Eric 351ac2b845 Website: Update created account record type (#34473)
Changes:
- Updated the `update-or-create-contact-and-account` helper to set a
`Type` on new account records created.
2025-10-17 16:40:21 -05:00
Ian LittmanandScott Gress 52410a3328 Add CPE translations to map various software where name/bundle ID based matching doesn't work (#34233)
Fixes #34282. Must be QA'd on-branch so split from the software name
cleanup PR for #34159.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

---------

Co-authored-by: Scott Gress <scott@fleetdm.com>
2025-10-17 16:09:14 -05:00
jacobshandling d9cedcba8d Consolidate rolling linux version possibilities (#34467)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34214

- [x] manual QA by @sgress454
2025-10-17 14:01:03 -07:00
RachelElysia 69494cc0d1 Fleet UI: Remove purple count and 2 links (#34462) 2025-10-17 15:56:13 -05:00
Luke Heath aba434b024 Bump font size variable from 13px to 14px and correct page header gap (#34453) 2025-10-17 15:00:14 -05:00
Jordan Montgomery 23a4b781fd ISII: Make setup_experience gitops flag on app_store_apps enable setup for all platforms an app supports (#34461)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #30890 

Unreleased bug(or requirements miss) in iOS/iPadOS setup experience. Per
@noahtalerman we want setup_experience on VPP apps to apply to all
platforms an app is available for:
https://github.com/fleetdm/fleet/issues/30890#issuecomment-3408445511

This will change with the upcoming inclusion of the `platform` field and
it will be possible to specify settings for each individual OS

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-17 15:16:34 -04:00
jacobshandling 1c0b004676 Server changes for matching JetBrains IDE vulnerabilities (#34459)
## Addresses #32266 

These are the same changes present in
https://github.com/fleetdm/fleet/pull/34331, excluding changes that
would instantly be released. This separation will allow these
already-approved server changes to be merged before the cut while
allowing more time for QA to get to the QA-on-branch parts of the issue
2025-10-17 11:50:29 -07:00
Jordan Montgomery eafade0a12 Remove Setup Experience preview for iOS/iPadOS (#34458)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33711 

See Noah comment here for why we are removing:
https://github.com/fleetdm/fleet/issues/30890#issuecomment-3411319996

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-17 13:57:25 -04:00
Sarah Gillespie e11ddc9866 Support $FLEET_VAR_SCEP_RENEWAL_ID in OU field (#34403) 2025-10-17 12:47:04 -05:00
RachelElysia bc837a7bd3 Fleet UI: Undo filter same row as header name (#34457) 2025-10-17 12:46:46 -05:00
Noah Talerman 5b78b69a72 Roadmap preview: October 2025 (#33179) 2025-10-17 13:24:00 -04:00
Rachael Shaw 047f1f8a96 Make Dale CODEOWNER of /docs/solutions (#34455) 2025-10-17 12:15:57 -05:00
Gabriel Hernandez af254bd43b fix unlock action and refresh host state for ios/ipad (#34444) 2025-10-17 17:59:22 +01:00
Scott Gress 81f589d661 Update add script UI (#34349)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32632

# Details

This PR updates the Script Library page in the following ways:
* When no scripts are uploaded for a team, it shows the "Add script" UI
with a button that opens a new "Add Script" modal
* When scripts are uploaded, the "Add script" button is instead added to
the header of the scripts list, and clicking it opens that modal

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests
working on this

- [X] QA'd all new/changed functionality manually
- [X] Test empty state: go to controls/scripts/library for a team with
no scripts. Clicking "upload" button in empty state should open the add
script modal.
- [X] In the modal, select a .ps1 script. Should not see additional
text.
- [X] Close modal without uploading. Re-open. File field should be
cleared & upload button visible again.
- [X] Select a .sh script. Should see additional text about macOS and
Linux.
  - [X] Add script. Make sure script saves and modal closes.
- [X] Once script has been added, make sure empty state is gone and "Add
script" button is at the top of the list.
- [X] Go to /controls/os-settings/custom-settings for a team with no
profiles uploaded. Make sure empty state text styles match the empty
state for script uploads.
- [X] Open modal to add profile. Make sure upload text styles match the
script upload modal.
- [X] Enable GitOps mode. Go to controls/scripts/library for a team with
scripts added. Make sure new "Add script" button is disabled w/ standard
tooltip in GitOps mode.

Scripts empty state:

<img width="697" height="352" alt="image"
src="https://github.com/user-attachments/assets/32f0f246-bddb-4bb7-bc39-48d9978de9fa"
/>

Scripts uploader:

<img width="745" height="590" alt="image"
src="https://github.com/user-attachments/assets/f82414e2-9318-4543-b5ca-41e759662587"
/>

Scripts uploader with .sh

<img width="750" height="539" alt="image"
src="https://github.com/user-attachments/assets/0b989067-921a-4d18-93ed-09aac90fc9cb"
/>

Scripts table:

<img width="686" height="256" alt="image"
src="https://github.com/user-attachments/assets/848f1b56-6e9e-48d4-9a03-6fdf5427301e"
/>

Profiles empty state:

<img width="700" height="377" alt="image"
src="https://github.com/user-attachments/assets/8f92bcd9-2215-41f6-a540-4774f7e9542b"
/>

Profiles uploader:

<img width="707" height="682" alt="image"
src="https://github.com/user-attachments/assets/eef216af-3447-48e7-882a-e42e888e1c17"
/>
2025-10-17 10:49:59 -05:00
jacobshandling 4dd9fe822e Improve alignment and logic around Arch linux os version displays (#34420)
**Related issue:** Resolves #34214 

- Alignment in host summary fixed for both Firefox(L) and Chrome(R):
<img width="1728" height="221" alt="Screenshot 2025-10-16 at 6 38 39 PM"
src="https://github.com/user-attachments/assets/27d4489d-7a90-41bd-9b7d-53dfc95e4000"
/>

- Fix detection logic in Software > OS Versions table, render
tooltip-wrapped "rolling" there when appropriate

- [x] QA'd all new/changed functionality manually
2025-10-17 08:26:42 -07:00
jacobshandlingandJacob Shandling 5f626e2a8c Add gigs_all_disk_space vital collection, storage, service, and UI rendering for Linux hosts (#34077)
## Addresses #31671 

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added total disk space metrics for all partitions on Linux hosts. The
disk space indicator now displays comprehensive storage information
including root partition and all other partitions, improving visibility
into host storage capacity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
2025-10-17 08:24:23 -07:00
Victor Lyuboslavsky 9295a82e83 Improved MySQL query performance software versions and vulnerabilities endpoints (#34262)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32178

Software optimization: skipping an unnecessary software_cve join when
vulnerability details are not needed. Vulnerabilities are still
returned, so functionality remains unchanged.

Vulnerabilities optimization: Query vulnerability_host_counts directly
and LEFT JOIN for metadata. This eliminates the expensive UNION of all
CVE rows that was causing performance issues.

Previous approach: UNION all CVEs (many rows) → JOIN
vulnerability_host_counts → filter
New approach: Start with filtered vulnerability_host_counts → LEFT JOIN
for metadata
This reduces the working set before any expensive operations

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests (see below for the test)
- [x] QA'd all new/changed functionality manually
- Planning to test in loadtest after also improving the software
endpoint

Performance test for replicating the problem and testing the fix:
```go
package mysql

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/ptr"
	"github.com/fleetdm/fleet/v4/server/test"
	"github.com/stretchr/testify/require"
)

// TestListVulnerabilitiesPerformance is a performance test that replicates
// the production performance problem with ListVulnerabilities.
//
// This test creates a realistic dataset with thousands of CVEs and measures
// query performance under various conditions. Run with:
//
//	go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
//
// To see detailed timing output, set the environment variable:
//
//	VERBOSE=1 go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
func TestListVulnerabilitiesPerformance(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping performance test in short mode")
	}

	ds := CreateMySQLDS(t)
	defer TruncateTables(t, ds)

	ctx := context.Background()

	// Create a realistic dataset
	t.Log("Setting up test data...")
	setupPerformanceTestData(t, ds)
	t.Log("Test data setup complete")

	// Test cases covering common query patterns
	testCases := []struct {
		name string
		opts fleet.VulnListOptions
	}{
		{
			name: "Global list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE:   true,
				TeamID: ptr.Uint(1),
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - with exploit filter",
			opts: fleet.VulnListOptions{
				IsEE:         true,
				TeamID:       ptr.Uint(1),
				KnownExploit: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - with CVE search",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					MatchQuery:     "2023",
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - second page",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           1,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Free version - global list",
			opts: fleet.VulnListOptions{
				IsEE: false,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
	}

	// Run performance tests
	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			// Warm up the query cache
			_, _, err := ds.ListVulnerabilities(ctx, tc.opts)
			require.NoError(t, err)

			// Measure query performance
			const iterations = 5
			var totalDuration time.Duration

			for i := 0; i < iterations; i++ {
				start := time.Now()
				vulns, meta, err := ds.ListVulnerabilities(ctx, tc.opts)
				duration := time.Since(start)
				totalDuration += duration

				require.NoError(t, err)
				require.NotNil(t, meta)
				require.NotEmpty(t, vulns, "expected vulnerabilities to be returned")

				if i == 0 {
					t.Logf("  First run: %v (returned %d results)", duration, len(vulns))
				}
			}

			avgDuration := totalDuration / iterations
			t.Logf("  Average of %d runs: %v", iterations, avgDuration)

			// Performance assertions
			// These thresholds represent the current performance problem
			// After optimization, these should be reduced significantly
			if avgDuration > 2*time.Second {
				t.Logf("  ⚠️  WARNING: Query took %v (>2s) - performance issue detected", avgDuration)
			} else if avgDuration > 500*time.Millisecond {
				t.Logf("  ⚠️  SLOW: Query took %v (>500ms)", avgDuration)
			} else {
				t.Logf("  ✓ GOOD: Query took %v (<500ms)", avgDuration)
			}
		})
	}

	// Test count query performance
	t.Run("Count vulnerabilities performance", func(t *testing.T) {
		opts := fleet.VulnListOptions{
			IsEE: true,
		}

		// Warm up
		_, err := ds.CountVulnerabilities(ctx, opts)
		require.NoError(t, err)

		// Measure
		const iterations = 5
		var totalDuration time.Duration

		for i := 0; i < iterations; i++ {
			start := time.Now()
			count, err := ds.CountVulnerabilities(ctx, opts)
			duration := time.Since(start)
			totalDuration += duration

			require.NoError(t, err)
			require.Greater(t, count, uint(0))

			if i == 0 {
				t.Logf("  First run: %v (count=%d)", duration, count)
			}
		}

		avgDuration := totalDuration / iterations
		t.Logf("  Average of %d runs: %v", iterations, avgDuration)

		if avgDuration > 2*time.Second {
			t.Logf("  ⚠️  WARNING: Count query took %v (>2s)", avgDuration)
		} else if avgDuration > 500*time.Millisecond {
			t.Logf("  ⚠️  SLOW: Count query took %v (>500ms)", avgDuration)
		} else {
			t.Logf("  ✓ GOOD: Count query took %v (<500ms)", avgDuration)
		}
	})
}

// BenchmarkListVulnerabilities provides benchmark results for ListVulnerabilities.
// Run with:
//
//	go test -bench=BenchmarkListVulnerabilities -benchmem -run=^$ ./server/datastore/mysql
func BenchmarkListVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()

	// Setup test data
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	// Benchmark the most common query pattern
	opts := fleet.VulnListOptions{
		IsEE: true,
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkListVulnerabilitiesWithTeam benchmarks team-specific queries
func BenchmarkListVulnerabilitiesWithTeam(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE:   true,
		TeamID: ptr.Uint(1),
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkCountVulnerabilities benchmarks the count query
func BenchmarkCountVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE: true,
	}

	for i := 0; i < b.N; i++ {
		_, err := ds.CountVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// setupPerformanceTestData creates a realistic dataset that mimics production
// This creates:
// - ~80,000+ unique CVEs (matching production scale)
// - ~73,000 software_cve entries
// - ~35,000 operating_system_vulnerabilities entries
// - Multiple teams
// - Various host counts per vulnerability
//
// Note: This will take several minutes to run but will replicate production performance issues
func setupPerformanceTestData(t testing.TB, ds *Datastore) {
	ctx := context.Background()

	// Create 100 hosts across different teams and OS types (doubled from 50)
	// More hosts = more realistic host count distributions
	hosts := make([]*fleet.Host, 100)
	for i := 0; i < 100; i++ {
		hosts[i] = test.NewHost(t, ds, fmt.Sprintf("host%d", i),
			fmt.Sprintf("192.168.1.%d", i%255+1), // Handle more than 255 hosts
			fmt.Sprintf("key%d", i),
			fmt.Sprintf("uuid%d", i),
			time.Now())
	}

	// Create 3 teams
	team1, err := ds.NewTeam(ctx, &fleet.Team{Name: "Engineering"})
	require.NoError(t, err)

	team2, err := ds.NewTeam(ctx, &fleet.Team{Name: "Sales"})
	require.NoError(t, err)

	team3, err := ds.NewTeam(ctx, &fleet.Team{Name: "Support"})
	require.NoError(t, err)

	// Distribute hosts across teams
	// 40 hosts in team1, 30 in team2, 20 in team3, 10 with no team
	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team1.ID, getHostIDs(hosts[0:40])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team2.ID, getHostIDs(hosts[40:70])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team3.ID, getHostIDs(hosts[70:90])))
	require.NoError(t, err)

	// Set up OS versions (Windows, macOS, Ubuntu)
	windowsOS := fleet.OperatingSystem{
		Name:     "Microsoft Windows 11 Enterprise",
		Version:  "10.0.22621.2715",
		Arch:     "x86_64",
		Platform: "windows",
	}

	macOS := fleet.OperatingSystem{
		Name:     "macOS",
		Version:  "14.1.2",
		Arch:     "arm64",
		Platform: "darwin",
	}

	ubuntuOS := fleet.OperatingSystem{
		Name:     "Ubuntu",
		Version:  "22.04",
		Arch:     "x86_64",
		Platform: "ubuntu",
	}

	// Assign OS to hosts: 50 Windows, 30 macOS, 20 Ubuntu
	for i := 0; i < 50; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, windowsOS)
		require.NoError(t, err)
	}
	for i := 50; i < 80; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, macOS)
		require.NoError(t, err)
	}
	for i := 80; i < 100; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, ubuntuOS)
		require.NoError(t, err)
	}

	err = ds.UpdateOSVersions(ctx)
	require.NoError(t, err)

	// Create realistic CVE distribution matching production scale
	// In production, we see:
	// - ~73,000 software CVEs
	// - ~35,000 OS CVEs
	// - Many CVEs overlap between software and OS
	// - ~80,000 unique CVEs after deduplication

	// We now create production-scale data:
	// - 50,000 software CVEs (70% of production)
	// - 30,000 OS CVEs (85% of production)
	// - Some overlap to create ~80,000 total CVEs after UNION
	// This should replicate the 500-1000ms+ query times seen in production

	t.Log("Creating software vulnerabilities... (this will take a few minutes)")
	createSoftwareVulnerabilities(t, ds, hosts[:60], 50000)

	t.Log("Creating OS vulnerabilities...")
	createOSVulnerabilities(t, ds, 30000)

	t.Log("Creating CVE metadata...")
	createCVEMetadata(t, ds, 80000)

	t.Log("Updating vulnerability host counts...")
	err = ds.UpdateVulnerabilityHostCounts(ctx, 10)
	require.NoError(t, err)

	t.Log("Setup complete - ready for performance testing")
}

// createSoftwareVulnerabilities creates software entries and their CVEs
func createSoftwareVulnerabilities(t testing.TB, ds *Datastore, hosts []*fleet.Host, numCVEs int) {
	ctx := context.Background()

	// Create more software packages to better distribute CVEs
	softwarePackages := []fleet.Software{
		{Name: "Chrome", Version: "120.0.1", Source: "programs"},
		{Name: "Firefox", Version: "121.0", Source: "programs"},
		{Name: "Node.js", Version: "18.19.0", Source: "programs"},
		{Name: "Python", Version: "3.11.7", Source: "programs"},
		{Name: "Docker", Version: "24.0.7", Source: "programs"},
		{Name: "nginx", Version: "1.24.0", Source: "deb_packages"},
		{Name: "postgresql", Version: "15.5", Source: "deb_packages"},
		{Name: "redis", Version: "7.2.3", Source: "deb_packages"},
		{Name: "mysql", Version: "8.0.35", Source: "deb_packages"},
		{Name: "git", Version: "2.43.0", Source: "deb_packages"},
		{Name: "openssl", Version: "3.0.12", Source: "deb_packages"},
		{Name: "curl", Version: "8.5.0", Source: "deb_packages"},
		{Name: "vim", Version: "9.0", Source: "deb_packages"},
		{Name: "apache2", Version: "2.4.58", Source: "deb_packages"},
		{Name: "php", Version: "8.2.14", Source: "deb_packages"},
	}

	// Install software on hosts
	for i, host := range hosts {
		// Each host gets 5-8 software packages
		numPackages := 5 + (i % 4)
		if numPackages > len(softwarePackages) {
			numPackages = len(softwarePackages)
		}
		hostSoftware := softwarePackages[:numPackages]
		_, err := ds.UpdateHostSoftware(ctx, host.ID, hostSoftware)
		require.NoError(t, err)
	}

	err := ds.SyncHostsSoftware(ctx, time.Now())
	require.NoError(t, err)

	// Create CVEs for software (distributed across 15 software IDs)
	// Each software gets many CVEs to simulate real-world vulnerability distribution
	cvesPerSoftware := numCVEs / 15
	t.Logf("  Creating %d CVEs per software package (15 packages)", cvesPerSoftware)

	for softwareID := uint(1); softwareID <= 15; softwareID++ {
		// Insert CVEs in batches for better performance
		batchSize := 1000
		for batchStart := 0; batchStart < cvesPerSoftware; batchStart += batchSize {
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerSoftware {
				batchEnd = cvesPerSoftware
			}

			for i := batchStart; i < batchEnd; i++ {
				cveNum := int(softwareID-1)*cvesPerSoftware + i
				// Use wider CVE number range to avoid duplicates
				cve := fmt.Sprintf("CVE-2023-%05d", cveNum)

				_, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
					SoftwareID: softwareID,
					CVE:        cve,
				}, fleet.NVDSource)
				require.NoError(t, err)
			}
		}

		if softwareID%5 == 0 {
			t.Logf("  Progress: %d/%d software packages completed", softwareID, 15)
		}
	}
}

// createOSVulnerabilities creates OS vulnerabilities
func createOSVulnerabilities(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()

	// Create CVEs for each OS type
	// OS ID 1 = Windows, 2 = macOS, 3 = Ubuntu
	osIDs := []uint{1, 2, 3}
	cvesPerOS := numCVEs / len(osIDs)

	t.Logf("  Creating %d CVEs per OS type (3 OS types)", cvesPerOS)

	for _, osID := range osIDs {
		// Insert in batches to avoid memory issues with large slices
		batchSize := 5000
		totalBatches := (cvesPerOS + batchSize - 1) / batchSize

		for batchNum := 0; batchNum < totalBatches; batchNum++ {
			batchStart := batchNum * batchSize
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerOS {
				batchEnd = cvesPerOS
			}

			vulns := make([]fleet.OSVulnerability, batchEnd-batchStart)
			for i := 0; i < len(vulns); i++ {
				actualIndex := batchStart + i
				// Use different year to avoid overlap with software CVEs
				// (but still create some overlap)
				cveNum := int(osID-1)*cvesPerOS + actualIndex
				cve := fmt.Sprintf("CVE-2022-%05d", cveNum)

				// 10% overlap with software CVEs for realism
				if actualIndex%10 == 0 {
					cve = fmt.Sprintf("CVE-2023-%05d", cveNum)
				}

				vulns[i] = fleet.OSVulnerability{
					OSID: osID,
					CVE:  cve,
				}
			}

			_, err := ds.InsertOSVulnerabilities(ctx, vulns, fleet.MSRCSource)
			require.NoError(t, err)

			if (batchNum+1)%2 == 0 || batchNum == totalBatches-1 {
				t.Logf("  Progress: OS %d - batch %d/%d completed", osID, batchNum+1, totalBatches)
			}
		}
	}
}

// createCVEMetadata creates CVE metadata entries
func createCVEMetadata(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()
	mockTime := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)

	// Create metadata in batches of 500 for better performance
	batchSize := 500
	totalBatches := (numCVEs + batchSize - 1) / batchSize
	t.Logf("  Creating CVE metadata in %d batches", totalBatches)

	for start := 0; start < numCVEs; start += batchSize {
		end := start + batchSize
		if end > numCVEs {
			end = numCVEs
		}

		batch := make([]fleet.CVEMeta, end-start)
		for i := start; i < end; i++ {
			// Alternate between 2022 and 2023 CVEs to match our created vulnerabilities
			year := 2023
			if i >= 50000 {
				year = 2022
			}

			// Use 5-digit format to match our CVE creation
			cve := fmt.Sprintf("CVE-%d-%05d", year, i%100000)

			// 30% have CISA known exploit
			cisaExploit := (i % 10) < 3

			batch[i-start] = fleet.CVEMeta{
				CVE:              cve,
				CVSSScore:        ptr.Float64(5.0 + float64(i%50)/10.0),
				EPSSProbability:  ptr.Float64(float64(i%100) / 100.0),
				CISAKnownExploit: ptr.Bool(cisaExploit),
				Published:        ptr.Time(mockTime.Add(time.Duration(i) * time.Hour)),
				Description:      fmt.Sprintf("Test vulnerability %s", cve),
			}
		}

		err := ds.InsertCVEMeta(ctx, batch)
		require.NoError(t, err)

		// Report progress every 10 batches
		batchNum := (start / batchSize) + 1
		if batchNum%10 == 0 || batchNum == totalBatches {
			t.Logf("  Progress: %d/%d batches completed (%d CVEs)", batchNum, totalBatches, end)
		}
	}
}

// getHostIDs extracts host IDs from a slice of hosts
func getHostIDs(hosts []*fleet.Host) []uint {
	ids := make([]uint, len(hosts))
	for i, h := range hosts {
		ids[i] = h.ID
	}
	return ids
}

// CreateMySQLDSForBenchmark creates a datastore for benchmarking
func CreateMySQLDSForBenchmark(b *testing.B) *Datastore {
	return CreateMySQLDS(b)
}
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance Improvements**
* Faster loading of the vulnerabilities list via optimized database
queries for the vulnerabilities API endpoint.

* **Bug Fixes**
* More accurate “created at” timestamps for vulnerabilities, improving
sorting and consistency.
* More consistent source attribution for vulnerabilities when multiple
sources are available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-17 09:57:47 -05:00
RachelElysiaandCarlo DiCelico 1ef91fe4e3 Feature: Script only package e2e followup (#34271)
Co-authored-by: Carlo DiCelico <carlo@fleetdm.com>
2025-10-17 10:54:00 -04:00