Commit Graph
19034 Commits
Author SHA1 Message Date
Konstantin Sykulev c42a732ef7 Unbatching checksum migration (#34586)
**Related issue:** https://github.com/fleetdm/fleet/issues/34559
PR for 4.75.1: https://github.com/fleetdm/fleet/pull/34485

needs to be added in main to be consistent.
2025-10-21 11:21:33 -05:00
Harrison Ravazzolo fdc184fe58 Windows CSP - Spotlight config + Okta scep (#34589) 2025-10-21 08:47:50 -07:00
Noah Talerman 140d1dd92f Fix wording and formatting in product design README (#34592) 2025-10-21 11:42:52 -04:00
Gabe Lopez df176aa047 Update release-qa.md MDM tests (#34470)
added the following to the MDM smoke tests:

- account-driven user enrollments for iOS/iPadOS
- Windows Autopilot enrollments
- Android tests
- Certificate Authorities
- IdP provisioning SCIM
2025-10-21 08:17:36 -07:00
fleet-releaseandmostlikelee 39931ccdec Update Fleet-maintained apps (#34580)
Automated ingestion of latest Fleet-maintained app data.

Co-authored-by: mostlikelee <16102903+mostlikelee@users.noreply.github.com>
2025-10-21 10:36:17 -04:00
Tim Lee 77df7e011c bugfix: correct santa json status based on santa docs (#34561) 2025-10-21 08:26:12 -06:00
Luke Heath 5ad1b1295a Add open positions for QA and Software Specialist (#34547) 2025-10-20 20:03:51 -05:00
Scott Gress b44a9ab6e4 Update script upload UI: updates from QA (#34564)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32632
**Related issue:** Resolves #34454 

# Details

Addresses the following QA-reported issues with #32632:

* Adding a .sh script file does not show it is valid for macOS
<img width="617" height="183" alt="image"
src="https://github.com/user-attachments/assets/39dcf884-ffbc-43b3-a06b-ad88520d6a16"
/>

* With Gitops mode enabled the 'add scripts' button is still clickable
<img width="693" height="151" alt="image"
src="https://github.com/user-attachments/assets/84c6e77d-ed1c-4e8a-8ebd-099d9da5405a"
/>

* Controls > Scripts: "Add script" is missing padding (#34454)
<img width="173" height="79" alt="image"
src="https://github.com/user-attachments/assets/d89cb728-f4ba-427b-868f-f072196777ae"
/>

# Checklist for submitter

For unreleased bug fixes in a release candidate, one of:

- [X] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-20 19:19:40 -05:00
Rachael ShawandNoah Talerman 16478197dc Update ritual for reviewing upcoming release reference docs (#34558)
Missed the step about reverting changes that were already merged into
the docs release branch.

---------

Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
2025-10-20 18:02:10 -04:00
jacobshandling f700d1029e Hide software host count, versions table when no hosts have the software installed (#34484)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32635
Hide host count and version table when no hosts have the software
installed

<img width="1518" height="568" alt="Screenshot 2025-10-17 at 4 36 13 PM"
src="https://github.com/user-attachments/assets/4f77d039-d9d0-427e-a3f4-a8774a3f6ff7"
/>



- [x] Changes file added for user-visible changes in `changes/`
- [x] QA'd all new/changed functionality manually
2025-10-20 14:46:35 -07:00
Ian Littman 75a3af5b43 Don't try to pull config profiles when MDM isn't enabled for any platform when generating GitOps YAML (#34549)
Fixes #33677.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-20 15:47:44 -05:00
George Karr 33650644c8 Update sprint retrospective guidelines with tool recommendations (#34543) 2025-10-20 15:22:43 -05:00
dependabot[bot] 5b57843922 Bump playwright from 1.39.0 to 1.56.1 (#34548) 2025-10-20 15:21:33 -05:00
malomar124 14f6697c6d Update README.md (#34538) 2025-10-20 15:03:19 -05:00
Scott GressandIan Littman 6e0235cb31 Update test file to allow Go to zip module (#34554)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** For #6994

This is a non-functional change to the `generate-gitops` tests that
renames a testfile to not contain an emoji, because Go can't zip and
publish modules that have filenames with emojis in them.

The actual output from the `generate-gitops` command _can_ contain a
file that has an emoji in its name, representing a team with an emoji in
its name. I updated the test here to translate the test filename (now
without emoji) to the actual expected filename (with the emoji).

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2025-10-20 14:50:13 -05:00
Rachael Shaw 688f50b619 Update product design rituals (#34482)
+ Remove "🦢🚀🤝 Product Design + Frontend Engineering" ritual
+ Formalize the rituals for release docs: add post-sprint-kickoff,
pre-release, and post-release documentation tasks
2025-10-20 15:47:58 -04:00
Sam Pfluger b49e7826d4 Fix links in communications.md for writing section (#34546) 2025-10-20 14:01:30 -05:00
Noah Talerman 49a458a8ca Add Fleet UI refresh details to release notes (#34536) 2025-10-20 12:36:12 -05:00
fleet-releaseandmostlikelee 81d963cc4a Update Fleet-maintained apps (#34498)
Automated ingestion of latest Fleet-maintained app data.

Co-authored-by: mostlikelee <16102903+mostlikelee@users.noreply.github.com>
2025-10-20 09:53:22 -04:00
Luke Heath 2c8ae8cc78 Adding changes for Fleet v4.75.0 (#33583) (#34483) 2025-10-17 21:51:17 -05:00
jacobshandling f80d8a5eb8 Remove unused mock endpoint handler (#34301) 2025-10-17 16:10:21 -07:00
Ian Littman e995891359 Revise 4.76 migrations to land after all 4.75 migrations (#34479)
- [x] QA'd locally blank -> 4.75 -> this branch
2025-10-17 17:55:57 -05:00
Noah Talerman 9318d26e90 Release article: 4.75 (#34390) 2025-10-17 17:46:50 -05:00
+3 7ed4aac2b8 Docs v4.75.0 (#34443)
Documentation changes for the 4.75 release

---------

Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Magnus Jensen <magnus@fleetdm.com>
Co-authored-by: Jordan Montgomery <elijah.jordan.montgomery@gmail.com>
Co-authored-by: Tim Lee <timlee@fleetdm.com>
Co-authored-by: Janis Watts <184028114+jmwatts@users.noreply.github.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
2025-10-17 17:45:52 -05:00
jacobshandling c9a14d5038 Update vuln processing guide, add change file for JetBrains IDE extension vulns (#34478)
Addresses #32266 

Update docs, add change file
2025-10-17 17:39:21 -05:00
Noah Talerman 93acdb8e8c Product Design handbook: Stories that don't require product changes (#34463) 2025-10-17 17:11:01 -05:00
Noah Talerman 19899d48d1 Product Design rituals: Revise sprint kickoff review and add sprint retro task (#34477)
- Separate sprint kickoff reviews for @marko-lisica (#g-mdm +
#g-software) and @rachaelshaw (#g-orchestration +
#g-security-compliance)
  - Up to the Product Designer to reschedule as needed
- Sprint retro is its own call
2025-10-17 18:01:38 -04:00
Eric 351ac2b845 Website: Update created account record type (#34473)
Changes:
- Updated the `update-or-create-contact-and-account` helper to set a
`Type` on new account records created.
2025-10-17 16:40:21 -05:00
Ian LittmanandScott Gress 52410a3328 Add CPE translations to map various software where name/bundle ID based matching doesn't work (#34233)
Fixes #34282. Must be QA'd on-branch so split from the software name
cleanup PR for #34159.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [x] Added/updated automated tests

---------

Co-authored-by: Scott Gress <scott@fleetdm.com>
2025-10-17 16:09:14 -05:00
jacobshandling d9cedcba8d Consolidate rolling linux version possibilities (#34467)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34214

- [x] manual QA by @sgress454
2025-10-17 14:01:03 -07:00
RachelElysia 69494cc0d1 Fleet UI: Remove purple count and 2 links (#34462) 2025-10-17 15:56:13 -05:00
Luke Heath aba434b024 Bump font size variable from 13px to 14px and correct page header gap (#34453) 2025-10-17 15:00:14 -05:00
Jordan Montgomery 23a4b781fd ISII: Make setup_experience gitops flag on app_store_apps enable setup for all platforms an app supports (#34461)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #30890 

Unreleased bug(or requirements miss) in iOS/iPadOS setup experience. Per
@noahtalerman we want setup_experience on VPP apps to apply to all
platforms an app is available for:
https://github.com/fleetdm/fleet/issues/30890#issuecomment-3408445511

This will change with the upcoming inclusion of the `platform` field and
it will be possible to specify settings for each individual OS

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-10-17 15:16:34 -04:00
jacobshandling 1c0b004676 Server changes for matching JetBrains IDE vulnerabilities (#34459)
## Addresses #32266 

These are the same changes present in
https://github.com/fleetdm/fleet/pull/34331, excluding changes that
would instantly be released. This separation will allow these
already-approved server changes to be merged before the cut while
allowing more time for QA to get to the QA-on-branch parts of the issue
2025-10-17 11:50:29 -07:00
Jordan Montgomery eafade0a12 Remove Setup Experience preview for iOS/iPadOS (#34458)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33711 

See Noah comment here for why we are removing:
https://github.com/fleetdm/fleet/issues/30890#issuecomment-3411319996

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] QA'd all new/changed functionality manually
2025-10-17 13:57:25 -04:00
Sarah Gillespie e11ddc9866 Support $FLEET_VAR_SCEP_RENEWAL_ID in OU field (#34403) 2025-10-17 12:47:04 -05:00
RachelElysia bc837a7bd3 Fleet UI: Undo filter same row as header name (#34457) 2025-10-17 12:46:46 -05:00
Noah Talerman 5b78b69a72 Roadmap preview: October 2025 (#33179) 2025-10-17 13:24:00 -04:00
Rachael Shaw 047f1f8a96 Make Dale CODEOWNER of /docs/solutions (#34455) 2025-10-17 12:15:57 -05:00
Gabriel Hernandez af254bd43b fix unlock action and refresh host state for ios/ipad (#34444) 2025-10-17 17:59:22 +01:00
Scott Gress 81f589d661 Update add script UI (#34349)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32632

# Details

This PR updates the Script Library page in the following ways:
* When no scripts are uploaded for a team, it shows the "Add script" UI
with a button that opens a new "Add Script" modal
* When scripts are uploaded, the "Add script" button is instead added to
the header of the scripts list, and clicking it opens that modal

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests
working on this

- [X] QA'd all new/changed functionality manually
- [X] Test empty state: go to controls/scripts/library for a team with
no scripts. Clicking "upload" button in empty state should open the add
script modal.
- [X] In the modal, select a .ps1 script. Should not see additional
text.
- [X] Close modal without uploading. Re-open. File field should be
cleared & upload button visible again.
- [X] Select a .sh script. Should see additional text about macOS and
Linux.
  - [X] Add script. Make sure script saves and modal closes.
- [X] Once script has been added, make sure empty state is gone and "Add
script" button is at the top of the list.
- [X] Go to /controls/os-settings/custom-settings for a team with no
profiles uploaded. Make sure empty state text styles match the empty
state for script uploads.
- [X] Open modal to add profile. Make sure upload text styles match the
script upload modal.
- [X] Enable GitOps mode. Go to controls/scripts/library for a team with
scripts added. Make sure new "Add script" button is disabled w/ standard
tooltip in GitOps mode.

Scripts empty state:

<img width="697" height="352" alt="image"
src="https://github.com/user-attachments/assets/32f0f246-bddb-4bb7-bc39-48d9978de9fa"
/>

Scripts uploader:

<img width="745" height="590" alt="image"
src="https://github.com/user-attachments/assets/f82414e2-9318-4543-b5ca-41e759662587"
/>

Scripts uploader with .sh

<img width="750" height="539" alt="image"
src="https://github.com/user-attachments/assets/0b989067-921a-4d18-93ed-09aac90fc9cb"
/>

Scripts table:

<img width="686" height="256" alt="image"
src="https://github.com/user-attachments/assets/848f1b56-6e9e-48d4-9a03-6fdf5427301e"
/>

Profiles empty state:

<img width="700" height="377" alt="image"
src="https://github.com/user-attachments/assets/8f92bcd9-2215-41f6-a540-4774f7e9542b"
/>

Profiles uploader:

<img width="707" height="682" alt="image"
src="https://github.com/user-attachments/assets/eef216af-3447-48e7-882a-e42e888e1c17"
/>
2025-10-17 10:49:59 -05:00
jacobshandling 4dd9fe822e Improve alignment and logic around Arch linux os version displays (#34420)
**Related issue:** Resolves #34214 

- Alignment in host summary fixed for both Firefox(L) and Chrome(R):
<img width="1728" height="221" alt="Screenshot 2025-10-16 at 6 38 39 PM"
src="https://github.com/user-attachments/assets/27d4489d-7a90-41bd-9b7d-53dfc95e4000"
/>

- Fix detection logic in Software > OS Versions table, render
tooltip-wrapped "rolling" there when appropriate

- [x] QA'd all new/changed functionality manually
2025-10-17 08:26:42 -07:00
jacobshandlingandJacob Shandling 5f626e2a8c Add gigs_all_disk_space vital collection, storage, service, and UI rendering for Linux hosts (#34077)
## Addresses #31671 

- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## Database migrations

- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added total disk space metrics for all partitions on Linux hosts. The
disk space indicator now displays comprehensive storage information
including root partition and all other partitions, improving visibility
into host storage capacity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
2025-10-17 08:24:23 -07:00
Victor Lyuboslavsky 9295a82e83 Improved MySQL query performance software versions and vulnerabilities endpoints (#34262)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #32178

Software optimization: skipping an unnecessary software_cve join when
vulnerability details are not needed. Vulnerabilities are still
returned, so functionality remains unchanged.

Vulnerabilities optimization: Query vulnerability_host_counts directly
and LEFT JOIN for metadata. This eliminates the expensive UNION of all
CVE rows that was causing performance issues.

Previous approach: UNION all CVEs (many rows) → JOIN
vulnerability_host_counts → filter
New approach: Start with filtered vulnerability_host_counts → LEFT JOIN
for metadata
This reduces the working set before any expensive operations

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [ ] Added/updated automated tests (see below for the test)
- [x] QA'd all new/changed functionality manually
- Planning to test in loadtest after also improving the software
endpoint

Performance test for replicating the problem and testing the fix:
```go
package mysql

import (
	"context"
	"fmt"
	"testing"
	"time"

	"github.com/fleetdm/fleet/v4/server/fleet"
	"github.com/fleetdm/fleet/v4/server/ptr"
	"github.com/fleetdm/fleet/v4/server/test"
	"github.com/stretchr/testify/require"
)

// TestListVulnerabilitiesPerformance is a performance test that replicates
// the production performance problem with ListVulnerabilities.
//
// This test creates a realistic dataset with thousands of CVEs and measures
// query performance under various conditions. Run with:
//
//	go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
//
// To see detailed timing output, set the environment variable:
//
//	VERBOSE=1 go test -v -run TestListVulnerabilitiesPerformance ./server/datastore/mysql
func TestListVulnerabilitiesPerformance(t *testing.T) {
	if testing.Short() {
		t.Skip("skipping performance test in short mode")
	}

	ds := CreateMySQLDS(t)
	defer TruncateTables(t, ds)

	ctx := context.Background()

	// Create a realistic dataset
	t.Log("Setting up test data...")
	setupPerformanceTestData(t, ds)
	t.Log("Test data setup complete")

	// Test cases covering common query patterns
	testCases := []struct {
		name string
		opts fleet.VulnListOptions
	}{
		{
			name: "Global list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - first page, sorted by host count",
			opts: fleet.VulnListOptions{
				IsEE:   true,
				TeamID: ptr.Uint(1),
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Team 1 list - with exploit filter",
			opts: fleet.VulnListOptions{
				IsEE:         true,
				TeamID:       ptr.Uint(1),
				KnownExploit: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - with CVE search",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					MatchQuery:     "2023",
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Global list - second page",
			opts: fleet.VulnListOptions{
				IsEE: true,
				ListOptions: fleet.ListOptions{
					Page:           1,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
		{
			name: "Free version - global list",
			opts: fleet.VulnListOptions{
				IsEE: false,
				ListOptions: fleet.ListOptions{
					Page:           0,
					PerPage:        20,
					OrderKey:       "hosts_count",
					OrderDirection: fleet.OrderDescending,
				},
			},
		},
	}

	// Run performance tests
	for _, tc := range testCases {
		t.Run(tc.name, func(t *testing.T) {
			// Warm up the query cache
			_, _, err := ds.ListVulnerabilities(ctx, tc.opts)
			require.NoError(t, err)

			// Measure query performance
			const iterations = 5
			var totalDuration time.Duration

			for i := 0; i < iterations; i++ {
				start := time.Now()
				vulns, meta, err := ds.ListVulnerabilities(ctx, tc.opts)
				duration := time.Since(start)
				totalDuration += duration

				require.NoError(t, err)
				require.NotNil(t, meta)
				require.NotEmpty(t, vulns, "expected vulnerabilities to be returned")

				if i == 0 {
					t.Logf("  First run: %v (returned %d results)", duration, len(vulns))
				}
			}

			avgDuration := totalDuration / iterations
			t.Logf("  Average of %d runs: %v", iterations, avgDuration)

			// Performance assertions
			// These thresholds represent the current performance problem
			// After optimization, these should be reduced significantly
			if avgDuration > 2*time.Second {
				t.Logf("  ⚠️  WARNING: Query took %v (>2s) - performance issue detected", avgDuration)
			} else if avgDuration > 500*time.Millisecond {
				t.Logf("  ⚠️  SLOW: Query took %v (>500ms)", avgDuration)
			} else {
				t.Logf("  ✓ GOOD: Query took %v (<500ms)", avgDuration)
			}
		})
	}

	// Test count query performance
	t.Run("Count vulnerabilities performance", func(t *testing.T) {
		opts := fleet.VulnListOptions{
			IsEE: true,
		}

		// Warm up
		_, err := ds.CountVulnerabilities(ctx, opts)
		require.NoError(t, err)

		// Measure
		const iterations = 5
		var totalDuration time.Duration

		for i := 0; i < iterations; i++ {
			start := time.Now()
			count, err := ds.CountVulnerabilities(ctx, opts)
			duration := time.Since(start)
			totalDuration += duration

			require.NoError(t, err)
			require.Greater(t, count, uint(0))

			if i == 0 {
				t.Logf("  First run: %v (count=%d)", duration, count)
			}
		}

		avgDuration := totalDuration / iterations
		t.Logf("  Average of %d runs: %v", iterations, avgDuration)

		if avgDuration > 2*time.Second {
			t.Logf("  ⚠️  WARNING: Count query took %v (>2s)", avgDuration)
		} else if avgDuration > 500*time.Millisecond {
			t.Logf("  ⚠️  SLOW: Count query took %v (>500ms)", avgDuration)
		} else {
			t.Logf("  ✓ GOOD: Count query took %v (<500ms)", avgDuration)
		}
	})
}

// BenchmarkListVulnerabilities provides benchmark results for ListVulnerabilities.
// Run with:
//
//	go test -bench=BenchmarkListVulnerabilities -benchmem -run=^$ ./server/datastore/mysql
func BenchmarkListVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()

	// Setup test data
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	// Benchmark the most common query pattern
	opts := fleet.VulnListOptions{
		IsEE: true,
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkListVulnerabilitiesWithTeam benchmarks team-specific queries
func BenchmarkListVulnerabilitiesWithTeam(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE:   true,
		TeamID: ptr.Uint(1),
		ListOptions: fleet.ListOptions{
			Page:           0,
			PerPage:        20,
			OrderKey:       "hosts_count",
			OrderDirection: fleet.OrderDescending,
		},
	}

	for i := 0; i < b.N; i++ {
		_, _, err := ds.ListVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// BenchmarkCountVulnerabilities benchmarks the count query
func BenchmarkCountVulnerabilities(b *testing.B) {
	ds := CreateMySQLDSForBenchmark(b)
	defer TruncateTables(b, ds)

	ctx := context.Background()
	setupPerformanceTestData(b, ds)

	b.ResetTimer()

	opts := fleet.VulnListOptions{
		IsEE: true,
	}

	for i := 0; i < b.N; i++ {
		_, err := ds.CountVulnerabilities(ctx, opts)
		if err != nil {
			b.Fatal(err)
		}
	}
}

// setupPerformanceTestData creates a realistic dataset that mimics production
// This creates:
// - ~80,000+ unique CVEs (matching production scale)
// - ~73,000 software_cve entries
// - ~35,000 operating_system_vulnerabilities entries
// - Multiple teams
// - Various host counts per vulnerability
//
// Note: This will take several minutes to run but will replicate production performance issues
func setupPerformanceTestData(t testing.TB, ds *Datastore) {
	ctx := context.Background()

	// Create 100 hosts across different teams and OS types (doubled from 50)
	// More hosts = more realistic host count distributions
	hosts := make([]*fleet.Host, 100)
	for i := 0; i < 100; i++ {
		hosts[i] = test.NewHost(t, ds, fmt.Sprintf("host%d", i),
			fmt.Sprintf("192.168.1.%d", i%255+1), // Handle more than 255 hosts
			fmt.Sprintf("key%d", i),
			fmt.Sprintf("uuid%d", i),
			time.Now())
	}

	// Create 3 teams
	team1, err := ds.NewTeam(ctx, &fleet.Team{Name: "Engineering"})
	require.NoError(t, err)

	team2, err := ds.NewTeam(ctx, &fleet.Team{Name: "Sales"})
	require.NoError(t, err)

	team3, err := ds.NewTeam(ctx, &fleet.Team{Name: "Support"})
	require.NoError(t, err)

	// Distribute hosts across teams
	// 40 hosts in team1, 30 in team2, 20 in team3, 10 with no team
	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team1.ID, getHostIDs(hosts[0:40])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team2.ID, getHostIDs(hosts[40:70])))
	require.NoError(t, err)

	err = ds.AddHostsToTeam(ctx, fleet.NewAddHostsToTeamParams(&team3.ID, getHostIDs(hosts[70:90])))
	require.NoError(t, err)

	// Set up OS versions (Windows, macOS, Ubuntu)
	windowsOS := fleet.OperatingSystem{
		Name:     "Microsoft Windows 11 Enterprise",
		Version:  "10.0.22621.2715",
		Arch:     "x86_64",
		Platform: "windows",
	}

	macOS := fleet.OperatingSystem{
		Name:     "macOS",
		Version:  "14.1.2",
		Arch:     "arm64",
		Platform: "darwin",
	}

	ubuntuOS := fleet.OperatingSystem{
		Name:     "Ubuntu",
		Version:  "22.04",
		Arch:     "x86_64",
		Platform: "ubuntu",
	}

	// Assign OS to hosts: 50 Windows, 30 macOS, 20 Ubuntu
	for i := 0; i < 50; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, windowsOS)
		require.NoError(t, err)
	}
	for i := 50; i < 80; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, macOS)
		require.NoError(t, err)
	}
	for i := 80; i < 100; i++ {
		err = ds.UpdateHostOperatingSystem(ctx, hosts[i].ID, ubuntuOS)
		require.NoError(t, err)
	}

	err = ds.UpdateOSVersions(ctx)
	require.NoError(t, err)

	// Create realistic CVE distribution matching production scale
	// In production, we see:
	// - ~73,000 software CVEs
	// - ~35,000 OS CVEs
	// - Many CVEs overlap between software and OS
	// - ~80,000 unique CVEs after deduplication

	// We now create production-scale data:
	// - 50,000 software CVEs (70% of production)
	// - 30,000 OS CVEs (85% of production)
	// - Some overlap to create ~80,000 total CVEs after UNION
	// This should replicate the 500-1000ms+ query times seen in production

	t.Log("Creating software vulnerabilities... (this will take a few minutes)")
	createSoftwareVulnerabilities(t, ds, hosts[:60], 50000)

	t.Log("Creating OS vulnerabilities...")
	createOSVulnerabilities(t, ds, 30000)

	t.Log("Creating CVE metadata...")
	createCVEMetadata(t, ds, 80000)

	t.Log("Updating vulnerability host counts...")
	err = ds.UpdateVulnerabilityHostCounts(ctx, 10)
	require.NoError(t, err)

	t.Log("Setup complete - ready for performance testing")
}

// createSoftwareVulnerabilities creates software entries and their CVEs
func createSoftwareVulnerabilities(t testing.TB, ds *Datastore, hosts []*fleet.Host, numCVEs int) {
	ctx := context.Background()

	// Create more software packages to better distribute CVEs
	softwarePackages := []fleet.Software{
		{Name: "Chrome", Version: "120.0.1", Source: "programs"},
		{Name: "Firefox", Version: "121.0", Source: "programs"},
		{Name: "Node.js", Version: "18.19.0", Source: "programs"},
		{Name: "Python", Version: "3.11.7", Source: "programs"},
		{Name: "Docker", Version: "24.0.7", Source: "programs"},
		{Name: "nginx", Version: "1.24.0", Source: "deb_packages"},
		{Name: "postgresql", Version: "15.5", Source: "deb_packages"},
		{Name: "redis", Version: "7.2.3", Source: "deb_packages"},
		{Name: "mysql", Version: "8.0.35", Source: "deb_packages"},
		{Name: "git", Version: "2.43.0", Source: "deb_packages"},
		{Name: "openssl", Version: "3.0.12", Source: "deb_packages"},
		{Name: "curl", Version: "8.5.0", Source: "deb_packages"},
		{Name: "vim", Version: "9.0", Source: "deb_packages"},
		{Name: "apache2", Version: "2.4.58", Source: "deb_packages"},
		{Name: "php", Version: "8.2.14", Source: "deb_packages"},
	}

	// Install software on hosts
	for i, host := range hosts {
		// Each host gets 5-8 software packages
		numPackages := 5 + (i % 4)
		if numPackages > len(softwarePackages) {
			numPackages = len(softwarePackages)
		}
		hostSoftware := softwarePackages[:numPackages]
		_, err := ds.UpdateHostSoftware(ctx, host.ID, hostSoftware)
		require.NoError(t, err)
	}

	err := ds.SyncHostsSoftware(ctx, time.Now())
	require.NoError(t, err)

	// Create CVEs for software (distributed across 15 software IDs)
	// Each software gets many CVEs to simulate real-world vulnerability distribution
	cvesPerSoftware := numCVEs / 15
	t.Logf("  Creating %d CVEs per software package (15 packages)", cvesPerSoftware)

	for softwareID := uint(1); softwareID <= 15; softwareID++ {
		// Insert CVEs in batches for better performance
		batchSize := 1000
		for batchStart := 0; batchStart < cvesPerSoftware; batchStart += batchSize {
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerSoftware {
				batchEnd = cvesPerSoftware
			}

			for i := batchStart; i < batchEnd; i++ {
				cveNum := int(softwareID-1)*cvesPerSoftware + i
				// Use wider CVE number range to avoid duplicates
				cve := fmt.Sprintf("CVE-2023-%05d", cveNum)

				_, err := ds.InsertSoftwareVulnerability(ctx, fleet.SoftwareVulnerability{
					SoftwareID: softwareID,
					CVE:        cve,
				}, fleet.NVDSource)
				require.NoError(t, err)
			}
		}

		if softwareID%5 == 0 {
			t.Logf("  Progress: %d/%d software packages completed", softwareID, 15)
		}
	}
}

// createOSVulnerabilities creates OS vulnerabilities
func createOSVulnerabilities(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()

	// Create CVEs for each OS type
	// OS ID 1 = Windows, 2 = macOS, 3 = Ubuntu
	osIDs := []uint{1, 2, 3}
	cvesPerOS := numCVEs / len(osIDs)

	t.Logf("  Creating %d CVEs per OS type (3 OS types)", cvesPerOS)

	for _, osID := range osIDs {
		// Insert in batches to avoid memory issues with large slices
		batchSize := 5000
		totalBatches := (cvesPerOS + batchSize - 1) / batchSize

		for batchNum := 0; batchNum < totalBatches; batchNum++ {
			batchStart := batchNum * batchSize
			batchEnd := batchStart + batchSize
			if batchEnd > cvesPerOS {
				batchEnd = cvesPerOS
			}

			vulns := make([]fleet.OSVulnerability, batchEnd-batchStart)
			for i := 0; i < len(vulns); i++ {
				actualIndex := batchStart + i
				// Use different year to avoid overlap with software CVEs
				// (but still create some overlap)
				cveNum := int(osID-1)*cvesPerOS + actualIndex
				cve := fmt.Sprintf("CVE-2022-%05d", cveNum)

				// 10% overlap with software CVEs for realism
				if actualIndex%10 == 0 {
					cve = fmt.Sprintf("CVE-2023-%05d", cveNum)
				}

				vulns[i] = fleet.OSVulnerability{
					OSID: osID,
					CVE:  cve,
				}
			}

			_, err := ds.InsertOSVulnerabilities(ctx, vulns, fleet.MSRCSource)
			require.NoError(t, err)

			if (batchNum+1)%2 == 0 || batchNum == totalBatches-1 {
				t.Logf("  Progress: OS %d - batch %d/%d completed", osID, batchNum+1, totalBatches)
			}
		}
	}
}

// createCVEMetadata creates CVE metadata entries
func createCVEMetadata(t testing.TB, ds *Datastore, numCVEs int) {
	ctx := context.Background()
	mockTime := time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC)

	// Create metadata in batches of 500 for better performance
	batchSize := 500
	totalBatches := (numCVEs + batchSize - 1) / batchSize
	t.Logf("  Creating CVE metadata in %d batches", totalBatches)

	for start := 0; start < numCVEs; start += batchSize {
		end := start + batchSize
		if end > numCVEs {
			end = numCVEs
		}

		batch := make([]fleet.CVEMeta, end-start)
		for i := start; i < end; i++ {
			// Alternate between 2022 and 2023 CVEs to match our created vulnerabilities
			year := 2023
			if i >= 50000 {
				year = 2022
			}

			// Use 5-digit format to match our CVE creation
			cve := fmt.Sprintf("CVE-%d-%05d", year, i%100000)

			// 30% have CISA known exploit
			cisaExploit := (i % 10) < 3

			batch[i-start] = fleet.CVEMeta{
				CVE:              cve,
				CVSSScore:        ptr.Float64(5.0 + float64(i%50)/10.0),
				EPSSProbability:  ptr.Float64(float64(i%100) / 100.0),
				CISAKnownExploit: ptr.Bool(cisaExploit),
				Published:        ptr.Time(mockTime.Add(time.Duration(i) * time.Hour)),
				Description:      fmt.Sprintf("Test vulnerability %s", cve),
			}
		}

		err := ds.InsertCVEMeta(ctx, batch)
		require.NoError(t, err)

		// Report progress every 10 batches
		batchNum := (start / batchSize) + 1
		if batchNum%10 == 0 || batchNum == totalBatches {
			t.Logf("  Progress: %d/%d batches completed (%d CVEs)", batchNum, totalBatches, end)
		}
	}
}

// getHostIDs extracts host IDs from a slice of hosts
func getHostIDs(hosts []*fleet.Host) []uint {
	ids := make([]uint, len(hosts))
	for i, h := range hosts {
		ids[i] = h.ID
	}
	return ids
}

// CreateMySQLDSForBenchmark creates a datastore for benchmarking
func CreateMySQLDSForBenchmark(b *testing.B) *Datastore {
	return CreateMySQLDS(b)
}
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance Improvements**
* Faster loading of the vulnerabilities list via optimized database
queries for the vulnerabilities API endpoint.

* **Bug Fixes**
* More accurate “created at” timestamps for vulnerabilities, improving
sorting and consistency.
* More consistent source attribution for vulnerabilities when multiple
sources are available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-17 09:57:47 -05:00
RachelElysiaandCarlo DiCelico 1ef91fe4e3 Feature: Script only package e2e followup (#34271)
Co-authored-by: Carlo DiCelico <carlo@fleetdm.com>
2025-10-17 10:54:00 -04:00
Jordan Montgomery 927fd1d240 Fix android policy non-compliance errors (#34430)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #34422 

Unreleased bug in 4.75.0 Android Config Profile feature. No changes file
as such

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [x] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [x] Confirmed that the fix is not expected to adversely impact load
test results
2025-10-17 10:27:24 -04:00
Noah Talerman 34049215c7 Tech leads are the DRIs (#34437) 2025-10-17 10:14:37 -04:00
Allen Houchins 6a04a40c9a Refine JIT user provisioning description in SSO docs (#34434)
Clarified explanation of JIT user provisioning and account creation
process.
2025-10-17 09:08:33 -05:00
Noah Talerman 08e5d451ba New design system (#34415) 2025-10-17 10:06:19 -04:00
Scott GressandIan Littman 61970118e9 Stop setup experience on software install failure (#34173)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #33173
**Related issue:** Resolves #33111 

# Details

This is the remaining work to implement the "Stop the setup experience
when required software fails to install" feature. This didn't turn out
to be quite as straightforward as expected so I ended up doing a bit of
design-by-code and expect some feedback on the approach. I tried to make
it as low-touch as possible. The general design is:

1. In the `maybeUpdateSetupExperienceStatus` function which is called in
various places when a setup experience step is marked as completed, call
a new `maybeCancelPendingSetupExperienceSteps` function if the setup
step was marked as failed. Similarly call
`maybeCancelPendingSetupExperienceSteps` if a VPP app install fails to
enqueue.
2. In `maybeCancelPendingSetupExperienceSteps`, check whether the
specified host is MacOS and whether the "RequireAllSoftwareMacOS" flag
is set in the team (or global) config. If so, mark the remaining setup
experience items as canceled and cancel any upcoming activities related
to those steps.
3. On the front-end, if the `require_all_software_macos` is set and a
software step is marked as failed, show a new failure page indicating
that setup has failed and showing details of the failed software.
4. On the agent side, when checking setup experience status, send a
`reset_after_failure` flag _only the first time_. If this flag is set,
then the code in the `/orbit/setup_experience/status` handler will clear
and re-queue any failed setup experience steps (but leave successful
steps to avoid re-installing already-installed software). This
facilitates re-starting the setup experience when the host is rebooted.

I also updated the way that software (packages and VPP) is queued up for
the setup experience to be ordered alphabetically, to make it easier to
test _and_ because this is a desired outcome for a future story. Since
the order is not deterministic now, this update shouldn't cause any
problems (aside from a couple of test updates), but I'm ok taking it out
if desired.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)

## Testing

- [X] Added/updated automated tests
* Added a new integration test for software packages, testing that a
failed software package causes the rest of the setup experience to be
marked as failed when `require_all_software_macos` is set, and testing
that the "reset after failure" code works.
* Added a new integration test for VPP packages, testing that a failed
VPP enqueue causes the same halting of the setup experience.
I _don't_ have test for a failure _during_ a VPP install. It should go
through the same code path as the software package failure, so it's not
a huge gap.

- [ ] QA'd all new/changed functionality manually
Working on it 

## fleetd/orbit/Fleet Desktop

- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [X] Verified that fleetd runs on macOS, Linux and Windows


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- New Features
- Configurable option to halt macOS device setup if any software install
fails.
- Device setup page now shows a clear “Device setup failed” state with
expandable error details when all software is required on macOS.
- Improvements
- Setup status now includes per-step error messages for better
troubleshooting.
- Pending setup steps are automatically canceled after a failure when
applicable, with support to reset and retry the setup flow as
configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Ian Littman <iansltx@gmail.com>
2025-10-17 08:38:53 -05:00