4e6591e09dc7c043b58d418fa367ef1833528511
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Scott Gress
|
20af889c56 | Add regression test for #46604 (#46613) | ||
|
Scott Gress
|
997a4097c4 | Add docs for chart bounded context (#47877) | ||
|
Scott Gress
|
6336443f37 | Report mobile devices in "hosts online" (#47222) | ||
|
Scott Gress
|
82f7405f19 | Allow setting default vuln chart filters via GitOps (#47634) | ||
|
Scott Gress
|
0301aea831 |
Add more filtering to Vulnerability Exposure chart (frontend) (#47674)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44746 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [x] Changing the filters in the UI causes the related API params to be set - [x] Changing the software filters causes the "filtered" tooltip to show up and include info about software filters - [x] Changing the host filters causes the "filtered" tooltip to show up and include info about host filters - [x] Changing both host and software filters causes the "filtered" tooltip to show up and include info about both filters - [x] CVE search works and utilizes infinite scroll <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added software category filtering options to vulnerability charts. * Added EPSS range filtering with validation to refine results. * Added known exploit toggle and CVE exclusion capabilities. * Improved filter status display with tabbed interface. * **Tests** * Added comprehensive test coverage for software filtering and validation logic. * **Style** * Enhanced filter UI styling and interactivity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
fcd99a1842 |
Fix GenerateHostStatusStatistics query so that hosts enrolled chart is accurate (#47791)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #47605 # Details The queries in GenerateHostStatusStatistics (which feeds the host_summary API) were incorrectly filtering out hosts that had been removed from ABM from the denominator (i.e. "total hosts") count, while keeping them in the per-platform counts. This PR fixes the query so that the sum of the platforms matches the total. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually **Before** <img width="499" height="420" alt="image" src="https://github.com/user-attachments/assets/9b8d22bc-1dac-49e6-9d2f-8ce9ffda5f17" /> **After** <img width="501" height="420" alt="image" src="https://github.com/user-attachments/assets/fe3705b7-9a90-4d9c-a9dc-b04b2905f3fb" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed an issue causing incorrect platform percentage breakdowns in the "Hosts enrolled" dashboard chart. * Corrected host status calculations to ensure accurate total and per-platform host counts by properly handling device enrollment assignment records. * **Tests** * Added regression tests to validate the correct handling of device enrollment records in host status calculations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
208715e2c8 |
Update some GitOps error messages for clarity (#47134)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45639 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [x] Added/updated automated tests - [ ] QA'd all new/changed functionality manually A bit hard to replicate these ones, but they're text changes only. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved error message clarity for macOS setup assistant and bootstrap package workflows, including more precise identification of the failed operation (such as verifying or uploading) and better details for script-reading failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c370a9672b |
Add CVE chart filtering and non-critical CVE data collection (backend) (#47470)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44746 # Details * Adds the ability to filter historical CVE data by software type, EPSS, CVSS, CVE ID (exclude only) and "has known exploit" * Hard-codes the CVSS filter to 9.0+ for now, since that's the only data that's been collected thus far * Un-gates the collection code so that it will collect CVE data for _all_ severities (but still in the restricted set of software) Related PRs [update the front-end](https://github.com/fleetdm/fleet/pull/47674) to allow sending these filters, and [update GitOps](https://github.com/fleetdm/fleet/pull/47634) to allow changing the default filters. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually ### Manual test plan — CVE chart filtering (backend smoke test) #### Setup - Premium dev server running with a few hosts carrying vulnerable software (so `cve_meta` / `software_cve` / `operating_system_vulnerabilities` are populated) - Chart data present — collector ran once, or seeded: `go run ./tools/charts-backfill --dataset cve --use-tracked-cves --days 7` - API token exported and helper set: ```bash BASE=https://localhost:8080/api/v1/fleet/charts peak() { curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/$1" | jq '[.data[].value] | max'; } #### Checks (compare against the no-filter baseline) - [x] Baseline returns data — GET /charts/cve?days=7 returns a data series; .filters is empty/default - [x] Severity force-pinned to critical — cve?days=7 and cve?days=7&severity_min=0&severity_max=10 give identical peaks (no low-severity leak; client severity ignored) - [x] Category narrowing — software_categories=browsers ≤ baseline; software_categories=os,browsers,office,adobe == baseline - [x] OS category includes kernel — software_categories=os returns OS-vuln + Linux-kernel CVE counts - [x] Known-exploit narrowing — known_exploit=true ≤ baseline - [x] EPSS narrowing — epss_min=0.9 ≤ baseline; epss_min=0&epss_max=1 == baseline (EPSS is 0.0–1.0 on the API) - [x] Exclude is subtractive + tolerant — excluding a visible CVE lowers/keeps counts; exclude_cves=CVE-0000-00000 == baseline (no-op) - [x] Filters echo back — filtered requests return applied values under .filters - [x] Uptime untouched — GET /charts/uptime?days=7 returns its normal series - [x] Free-tier safety (optional) — on non-Premium, /charts/cve returns an empty series, no error - [x] > 0 rows from: SELECT COUNT(DISTINCT scd.entity_id) AS below_critical FROM host_scd_data scd JOIN cve_meta cm ON cm.cve = scd.entity_id WHERE scd.dataset='cve' AND cm.cvss_score < 9.0; - (confirms lower-severity CVEs are stored) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary of changes * **New Features** * Added advanced CVE chart request filters: software categories, known-exploit flag, EPSS min/max, severity min/max, and excluded CVEs. * Expanded CVE chart coverage to use the full “collectible” CVE set, with filtering applied when serving chart data. * **Tests** * Added coverage for collecting collectible CVEs and resolving chart entities based on filter combinations and exclusions. * **Chores** * Updated CVE chart backfill to use collectible CVE discovery. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
1c2b229805 |
Add current-timeframe border and 'No data' tooltip to checkerboard chart (#47812)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #47811 # Details - Outline the slot containing 'now' (the timeframe still being collected) with a fleet-black-50 border. - Show 'No data' instead of a host count in the tooltip for the current and future timeframes. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually <img width="708" height="416" alt="image" src="https://github.com/user-attachments/assets/2878c72b-6c56-4302-b77f-2c9ebdaf9c8c" /> <img width="706" height="412" alt="image" src="https://github.com/user-attachments/assets/a17a37a3-7264-42aa-9880-db3c68946c56" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced checkerboard graph visualization to clearly distinguish current time slots from future (uncollected) time slots with distinct visual styling * Updated tooltips and accessibility labels to display "No data" for future time periods * **Tests** * Added test coverage for current and future time cell behavior <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott GressandCopilot Autofix powered by AI
|
6f43df7305 |
Add custom hooks to frontend patterns doc (#46397)
As per our discussion of https://gist.github.com/sgress454/1e956cbaa2cea3d70c25fc14d4cbc4e7 and the TODO item in https://docs.google.com/document/d/1cuAP-HWcu0KoK5bMmNsax-0-H819OW2y50DUVYoICX4/edit?tab=t.0, this adds a custom hooks section to our patterns doc. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Scott Gress
|
ae3ff6081c |
Fix issue exporting CSV when "agent" field is present (#47125)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #47085 # Details When "Agent" columns is visible in the hosts table, using the "export" feature caused a 400 because there's no "Agent" field in the struct with a `csv` tag. This PR fixes the issue by exporting both osquery and orbit version columns in the CSV, which are the constituent fields that make up "Agent". # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [ ] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] on main, got the 400 trying to export hosts list with the "Agent" column present - [X] on this branch, exporting with "Agent" present results in the "osquery version" and "orbit version" columns in the exported CSV. For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved hosts CSV export: empty and selection-only columns are excluded and the display-only "agent" column is replaced by the actual backend version fields, yielding cleaner, more accurate exports. * **Tests** * Added automated test to verify the CSV export requests the correct columns and prevents download-side side effects. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott GressandLucas Manuel Rodriguez
|
ffbbb9e866 |
Validate SSO settings correctly for GitOps (#46487)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43371 # Details * Ensures that if `enable_sso: true` is set in a global config, then all required sso keys (`entity_id`, `idp_name` and one of `metadata`/`metadata_url`) are provided * Ensures that if `end_user_authentication: true` is set on a fleet, then all required sso keys (`entity_id`, `idp_name` and one of `metadata`/`metadata_url`) are provided, _even if the fleet's config file is not provided in the gitops run_. * Ensures that if `end_user_authentication: true` is set in a fleet config in a gitops run, then all required sso keys (`entity_id`, `idp_name` and one of `metadata`/`metadata_url`) are provided, _even if the global config file is not provided in the gitops run_. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually ### Org SSO — gitops client validation (`fleetctl gitops`) - [x] `enable_sso: true` with **empty `metadata` and `metadata_url`** → fails (metadata-or-url) - [x] `enable_sso: true` with **empty `idp_name`** → fails (idp_name) - [x] `enable_sso: true` with **empty `entity_id`** → fails (entity_id) - [x] Multiple fields missing at once → **one error line per missing field** - [x] `enable_sso: true` + complete IdP (`metadata_url`) → succeeds - [x] `enable_sso: true` + complete IdP using inline `metadata` (no url) → succeeds - [x] `enable_sso: false` + empty IdP fields → succeeds - [x] `sso_settings` key **omitted entirely** → succeeds, and apply **clears** stored SSO - [x] The literal `generate-gitops` output (`metadata: # TODO: ...`) applied as-is → **rejected** ### MDM EUA — gitops group cross-file validation - [x] Team file enables EUA **+** global file **omits** the EUA IdP block → fails - [x] **#43371 core repro:** stored team EUA on, file NOT in run, global-only run blanks metadata → fails, names the team - [x] Same but the team's file **is** in the run with EUA `false` → succeeds - [x] EUA disabled everywhere + **empty** stored IdP → succeeds ### `--delete-other-fleets` - [x] Run with `--delete-other-fleets` degrading the IdP while a stored not-in-run team has EUA on → succeeds - [x] Confirm the omitted team is actually deleted on apply - [x] Known corner: `--delete-other-fleets` + omitted ABM/VPP team with EUA on + degraded IdP → fails at apply time ### Server-side backstop (REST API) - [x] `PATCH /config` (overwrite=false), `enable_sso:true`, metadata omitted, existing has metadata → **200**, metadata preserved - [x] `PATCH /config?overwrite=true`, `enable_sso:true` + empty metadata/url → **422** field `metadata` - [x] `?overwrite=true`, metadata_url set, empty `entity_id`/`idp_name` → **422** both `required` - [x] `?overwrite=true`, `enable_sso:false` → **200** (no IdP required when disabled); `sso_settings` omitted entirely → clears (covered by gitops POS-2) ### Server-side EUA (`euaStrict` keyed on incoming global flag only) - [x] `?overwrite=true` + incoming **global** EUA enabled + incomplete IdP → **422** `entity_id`/`idp_name` - [x] `?overwrite=true` + global EUA **off** + stored team EUA + payload degrades IdP → **succeeds** (via gitops #43371-OVERRIDE) - [x] `?overwrite=true` + global EUA off + payload **fully clears** IdP while a team has EUA → **422** `end_user_authentication` (IsEmpty guard) ### Regression / false-positive guards - [x] Multi-file gitops `--dry-run` configuring IdP AND enabling team EUA (empty stored IdP) → dry-run passes (EE dry-run skip) - [x] A previously-working gitops run with a complete SSO/EUA config → still applies cleanly ### End state verification - [x] After any **rejected** run, stored SSO/EUA config **unchanged** - [ ] After a valid complete-IdP run, SSO login + ADE/EUA enrollment works end-to-end (live device) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * GitOps now validates SSO and MDM end-user authentication (EUA) configs before applying changes, rejecting incomplete settings when SSO/EUA are enabled globally or for any team. Overwrite (GitOps) mode enforces stricter validation than standard updates; dry-run behavior adjusted to avoid spurious EUA rejections. * **Tests** * Added comprehensive tests covering SSO/EUA validation, overwrite vs patch semantics, cross-file EUA scenarios, and delete-other-fleets behavior. * **Refactor** * Reorganized validation and config-parsing helpers for reuse in GitOps checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com> |
||
|
Scott Gress
|
3cd8e118e0 |
Use brightest color for checkerboard when only one is available (#47129)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #47128 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually on main: <img width="499" height="404" alt="Image" src="https://github.com/user-attachments/assets/3d5c3663-61de-444b-ba42-3ebab984d7a3" /> on branch: <img width="493" height="409" alt="image" src="https://github.com/user-attachments/assets/0e7c7fd1-b729-4de8-bb77-41d1e593e926" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed checkerboard visualization color scaling to properly render non-zero data cells at the brightest color level when all non-zero values are identical. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
07df7c5cfd |
Track software deletions in GitOps (#46764)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43729 # Details Adds output to GitOps runs indicating which custom/FMA software packages would be deleted. This involves adding a `deleted_packages` key to the `/software/batch/:request_uuid` ("Get status of software batch-apply request") API, which will be documented separately. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] verified that a GitOps dry run produces one "would've deleted" line per custom package / fma that would be deleted - [X] verified that a GitOps real run produces one "deleted" line per custom package / fma that was deleted - [X] verified that adding software is unaffected <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GitOps batch software operations now report packages pending deletion: dry-runs show "would've deleted" warnings and real runs show deletions; apply flows surface per-package deletion messages. * Empty payload dry-run now still reports pending deletions when applicable. * **Tests** * Added integration and datastore tests validating deletion-warning output, pending-deletion detection, and related result handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
10f65595f8 |
Update error message for GitOps exceptions violations (#46700)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45306 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually <img width="1470" height="19" alt="image" src="https://github.com/user-attachments/assets/726b1efe-176f-4460-a140-a1f571990010" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Enhanced GitOps exception enforcement error messages for labels, secrets, and software to include a direct link to the Fleet settings page where exceptions can be disabled. Users now receive actionable guidance when enforcement is triggered, improving troubleshooting efficiency and reducing time spent resolving configuration issues. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
9cf20fbab3 |
Fix preview config (#46677)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #46560 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [x] Added/updated automated tests - updated preview test. This won't run in CI right now b/c we didn't update fleetctl, but I ran it successfully locally - [X] QA'd all new/changed functionality manually - [x] on main, did `fleetctl preview` with the 4.86.0 tag and verified that charts were disabled - [x] on this branch, did the same and verified charts were enabled <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Dashboard chart data collection (Hosts online and Vulnerability exposure) is no longer disabled when starting preview mode. * **Chores** * Software inventory config moved to the current features flag so historical chart data is preserved. * **Tests** * Added regression checks to ensure uptime, vulnerabilities, and host-users historical data remain enabled in preview. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
6d004b98bc |
Update error message in GitOps when unknown env vars are encountered (#46476)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44053 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Error messages for undefined environment variables in GitOps configurations now include clearer, actionable guidance with examples of how to escape literal dollar-sign syntax (e.g., showing escaped forms). This improves clarity when a variable is missing and helps users distinguish between intended variable references and literal values, reducing confusion. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/46476?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
2bd7fec8a7 |
Handle edge case of adding new fleet + vpp at the same time (#46533)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44444 # Details This PR fixes the following edge case when running `fleetctl gitops`: 1. A new fleet is added 2. That fleet is declared as an ABM default fleet and/or a fleet in a VPP token location 3. The _other_ fleets declared as ABM defaults or VPP fleets are _not_ all provided in the GitOps run In that case, the GitOps run would fail with an error that one of the previously-existing fleets could not be found. This PR fixes the bug by making sure that GitOps looks at both the currently-persisted fleets (via the API) and any fleets that are being created in the current run. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - Reproduced the issue on `main` by attempting to create a new fleet _and_ add it both as an ABM default fleet and to the set of VPP token users in a single run, and getting an error about one of the existing fleets not being found - Verified that I was able to complete a gitops run successfully on this branch with a new fleet as a VPP token user and a default ABM fleet <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Improved validation for Apple Business Manager and Volume Purchasing Program token team assignments with clearer error messages when referenced teams aren't found in Fleet. * Enhanced team name matching to properly handle Unicode characters, ensuring consistent team identification across GitOps configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
851ce172ef | Fix CheckerboardViz relative scale to use cell value instead of percentage (#45791) | ||
|
Scott Gress
|
6f8942f8da |
Skip unneeded query when getting CVE chart (#45813)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45720 # Details When requesting CVE chart data, we were making a call to get the set of tracked CVEs to filter the data by. Currently we're only _collecting_ data for the tracked CVEs, so there's no reason to make this call at all. When we add more filtering options and start collecting more data, we'll need a call like this again, and will likely need to start caching the results. Otherwise it's a multi-second cost per query on large deployments. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - removed some outdated tests - replaced with a test that checks that when an entity filter returns no items (an empty, rather than nil slice) we get empty buckets returned rather than getting data for all entities. This is a regression test for when we add filtering back. - [X] QA'd all new/changed functionality manually - validated that chart still loads and shows the same data. - tried it on a load test env and saw dramatic API request time improvement <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Optimized CVE chart data retrieval by eliminating redundant queries, reducing unnecessary database operations and improving performance. * Fixed entity ID filtering logic to correctly handle edge cases and prevent unintended filter interactions across metrics. * **Tests** * Added test coverage for chart data queries with empty entity filters. * Improved test isolation to ensure metric-specific filtering behavior is properly separated. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45813?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
d7fa35e417 |
Implement roaring bitmaps for historical data collection (#45709)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45715 # Details This PR refactors the way the charts module stores historical data to use the [roaring bitmap](https://github.com/RoaringBitmap/roaring) package instead of saving raw bitmaps. See [this blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives) to learn how roaring compresses data, but TL;DR for our purposes it represents a huge improvement especially for larger deployments where host ID numbers may be very large. In testing, some data was reduced 96%. The majority of the changes in this PR are straight swapping of types from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and updating the internals of our bit math helpers to use roaring methods instead of native AND and OR methods. I've tried to comment on all functional changes. Since the charts have been shipped already, so there will be data in the wild in the prior "dense" format, the code still handles dense bitmaps on _read_, but will always _write_ roaring bitmaps. The majority of the data will therefore have turned over within 30 days on its own, but I plan on a follow-up PR that will transform open rows when the cron runs so that we should be guaranteed to turn over completely within 30 days. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - Tests updated to accommodate the new format, and existing unchanged tests act as proof against regression - [X] QA'd all new/changed functionality manually - Using a tool that dumps the `host_scd_data` rows data into a JSON file (with the keys being entity_id+data and the values being host IDs on that date), compared the data from main branch and this and confirmed they're identical - With a host count of ~9000, some of which have IDs of over 1,000,000, the data storage requirements were: * 82,558,976 bytes for dense * 2,867,200 for roaring (a 96% decrease) For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - should hugely improve - [X] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented roaring bitmaps in historical data collection to optimize bitmap handling for chart data aggregation * Added encoding support to bitmap storage schema for flexible data representation <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c77d1b4ff4 |
allow gitops mode to be set in yaml (#45537)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45330 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests so many - [X] QA'd all new/changed functionality manually - [X] was able to set gitops mode to enabled via `fleetctl gitops` - [X] attempting to set gitops mode w/out repository_url in `fleetctl gitops` failed w/ helpful error - [X] attempting to set gitops mode w/ invalid repository_url in `fleetctl gitops` failed w/ helpful error - [X] attempting to set gitops exceptions in `fleetctl gitops` failed w/ helpful error - [X] was able to unset gitops mode via `fleetctl gitops` - [X] leaving `gitops:` blank in `fleetctl gitops` left the mode untouched (it would retain its previous value) ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [ ] Verified that the setting is exported via `fleetctl generate-gitops` it is not, but it's not a requirement here and leaving it out is a no-op - [x] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) - [ ] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) it is not, nor should it be, as that would clear gitops mode on every customer currently using it - [ ] Verified that any relevant UI is disabled when GitOps mode is enabled n/a, you still need to be able to do gitops mode in the UI <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GitOps mode and repository URL can now be set via GitOps YAML. * **Bug Fixes** * Server preserves existing GitOps settings during config updates; requires repository URL when enabling and rejects unsupported exceptions in GitOps YAML. * **Tests** * Added tests covering apply behavior, YAML validation, activity emission on mode changes, and license-restricted rejection on free tier. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45537) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
af5e240e83 |
Update goreleaser download-artifact action (#45498)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45497 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. not user-facing ## Testing - [ ] Added/updated automated tests - [ ] QA'd all new/changed functionality manually Kinda hard to test outside of a release, but the sibling .msi upload job works using the same action SHA. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated pinned GitHub Action digests used in CI to newer commits for improved security and build reliability. * Reformatted a CI step parameter across multiple YAML lines (no functional change) to improve readability and maintainability. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45498) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
313df2c45a |
Fix checkout action version in fleetctl new template (#45502)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves https://github.com/fleetdm/confidential/issues/15917 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [ ] Added/updated automated tests - I didn't see any tests that checked the contents of the templates directly; will update if anything fails. - [X] QA'd all new/changed functionality manually - Tested on my test gitops repo: https://github.com/sgress454/fleet-gitops-test/actions/runs/25875796027/job/76042556514 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Resolved Node-related warnings that appeared when using the Fleet "new" project and GitOps workflow templates, improving clarity during template execution and initial project setup. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45502) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
4f8737efda |
bump node-sql-parser fork (#45472)
Resolves #33759 Bumps Fleet's `@sgress454/node-sql-parser` to the version that includes https://github.com/sgress454/node-sql-parser/pull/8, with fixes for #33759. |
||
|
Scott Gress
|
2c13d05d79 |
Don't show vulns chart on free tier (#45440)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45434 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [ ] Added/updated automated tests - [X] QA'd all new/changed functionality manually **Free tier** <img width="796" height="500" alt="image" src="https://github.com/user-attachments/assets/d03df638-529b-45b7-ae7d-c07a7f690a29" /> <img width="513" height="425" alt="image" src="https://github.com/user-attachments/assets/ec3e9fc9-9183-41df-83ea-d89ac4bf7c39" /> <img width="507" height="419" alt="image" src="https://github.com/user-attachments/assets/248ceed8-4032-4ca9-b6c5-80e69c2d5a24" /> **Premium tier** <img width="690" height="534" alt="image" src="https://github.com/user-attachments/assets/bea25fc7-a603-406c-936b-9d6743d042ce" /> <img width="501" height="433" alt="image" src="https://github.com/user-attachments/assets/4a09bb9d-7662-438a-9538-2f6d215692ff" /> <img width="499" height="421" alt="image" src="https://github.com/user-attachments/assets/00d8ceb4-4b9e-4432-add2-5e0770be851d" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** - Chart datasets now adapt to subscription tier: standard accounts see a simplified chart (uptime only); premium accounts see additional dataset options. - Vulnerabilities toggle in activity/data retention settings is shown only for premium accounts. * **Tests** - Admin settings tests updated to run with premium-tier context and verify related behaviors. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45440) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
a29ba6befc | Update data collection interval and strategy (#45293) | ||
|
Scott Gress
|
7d88bd0937 |
Remove mobile platforms from chart filter list (#45310)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45290 Just drops the mobile platforms from the platform list in the chart filters modal. You can still find mobile devices by searching for now, as filtering them out is a bigger change requiring either backend work or on-the-glass filtering + pagination which is bug-prone. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated platform filter options in chart filters. iOS, iPadOS, and Android are no longer available as filter options; macOS, Windows, Linux, and ChromeOS remain supported. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45310) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
24e5baf21f |
Only collect data about tracked CVEs (#45247)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45163 # Details Limits CVE data collection to only those CVEs which we report on in the chart. This is a performance optimization necessitated by the large amount of data that bigger fleets may generate. The plan is to implement a data compression strategy so that we can go back to collecting full CVE data soon. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] Ran some collection jobs and verified that only tracked CVEs were represented in "open" rows. - [ ] Ran load test w/ new code For unreleased bug fixes in a release candidate, one of: - [ ] Confirmed that the fix is not expected to adversely impact load test results should improve results! - [X] Alerted the release DRI if additional load testing is needed <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Enhancements** * CVE vulnerability tracking is now scoped to a curated set of critical vulnerabilities, improving the relevance of security impact data displayed across your systems. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45247) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
049f29ee3f |
Apply labels before consumers in GitOps (#45194)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44046 # Details Updates GitOps to ensure that team labels are created before any consumers (e.g. profiles, policies, software) that might use the labels are applied. It does this by adding a new `afterTeamApply` callback option to `ApplyGroup` that is called after team config is applied -- this is when team labels are now created, instead of after `ApplyGroup` runs. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed GitOps deployments failing when a label and a profile that references it are created in the same run. * Ensures the correct apply ordering so dry-run/log output shows label application before profile application. * **Tests** * Added a regression test validating label → profile apply ordering to prevent future regressions. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/45194) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
fe16654729 |
Allow glob literals in filenames (#44547)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43598 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] added a script file `some-*-script[].sh` and referred to it in a gitops file using `path:`. Failed on main; on this branch it successfully uploaded the script - [X] still got expected error message when using `path: ` with a value that had glob characters that _didn't_ match an actual file - [X] `paths:` still worked and uploaded multiple files, including `some-*-script[].sh` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed path validation in fleetctl gitops so path values containing glob metacharacters (e.g., brackets, asterisks, question marks) are accepted when a literal file with that name exists on disk; missing files still produce the appropriate error. * **Tests** * Added regression tests covering glob metacharacter handling in path validation. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/44547) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gressandcoderabbitai[bot]
|
c7364d555c |
Officially deprecated setup_experience.software (#44549)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42503 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] setting `software:` under `macos_setup` or `setup_experience` triggers the expected warning. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Deprecations** * Using setup_experience.software or macos_setup.software now emits a deprecation warning. Migrate by setting setup_experience: true on individual software items (packages, App Store apps, or fleet-maintained apps). * **Tests** * Added test coverage to verify the deprecation warning is emitted when applicable and absent otherwise. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/44549) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Scott Gress
|
348fa47c84 |
Improve name key validation in GitOps (#44553)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves # # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] omitted `name:` from a file without `org_settings:`, got: ``` * No `name` was provided in /tmp/testback/fleets/third-fleet.yml. If this file is intended to define org-level settings, add `org_settings:` as a top-level key. Otherwise, use `name` to specify the fleet name. ``` - [X] omitted `name:` from a file with `org_settings:`, got no error. - [X] omitted `name:` from `no-team.yml`, got: ``` * `name` must be `No Team` for `no-team.yml` ``` - [X] omitted `name:` from `unassigned.yml`, got: ``` * `name` must be `Unassigned` for `unassigned.yml` ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved error messages when GitOps YAML files omit the required `name` field, with specific remediation guidance tailored to each configuration file type * Enhanced validation error messaging when top-level `org_settings` is missing or incorrectly placed, providing clearer instructions on required YAML structure <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
8b779110a8 |
Remove verb and url from gitops errors (#44555)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43721 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually Added a Google Calendar integration to gitops .yml with `client_email` missing from the `api_json_key`. - [X] on main, got error: ``` Error: applying fleet config: PATCH /api/latest/fleet/config received status 422 Validation Failed: client_email is required (API time: 13ms) ``` - [X] on this branch, got: ``` Error: applying fleet config: Validation Failed: client_email is required (API time: 134ms) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Cleaner CLI error messages: removed extraneous HTTP path/status-code details from GitOps-related errors, making output easier to read. * **Tests** * Added tests to verify the improved error message handling and nil/non-wrapped error behavior. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/44555) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f5c59ae3b4 |
Fix google calendar key validation (#44556)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42886 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] gitops run with extra keys (besides `client_email` and `private_key` in `api_key_json` fails on main, passes on this branch - [X] gitops run with missing `client_email` or `private_key` in `api_key_json` still fails gitops (including dry run) - [X] gitops run with extra keys sibling to api_key_json still fails as expected <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Corrected GitOps validation so Google Calendar API key JSON no longer rejects valid nested keys; required-field validation for the integration still enforced. * **Tests** * Added test coverage to ensure nested unknown keys are accepted while sibling-level unknown fields are reported as validation errors. [](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/44556) <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Tim Lee <timlee@fleetdm.com> |
||
|
Scott GressandCopilot Autofix powered by AI
|
4daed869ad |
Add API param linter (#44045)
Adds a linter to ensure we don't add new instances of `team` or `query`
in API params. This will be used incrementally, but this PR also adds
`nolint` directives to places that still have these terms, both to avoid
false-positives later and to help with full migration away from these
terms in in Fleet 5.
Example:
```
server/fleet/campaigns.go:51:16: json tag "team_id": uses deprecated "team"/"teams" — use "fleet"/"fleets" instead (apiparamcheck)
Team *uint `json:"team_id,omitempty"`
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a new static analyzer (apiparamcheck) to flag deprecated API
parameter names ("team/teams") and improper usages of "query/queries".
* **Chores**
* Integrated the new check into CI tooling and configuration.
* Added analyzer tests and plugin registration.
* Applied targeted lint-suppression annotations across code and tests
where legacy parameter names must remain.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
||
|
Scott Gress
|
fadd803793 |
Don't wipe out dataset collection config when not provided in GitOps (#45049)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45042 # Details On Dogfood, we have v4.85.0 server running but we run our GitOps with the currently published fleetctl (4.84). This mismatch caused us to disable (and therefore wipe out data for) both of our historical chart datasets. This PR patches the "update app config" code so that when in "overwrite mode" (i.e. GitOps), it checks for empty `historical_data` keys in the incoming JSON and replaces them with the default values (currently `true`, i.e. "collect the data"). Tested manually (see testing below). # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] reproduced issue on both current fleet v4.85 and main branch servers, using fleetctl v4.84 - [X] on this branch, ran fleetctl v4.84 w/out `historical_data` in gitops and verified that charts were enabled. - [X] on branch applied to 4.85, ran fleetctl v4.84 w/out `historical_data` in gitops and verified that charts were enabled. - [X] disabled one chart in the UI, and verified that updating unrelated app config in the UI did not affect that config (PATCH still works) For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Bug Fixes** * Fixed GitOps configuration handling for historical data settings to properly apply default values when fields are omitted by clients. This ensures that previous configuration settings are preserved correctly in overwrite mode, preventing incorrect defaults from being inadvertently persisted when managing configurations with older clients. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
64a50d0c16 |
Fix relative spread + calendar dates on checkerboard (#44959)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44958 # Details Fixes two issues on the checkerboard: 1. Ensures that the chart shows data going back 30 calendar days (not 720 hours) if it has it 2. Leaves `0` values out of the chart color band calculations in "relative" color mode # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually **Before** Colors clustered in top 3 levels, empty boxes in first column: <img width="705" height="416" alt="image" src="https://github.com/user-attachments/assets/b867a1a9-4c52-4b96-92fd-04e7848c6295" /> **After** Colors spread over all levels, no empty boxes in first column: <img width="707" height="412" alt="image" src="https://github.com/user-attachments/assets/c67e11c5-6dd8-4e66-9c32-9d9213ccb24f" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Charts request an extra day to ensure full calendar-day coverage across timezones. * Checkerboard visualization excludes empty/no-data slots from relative color scaling so color ramps reflect non-zero data. * Calendar view trims leading partial days so the displayed window matches the selected range. * **New Features** * Chart date-range selection expanded to support any value from 1–31 days. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b47b1fc205 |
Update chart styles / fix bugs (#44813)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44676 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [ ] Added/updated automated tests just style fixes, but the ["disable features: frontend" PR](https://github.com/fleetdm/fleet/pull/44797) has some tests - [X] QA'd all new/changed functionality manually - [x] make hosts enrolled chart match height w/ checkerboard - [x] use empty boxes with outline instead of filled boxes for 0 value - [x] use dark-mode colors for green theme - [x] put host enrolled chart bars between horizontal rules instead of on top of them - [x] only show date and # of hosts in tooltip, no percentage or denominator - [X] add day of week to checkerboard tooltip - [x] use comma in tooltip numbers - [x] use relative shading in both datasets - [x] make text size match between hosts enrolled and checkerboard titles - [x] on vuln chart tooltip, add "Want more control over this chart? Comprehensive vulnerability filtering is [coming soon](link to this issue: https://github.com/fleetdm/fleet/issues/44746)." - [x] on vuln chart tooltip, fix link styling in dark mode - [x] change "Hosts active" => "Hosts online" - [x] Update tooltip copy to "The number of hosts detected online during a given hour. A host is considered online if it's actively checking in to Fleet. This includes sleeping hosts (e.g. lid closed)." - [X] Fix issue where at smaller widths, the entire page has a horizontal scrollbar - [X] Make charts stay pegged to the right side when first loaded and when resizing window Charts align: <img width="1223" height="479" alt="image" src="https://github.com/user-attachments/assets/4a7ba859-a0aa-43c6-a06b-4ef8361e2418" /> Data tooltip: <img width="162" height="77" alt="image" src="https://github.com/user-attachments/assets/5b1768e4-0736-4e45-ad49-6faf31958352" /> Vuln tooltip: <img width="387" height="123" alt="image" src="https://github.com/user-attachments/assets/98c2096f-fa4e-4b06-917e-ec6eed844b05" /> Hosts online tooltip: <img width="360" height="113" alt="image" src="https://github.com/user-attachments/assets/bd65cd1d-0f0c-4c72-a31d-ffe602cacd9c" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Optional relative scaling for chart visualizations * Responsive chart sizing in the Hosts Enrolled card * **Improvements** * Dataset label updated to "Hosts online" * Tooltips and labels show formatted host counts with proper pluralization * Day labels now include weekday for clearer dates * Chart descriptions mention upcoming vulnerability filtering * Tighter header, dropdown, scrollbar, and dark‑mode styling * **Tests** * Updated dataset heading test and added a relative‑scale color ramp test <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Rachael Shaw <r@rachael.wtf> Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com> Co-authored-by: Luke Heath <luke@fleetdm.com> |
||
|
Scott Gress
|
684becade8 |
Allow disabling chart datasets: backend (#44769)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44077 # Details This PR implements enforcement of the "disable dataset" feature. When a dataset is disabled globally, we: * Stop collecting all data for that dataset (the `Collect` method for that dataset is not called in the cron job) * Remove all previously-collected data for the dataset via an asynchronous job When a dataset is disabled for one or more fleets, we: * Provide the list of disabled fleets as an argument to each dataset's `Collect` method. Each dataset is responsible for filtering out hosts in the most efficient way possible * Scrub the data for the relevant datasets using a bitmask, so that all hosts from the disabled fleets are removed from the data. This is done via an asynchronous job. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [X] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [ ] QA'd all new/changed functionality manually ### Prerequisites / Test Setup - [ ] Fleet running with at least 3 teams (call them T1, T2, T3) and ≥3 hosts in each, plus ≥2 hosts with no team - [ ] At least one host on each team has reported recent uptime (within the bucket window) - [ ] At least one host in each team is affected by a tracked CVE (so `host_scd_data` for `dataset='cve'` will have non-empty bitmaps) - [ ] AppConfig: both `features.historical_data.uptime` and `features.historical_data.vulnerabilities` start as `true`; same for every team - [ ] Let the collection cron run at least one full tick to populate baseline rows in `host_scd_data` for both `uptime` and `cve` - [ ] Note the current row count per dataset: `SELECT dataset, COUNT(*) FROM host_scd_data GROUP BY dataset;` --- ### 1. Cron Skips Globally-Disabled Datasets #### 1.1 Global disable of `uptime` - [x] Disable globally: `PATCH /api/v1/fleet/config` with `features.historical_data.uptime = false` - [x] Verify activity feed shows `disabled_historical_dataset` for `uptime` (existing behavior) - [x] Wait for next collection tick (or trigger it via fleetctl debug if available) - [x] Confirm **no new rows** appear for `dataset='uptime'`: `SELECT MAX(valid_from) FROM host_scd_data WHERE dataset='uptime';` should not advance after the disable - [x] Confirm cron still writes `cve` rows on the same tick (per-dataset isolation) - [x] Re-enable: PATCH `historical_data.uptime = true` - [x] Verify next tick resumes writing `uptime` rows #### 1.2 Global disable of `vulnerabilities` - [x] Repeat 1.1 with `features.historical_data.vulnerabilities` - [x] Confirm `cve` writes stop, `uptime` continues #### 1.3 Both disabled globally - [x] Disable both globally - [x] Confirm cron tick produces zero new rows for either dataset - [x] Confirm cron does not error or get stuck - [x] Re-enable both --- ### 2. Per-Fleet Disable — Cron Filters at SQL #### 2.1 Single team disabled for one dataset - [x] Disable uptime for T1 only: PATCH team T1 with `features.historical_data.uptime = false` - [x] Verify scoped `disabled_historical_dataset` activity emitted for T1 - [x] Wait for next cron tick / trigger cron - [x] Pick a host known to be in T1 (call it `H_T1`); confirm its bit is NOT set in any `uptime` row written *after* the disable by filtering the chart to that host - [x] Pick a host in T2 (`H_T2`); confirm its bit IS still set in the same rows (T2 is not disabled) - [x] Pick a no-team host (`H_none`); confirm its bit IS still set (no-team hosts follow the global value) #### 2.2 Same fleet, different dataset - [x] With T1's uptime disabled, confirm T1's hosts ARE still written into `cve` rows on subsequent ticks (per-dataset isolation) #### 2.3 All teams disabled, global on, no-team hosts - [x] Disable uptime on every team (T1, T2, T3) - [x] Confirm next tick still writes a row containing only no-team hosts' bits (global is on, no-team hosts always count) - [x] Re-enable uptime on all teams --- ### 3. Global Scrub — DELETE #### 3.1 Successful global scrub - [x] Note baseline: `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';` (should be > 5000 to exercise the loop; if not, manually insert filler rows or run multiple cron ticks) - [x] Disable uptime globally via the API - [x] Wait for the worker to pick up the scrub / trigger the job - [x] Confirm the count drops to 0: `SELECT COUNT(*) FROM host_scd_data WHERE dataset='uptime';` - [x] Confirm rows for **other datasets** are untouched - [ ] Test again but disable via GitOps --- ### 4. Per-Fleet Scrub — ANDNOT #### 4.1 Single-fleet scrub clears bits - [x] Identify hosts in T1 and record their IDs (call this set `S`) - [x] Pre-disable, confirm at least one `host_scd_data` row for `dataset='uptime'` has bits set at positions in `S` by filtering the chart to those hosts - [x] Disable uptime on T1 only, via the API - [x] Wait for the scrub to run / trigger it - [x] Confirm: every existing row for `dataset='uptime'` now has NO bits set at any position in `S`. Spot-check by filtering the chart to those hosts - [x] Confirm rows for `dataset='cve'` (different dataset) are untouched - [x] Confirm bits for hosts in T2/T3 (not disabled) are still set - [x] Run test again but disable via GitOps #### 4.2 Multi-fleet scrub via GitOps batch - [x] Apply a GitOps spec that flips cve to false on T1 and T3 in a single apply - [x] Wait for scrub(s) to complete - [x] Confirm bits for the union of T1∪T2 hosts are cleared from every row of `dataset='cve'` - [x] Confirm T2 hosts' bits remain set --- ### 5. Activity Feed Cross-Check - [x] Each global flip emits exactly one `disabled_historical_dataset` activity (existing behavior, unchanged) - [x] Each per-team flip emits one scoped activity with the team's ID and name - [x] PATCH submitting unchanged values emits **no** activity and causes **no** scrub (no `host_scd_data` data change observed after the cron tick) - [x] No new "scrub completed" or "scrub started" activity is emitted (out of scope for v1) - [x] Re-enable flips emit `enabled_historical_dataset` activities and do NOT emit any scrub-related activity --- ### 6. Regression Spot Checks - [x] With everything enabled (default), the chart UI renders the same data as before this change (no behavior change in the "all on" case) - [x] AppConfig YAML round-trip (`fleetctl apply`) is benign: applying the unchanged config produces no scrub jobs and no activities - [x] GitOps apply with `historical_data` omitted from team specs defaults to `true` (per the gitops-api change) and does not trigger spurious scrubs - [x] After a full disable+scrub of cve, the `host_scd_data` table has no `dataset='cve'` rows; the chart UI for "vulnerable hosts over time" shows an empty/zero state without errors --- <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Chart collection now supports per-dataset scoping and honors team-level disables; new scrub jobs are registered and worker handlers added. * New dataset scrub operations: global and fleet-scoped scrubs; scrubs can be enqueued and are deduplicated to avoid duplicate pending jobs. Historical-data changes enqueue scrubs after save (errors logged, non-blocking). * **Tests** * Added unit tests for scope resolution, scrub enqueue/dedup behavior, scrub workers, scrub application, and low-level blob scrub logic. * **Documentation** * Added OpenSpec metadata for the chart scrub change. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
917ce99324 |
Allow disabling chart datasets: frontend (#44797)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44077 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually ### Manual QA — Chart disabling (frontend) Setup notes: - Test as a global admin and (where indicated) a fleet admin - Need a Premium deployment to test per-fleet behavior - Have at least one fleet (e.g. "Engineering") for fleet-scoped checks - Be able to flip `appConfig.features.historical_data.{uptime,vulnerabilities}` and the equivalent per-fleet keys (via API or settings UI) #### Advanced settings page — `/settings/organization/advanced` - [ ] "Activity & data retention" subheading appears on the page - [ ] Section contains a "Disable hosts online" checkbox - [ ] Section contains a "Disable vulnerabilities" checkbox - [ ] Section is visible on Free tier - [ ] Section is visible on Premium tier - [ ] With `historical_data.uptime = true` in appConfig, "Disable hosts online" loads **unchecked** - [ ] With `historical_data.uptime = false` in appConfig, "Disable hosts online" loads **checked** - [ ] With `historical_data.vulnerabilities = false` in appConfig, "Disable vulnerabilities" loads **checked** - [ ] Saving with "Disable hosts online" checked sends `features.historical_data.uptime = false` in the `PATCH /api/v1/fleet/config` payload (verify in network tab) - [ ] Saving with "Disable vulnerabilities" unchecked sends `features.historical_data.vulnerabilities = true` #### Per-fleet settings page (Premium only) - [ ] Fleet settings page shows an "Activity & data retention" section - [ ] Section contains "Disable hosts online" and "Disable vulnerabilities" checkboxes scoped to the active fleet - [ ] On **Free** tier, the per-fleet TeamSettings page is not reachable (and therefore the section is not visible) - [ ] On **Premium** tier, the section is visible to fleet admins - [ ] Saving with "Disable vulnerabilities" checked sends `features.historical_data.vulnerabilities = false` in `PATCH /api/v1/fleet/teams/{id}` (verify in network tab) #### Per-fleet checkbox locks when global is disabled - [ ] With global `historical_data.uptime = false`, fleet's "Disable hosts online" is **disabled / non-interactive** - [ ] Hovering the locked fleet checkbox shows the tooltip "Disabled globally" - [ ] **Stored value preserved across lockout**: - [ ] Set fleet `historical_data.uptime = false` (saved) - [ ] Then set global to `false` — fleet checkbox renders **checked but disabled** - [ ] Flip global back to `true` — fleet checkbox renders **checked and interactive** (fleet's `false` survived the round trip) #### Confirmation modal on Save (both global and per-fleet) - [ ] Loading with `uptime = true`, then checking "Disable hosts online" and clicking Save → modal opens listing **"Hosts online"** - [ ] No PATCH is issued until the user confirms - [ ] No-op save (page loaded with `uptime = false`, no changes, click Save) → modal does **not** open - [ ] Re-enable (page loaded with `uptime = false`, uncheck "Disable hosts online", Save) → modal does **not** open; PATCH is issued - [ ] Disabling **both** datasets at once → modal lists **both** "Hosts online" and "Vulnerabilities" - [ ] Modal "Save and disable" button issues the PATCH and modal closes once it resolves successfully - [ ] Modal Cancel closes the modal, no PATCH is issued, checkboxes remain checked (form state unchanged, ready to retry or revert) #### Confirmation modal copy by scope - [ ] Modal opened from Advanced page → body describes the change as affecting the **entire Fleet deployment** - [ ] Modal opened from fleet "Engineering" settings → body **references "Engineering" by name** and describes the change as affecting **only that fleet** #### GitOps mode lockout - [ ] With GitOps mode enabled, on `/settings/organization/advanced`: - [ ] "Disable hosts online" is disabled with the GitOps tooltip - [ ] "Disable vulnerabilities" is disabled with the GitOps tooltip - [ ] With GitOps mode enabled, on a fleet's settings page: - [ ] "Disable hosts online" is disabled with the GitOps tooltip - [ ] "Disable vulnerabilities" is disabled with the GitOps tooltip - [ ] When **both** GitOps mode and "Disabled globally" would apply, the fleet checkbox shows the **GitOps tooltip** (GitOps wins) #### Dashboard chart card — empty state - [ ] With `appConfig.features.historical_data.uptime = false`, viewing dashboard in "All teams" with the **uptime** dataset selected → visualization area shows the **"Data collection disabled"** empty state - [ ] Dataset selector, time range selector, and filter gear remain visible **and functional** while the empty state is shown - [ ] With global `vulnerabilities = true` but the active fleet's `historical_data.vulnerabilities = false`, viewing the dashboard scoped to that fleet with a vulnerabilities-driven dataset selected → empty state shown - [ ] With both global and fleet sub-keys `true` (or fleet sub-key absent in All-teams scope) → visualization renders normally; empty state does **not** appear - [ ] Switching from a disabled dataset (A) to an enabled dataset (B) replaces the empty state with B's visualization - [ ] A dataset whose internal name has no `DATASET_CONFIG_KEY` mapping renders normally (no empty state — implicitly enabled) #### Empty-state content - [ ] When the empty state is rendered for the "Hosts online" dataset, "Hosts online" appears as the dataset name - [ ] Link text describes re-enabling collection (e.g. "Manage data collection in Advanced settings") - [ ] Empty state includes a link to `/settings/organization/advanced` when viewing All Fleets as a global admin - [ ] Empty state includes a link to the correct Fleet settings page when viewing a specific fleet as a global or team admin - [ ] Empty state includes no link or call to action when viewed as a non-admin #### Activity feed rendering - [ ] `enabled_historical_dataset` with `dataset = "uptime"`, `fleet_id = null` → renders **"Enabled data collection for Hosts online."** ("Hosts online" bolded) - [ ] `disabled_historical_dataset` with `dataset = "vulnerabilities"`, `fleet_id = null` → renders **"Disabled data collection for Vulnerabilities."** - [ ] `enabled_historical_dataset` with `dataset = "uptime"`, `fleet_id = 7`, `fleet_name = "Engineering"` → renders **"Enabled data collection for Hosts online for the Engineering fleet."** - [ ] `disabled_historical_dataset` with `dataset = "vulnerabilities"`, `fleet_id = 7`, `fleet_name = "Engineering"` → renders **"Disabled data collection for Vulnerabilities for the Engineering fleet."** - [ ] `disabled_historical_dataset` with an **unknown** `dataset = "policy_compliance"` → does **not** throw; renders **"Disabled data collection for Policy compliance."** (raw key sentence-cased — `_` → space, first letter capitalized) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Org- and team-level controls to enable/disable historical data collection for "Hosts online" and "Vulnerabilities." * Updated chart label: "Hosts active" → "Hosts online." * Confirmation modal and a disabled-state UI when data collection is turned off; per-team confirmation flow to prevent accidental data loss. * Activity feed now records enable/disable actions for historical dataset collection with readable dataset labels and scope. * **Tests** * Added comprehensive tests for controls, confirmation flows, disabled states, chart headings, and activity items. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
e72c38ad60 |
Allow GitOps user to list software (#44721)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44696 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] reproduced issue on main branch (with software exceptions on and policies with software automation, `fleetctl gitops` failed for a gitops user with a 403) - [X] verified issue fixed on this branch -- `fleetctl gitops` synced successfully For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Expanded GitOps permissions to read and list software inventory, software titles, installable software, and maintained apps at both global and team scopes; adjusted related read behaviors and capitalization in messaging. * **Tests** * Updated and added authorization and integration tests to reflect the new GitOps read/list behavior across software-related and maintained-app endpoints, including team-scoped scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c3b82539a5 |
Allow disabling historical data collection (GitOps / API) (#44488)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44077 # Details * Adds `historical_data` key to app and team config (and gitops) with `uptime` and `vulnerabilities` subkeys. Keys default to `true`, meaning "collect this data" * Adds `enabled_historical_dataset` and `disabled_historical_dataset` activities when these values are flipped via GitOps or the config APIs The majority of the file changes in here are GitOps test files that need to be updated to have the new config in them. **This PR does _not_ implement using these configs to actually disable data collection or purge data; that will come in a follow-up PR (as well as the front-end)** # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually #### Defaults - [X] Fresh install: `GET /api/v1/fleet/config` returns `features.historical_data.uptime: true` and `features.historical_data.vulnerabilities: true` - [X] Created a new fleet via `POST /api/v1/fleet/teams`, then `GET /api/v1/fleet/fleets/{id}` returns `features.historical_data.uptime: true` and `features.historical_data.vulnerabilities: true` #### Global PATCH (`POST /api/v1/fleet/config`) - [X] PATCHed `{"features": {"historical_data": {"vulnerabilities": false}}}` — `vulnerabilities` flipped to `false`, `uptime` unchanged at `true` - [X] PATCHed `{"features": {"historical_data": {"uptime": false, "vulnerabilities": true}}}` — both values applied as sent - [X] PATCHed `{"features": {"historical_data": {"vulnerabilites": false}}}` (typo in sub-key) — request rejected with 4xx, stored config unchanged #### Fleet PATCH (`PATCH /api/v1/fleet/fleets/{id}`) - [X] PATCHed a fleet with `{"features": {"historical_data": {"uptime": false}}}` — fleet's `uptime` flipped to `false`, `vulnerabilities` unchanged - [X] Subsequent `GET /api/v1/fleet/fleets/{id}` returns the toggled values under `features.historical_data` (storage shape is symmetric with global) - [X] PATCHed a fleet with `{"features": {"enable_host_users": false}}` (a non-`historical_data` features sub-field) — request returned 200 but the fleet's `enable_host_users` is unchanged (silently ignored, per existing endpoint convention) #### GitOps — global (`fleetctl gitops -f global.yml`) - [X] Applied a YAML with `features.historical_data: {uptime: true, vulnerabilities: false}` — `vulnerabilities` is `false` after apply, `uptime` is `true` - [X] Applied a YAML whose `org_settings` omits `features` entirely — both sub-keys are `true` after apply (defaults injected even if previously disabled) - [X] Applied a YAML where `historical_data` only contains `uptime: false` — `uptime: false` is honored, `vulnerabilities` defaults to `true` - [X] Disabled `vulnerabilities` via the API, then ran `fleetctl gitops` with a YAML that doesn't pin it — `vulnerabilities` flips back to `true` (this is intentional; gitops is the source of truth) #### GitOps — fleet - [X] Applied a fleet YAML with `features.historical_data: {uptime: false}` — that fleet has `uptime: false`, `vulnerabilities: true` after apply - [X] Applied a fleet YAML whose `team_settings.features` omits `historical_data` — both sub-keys are `true` after apply - [X] Applied a fleet YAML that omits `features` entirely — both sub-keys are `true` after apply #### `fleetctl apply` (legacy, partial-merge) - [ ] Disabled `vulnerabilities` via the API, then ran `fleetctl apply` with a YAML that doesn't mention `historical_data` — `vulnerabilities` is still `false` (apply leaves omitted fields alone) #### Activities — global - [X] After PATCHing global to disable `vulnerabilities`, the latest activity is `disabled_historical_dataset` with payload `{"dataset": "vulnerabilities", "fleet_id": null, "fleet_name": null}` - [X] After PATCHing global with both sub-keys flipping in one request, two activities are emitted (one per sub-key) - [X] After PATCHing global with the same values that are already stored, zero new activities are emitted - [X] After re-enabling a previously disabled dataset, the activity type is `enabled_historical_dataset` #### Activities — per fleet - [X] After PATCHing fleet `workstations` to disable `uptime`, the activity is `disabled_historical_dataset` with payload `{"dataset": "uptime", "fleet_id": <workstations id>, "fleet_name": "workstations"}` - [X] Toggling the same dataset on two different fleets produces two distinct activities, one per fleet - [X] After a fleet PATCH with the same values already stored, zero new activities are emitted For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [ ] Alerted the release DRI if additional load testing is needed ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [X] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) - https://github.com/fleetdm/fleet/pull/44703 - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - [X] Verified that any relevant UI is disabled when GitOps mode is enabled <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Historical-data controls: per-org and per-team toggles for uptime and vulnerability time‑series, with defaults applied when keys are omitted and enable/disable activities emitted on changes. * **Bug Fixes** * Partial updates and PATCH/GitOps flows preserve unspecified historical-data sub-keys instead of clearing them. * **Tests** * Expanded unit and integration tests covering defaults, partial PATCH/GitOps behavior, idempotency, and activity emission. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
5e7f5a7584 |
Optimize data collection: add index and batch deletes (#44692)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44609 # Details This PR optimizes the historical data collection system in two ways: 1. Adds an additional index on the `host_scd_data` table allowing more efficient lookups of rows by their `valid_to`, to optimize both closing out open rows and deleting old rows 2. Implements batching in the job that deletes old rows, so that it no longer blocks writes if the collection job happens to happen at the same time as the cleanup job # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [ ] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [ ] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually SQL explains -- before: ``` +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ | 1 | DELETE | host_scd_data | NULL | ALL | NULL | NULL | NULL | NULL | 144320 | 100.00 | Using where | +----+-------------+---------------+------------+------+---------------+------+---------+------+--------+----------+-------------+ +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ | 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range | uniq_entity_bucket | 604 | const,const | 3030 | 100.00 | Using where | +----+-------------+---------------+------------+-------+--------------------------------------+--------------------+---------+-------------+------+----------+-------------+ ``` Using a test set of data (~144k "open" rows), UPDATES happened at 9 ops per second. after: ``` +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ | 1 | DELETE | host_scd_data | NULL | range | idx_valid_to_dataset | idx_valid_to_dataset | 5 | const | 55749 | 100.00 | Using where | +----+-------------+---------------+------------+-------+----------------------+----------------------+---------+-------+-------+----------+-------------+ +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ | 1 | UPDATE | host_scd_data | NULL | range | uniq_entity_bucket,idx_dataset_range,idx_valid_to_dataset | idx_valid_to_dataset | 609 | const,const,const | 4 | 100.00 | Using where; Using temporary | +----+-------------+---------------+------------+-------+-----------------------------------------------------------+----------------------+---------+-------------------+------+----------+------------------------------+ ``` Using the same test set of data, UPDATES happened at 4,910 ops per second. For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results this should significantly improve results! - [ ] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [ ] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [ ] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Cleanup now runs in controlled, ordered batches, removing only closed/historical records while respecting cancellation; error reporting for cleanup was strengthened. * Added a new composite index on historical data to improve cleanup and query performance. * **Tests** * Added tests and test helpers validating batched cleanup behavior, preservation of open records, multi-batch operation, and cancellation handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
56a8bc03a8 |
Fix lint issue in checkerboard viz (#44510)
Fixes a lint issue in (currently unused) code for the 24-hr checkerboard, that was causing one of our build tests to fail. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Enhanced 24-hour checkerboard visualization tooltips to display complete data values and totals alongside percentage information. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
5ac999187d | Vulnerability exposure chart front-end (#44261) | ||
|
Scott Gress
|
07b6679ead | Build fleetctl MSI packages (#43403) | ||
|
Scott Gress
|
4334017b38 |
Add Vulnerabilities exposure dataset (#44124)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #43769 # Details Adds methods to collect data for the `cve` dataset. As with all sets this is collected at hourly granularity, but unlike the `uptime` set, the `cve` set uses the "snapshot" strategy so that we record at most one change (the most recent) per hour. For this first iteration, we are _recording_ data for all CVEs (i.e., which hosts were exposed to which CVEs at a given time), but we are only _reporting_ a subset of CVEs for the dashboard chart. See [this comment](https://github.com/fleetdm/fleet/pull/44124#discussion_r3155554405) for more info. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] Spot-checked the CVEs chosen by the `trackedCVESoftwareMatchers` and didn't find any outside of the expected - [X] With [front-end PR](https://github.com/fleetdm/fleet/pull/44261), generated chart: <img width="706" height="421" alt="image" src="https://github.com/user-attachments/assets/539d9877-6573-4406-a159-1d2a711a045f" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Host vulnerability (CVE) chart added to the dashboard; CVE chart data collection is now active. * Critical CVE tracking surfaces high-severity vulnerabilities. * **Improvements** * CVE chart refreshes every 3 hours (was daily) for more timely insights. * Snapshot collection reconciles and closes prior data during empty runs to keep charts accurate. * CVE queries may produce zero datapoints when no tracked CVEs exist, without affecting other metrics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
dd2e21d788 |
Fix some bugs in charts UI (#44329)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44249 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [ ] Added/updated automated tests not worth it for style fixes and accidental content removal - [ ] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] can click on hosts enrolled labels and bars to go to the appropriate hosts list - [X] no more weird focus rectangles on hosts enrolled chart - [X] hosts enrolled chart y-axis respects dark mode - [X] metrics cards returned to dashboard - [X] hovering over a bar on the hosts enrolled chart shows a tooltip with the # of hosts in that platform For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Hosts Enrolled card: click platform bars or Y-axis labels to navigate to platform-specific host lists; interactive tooltips show platform and host count. * **UI Improvements** * Streamlined dashboard host-count display with a small loading spinner while summaries load. * Improved interactivity cues: hover styles, pointer cursor, and refined tooltip visuals for chart elements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
9609f0ad4b |
Upgrade fork-ts-checker-webpack-plugin (#44304)
Resolves #44301 This PR updates the `fork-ts-checker-webpack-plugin` to resolve the errors we were seeing when running it. Since it now runs cleanly, it identified a few typescript errors that need to be resolved at the same time: * `<EmptyState>` no longer takes a `graphicName` param since it's been redesigned in #43896 * `<PoliciesCard>` was reworked in #43411 and no longer takes `router`. I verified that this compiles cleanly in `make generate` and `make generate-dev`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Refactor** * Simplified component properties and prop handling across host details, packs, and queries pages. * Refined empty-state configuration in packs management and queries tables. * **Chores** * Updated development build tool dependency from version 6.5.0 to 9.1.0 for TypeScript type-checking integration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
9ae4373f89 |
Don't ignore GitOps secrets on free tier (#44148)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44118 # Details On free tier, ignore exceptions and always apply enroll secrets when present. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually @AndreyKizimenko QA'd manually For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed GitOps to correctly apply enrollment secrets and labels on free tier licenses, even when exception flags are configured. * **Tests** * Added tests validating that GitOps properly applies secrets and labels for free tier customers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6c28db8206 |
Add activities when toggling GitOps exception settings (#44094)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #40171 # Details Adds audit activity when enabling or disabling GitOps exceptions. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually <img width="714" height="699" alt="image" src="https://github.com/user-attachments/assets/161bd084-347b-4cde-893e-9b385f13872c" /> For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Activity feed now records when GitOps exceptions (labels, software, secrets) are enabled or disabled. * **UI** * Activity messages show which specific exception was enabled or disabled. * **Tests** * Added unit and integration tests verifying generation and rendering of enable/disable exception activities, including single and multiple flips and no-op updates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sgress454 <553428+sgress454@users.noreply.github.com> Co-authored-by: Luke Heath <luke@fleetdm.com> |
||
|
Scott Gress
|
9b01710a80 |
Don't throw gitops-exceptions-related errors on Free tier (#44118)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #44098 # Details We set the "secrets" exception on for all new instances (and the label exception for existing instances), but you can't turn them off in the free tier. That means GitOps runs (including the one we use to initialize new instances) would fail with the "you can't use this key because the exception is on" error. This PR fixes the issue by not enforcing that rule for free tier instances. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [X] added test verifying that the free tier can run gitops using excepted keys w/out error, and verified that it fails on main and passes on this branch - [X] QA'd all new/changed functionality manually - [X] spun up a new free-tier server successfully For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * GitOps exception enforcement no longer blocks free-tier users; enforcement is applied only for premium licenses, allowing GitOps applies on free tiers. * **Tests** * Added an integration test validating free-tier GitOps behavior to prevent regressions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
207c3058c5 |
Add dashboard charts frontend (#43878)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #43769 # Details This PR adds "Hosts active" and "Hosts enrolled" charts to the dashboard. New components: * **ChartCard.tsx**: encapsulates a visualization-agnostic chart, for data provided by the new `/charts` endpoint created in https://github.com/fleetdm/fleet/pull/43910 * **ChartFilterModal.tsx**: modal for setting filters on a chart. Currently supports filtering by label, platform and individual host. * **CheckerboardViz.tsx**: a checkerboard visualization for use in ChartCard. Capable of charting 1, 7, 14 or 30 days at a time, although only 30 day charts are used right now. Bespoke rendering using SVG, since recharts scatterchart was harder to wrangle than it was worth. * **LineChartViz.tsx**: a line-chart visualization using Recharts * **HostsEnrolledCard.tsx**: a bar chart of enrolled hosts using Recharts # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - With backend provided by https://github.com/fleetdm/fleet/pull/43910: <img width="1426" height="428" alt="image" src="https://github.com/user-attachments/assets/0f53b9d1-c87b-4225-a175-2d40af5afe41" /> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Dashboard now shows interactive "Hosts active" (line/heatmap) and "Hosts enrolled" (bar) charts with metric selection, filter modal (labels/platforms/hosts), legends, tooltips, and responsive layout. * **Tests** * Added comprehensive tests covering chart rendering, checkerboard heatmap, and no-data states. * **Chores** * Added charting library dependency to support visualizations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
28908e6083 |
Dashboard charts backend (#43910)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #42812 # Details This PR implements a new bounded context, `chart`, with a single endpoint `/charts`. The context encompasses a framework for recording and querying and aggregating historical data for Fleet hosts, and returning that data via the API for the purpose of charting. This initial iteration has a full implementation of a dataset called "uptime" which captures which hosts were online hour-by-hour (online meaning, having been "seen" at some point during that hour). It has a partial implementation of a "cve" dataset which will capture which hosts were vulnerable to which CVEs during a given day. ### Data storage Data is stored in an SCD (slowly-changing dimension) format in the `host_scd_data` table, where the main "value" in a row is stored in the `host_bitmap` column, which is a `mediumblob` where each bit encodes a host ID (bit one represents host ID 1, bit 1444 represents host ID 1444, etc.). The set of bits set on a row represents that hosts for which that dataset is "on" during a given time period represented by the `valid_from` (inclusive) and `valid_to` (exclusive) dates, where a `valid_to` can have the special "sentinel" value 9999-12-31T00:00:00.000 meaning that the row is still "open" (the value represents everything from `valid_from` to the present). Additionally an `entity_id` column can be used for datasets with multiple dimensions, e.g. CVE exposure or software usage which would have entity IDs representing CVEs or software items respectively. ### Data collection Data is collected via a cron job that runs every 10 minutes. Each dataset has its own `Collect` method which will sample the data for the given moment. For example the "uptime" dataset gathers the set of hosts that are online at the moment, and the "cve" dataset will gather the set of hosts that are vulnerable to each CVE at that moment. The sample can then be recorded using one of two strategies: * `accumulate`: bitwise OR the sample with any data already recorded for the current hour, or add a new pre-closed row for that hour. * `snapshot`: if there is no open row, create one with the sample and `valid_to set` to the sentinel. Otherwise: * If the sample has the same value as the current open row, do nothing * If the sample has a different value and the current open row's `valid_from` is within the same hour, update the current row's value * If the sample has a different value and the current open row's `valid_from` is not within the same hour, close the current open row and start a new one with `valid_from` = the start of the current hour ### Data retrieval 1. Gets the set of host IDs to retrieve data for. This starts with the set of host IDs in the requested fleet (or all the hosts a user has access to if no `fleet_id` param was passed to the `/charts` endpoint), and further whittled down by any filter options supplied with the request (labels, platforms, etc.). 2. Finds all `host_scd_data` rows for the requested dataset and date range (i.e. all rows whose `valid_from` is < the date range end and `valid_to` is > the date range start). 3. Calculates the date ranges of the "buckets" to return datapoints for. For the uptime chart we default to 3-hour buckets, so we want 8 buckets per day. 4. Iterates over each bucket and finds the row or rows from host_scd_data that cover that bucket range. For datasets using the "accumulate" strategy, the values for those rows are ORed together. For "snapshot"s, we take the one active at the bucket end time to represent the bucket (e.g. "which hosts had a given CVE at the end of the day") ### Tools This PR includes two dev tools that don't require deep review: * **chart-backfill** - used to backfill data to various datasets for testing * **charts-collect** - used to collect data from a live server via the API and put into a local hosts_scd_data table # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - With [front-end branch](https://github.com/fleetdm/fleet/pull/43878) <img width="712" height="434" alt="image" src="https://github.com/user-attachments/assets/b2ccce49-b5fd-4076-b47f-0eea6a53260c" /> ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added charting bounded context: HTTP API for metrics (uptime, CVE), dataset registry, hosted dataset collection, background collection/cleanup with opt-out env. * New utilities: host bitmap operations and string-list/uint-list parsers. * New CLI tools to collect and backfill chart data. * **Database** * Migration and schema to store host time-series SCD chart data. * **Tests** * Extensive unit and integration tests for service, storage, caching, cron, and utilities. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
fe66e3fd08 |
Don't refetch "change management" settings page on focus (#44054)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43977 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [ ] Added/updated automated tests - [X] QA'd all new/changed functionality manually - Repro'd issue on main - Verified on this branch that dirtying the form and then focusing elsewhere / refocusing on page doesn't lose changes. For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Bug Fixes * Fixed an issue where the Change Management form would reset upon losing and regaining page focus. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
75d8102922 | Update class on exceptions field to fix padding (#43927) | ||
|
Scott Gress
|
91d9b25924 |
Allow conditional downloads across fleets (#43679)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43417 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. done in https://github.com/fleetdm/fleet/pull/42216 ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - Using a local fileserver, added the same software to two fleets and ran `fleetctl gitops`. Verified that the first fleet downloaded the file, the second fleet used the cache, and both fleet showed the software installer in the UI. ## Summary by CodeRabbit * **Chores** * Updated software installer lookup mechanism to support optional team-scoped searches, enabling fallback to cross-team installer cache when team-specific installers are unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
df44a9342e |
Use Docker as default WiX runtime on macOS arm64 (#43715)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43484 # Details Apple Silicon Macs were being forced down the Wine+local-wix-dir path because the fleetdm/wix:latest image was deemed unreliable on arm64 in Jan 2024. Docker Desktop's amd64 emulation has matured since -- the image builds both amd64 and arm64 MSIs on arm64 macOS successfully. This PR: - Drops the arm64-forces-Wine guard in BuildMSI so the Docker path is the default on every macOS arch when --local-wix-dir isn't provided. - Drops the macOS "Install wine and wix" + "Build MSI on macOS (using local Wix)" CI steps. The ubuntu-latest matrix entry already exercises the Docker path, and the install-wine.sh flow is brittle against Gcenx release churn and homebrew-cask deprecation. - Updates the install-wine.sh script to fail and output a message indicating that Docker should be used, or else Wine installed manually. ``` ============================================================ This script no longer installs Wine. ============================================================ Wine is no longer required to build Windows (.msi) packages on macOS. fleetctl package now uses Docker by default on all macOS architectures. RECOMMENDED: install Docker Desktop https://docs.docker.com/get-docker If you cannot use Docker and still need to build MSIs with Wine on macOS see the upstream WineHQ wiki for installation instructions: https://gitlab.winehq.org/wine/wine/-/wikis/MacOS Automatic Wine installation via Homebrew is no longer attempted here because the wine-stable cask is deprecated and upstream Wine releases have caused repeated breakage. ``` - Retains the wix auto-download helper (downloadAndExtractZip, extractZipFile, wixDownload) for backwards-compatibility when Docker isn't detected, with a deprecation warning. The Wine + --local-wix-dir path remains available for macOS users who opt into it, but is no longer documented. See #43484. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - Dropped MacOS packaging tests. The Ubuntu test already exercises the Docker path that MacOS now uses. - [X] QA'd all new/changed functionality manually - Built and installed both amd64 and arm64 .msi packages successfully <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * MSI packages on macOS now build using Docker by default, removing the Wine dependency. * **Documentation** * Updated macOS setup guidance: Docker Desktop is now required for MSI packaging instead of Wine. * **Chores** * Simplified Wine-related helper scripts and removed outdated installation logic. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c3f50293e9 |
Pin new Wine version in install-wine.sh script (#43482)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43476 # Details Bumps the cask commit we're pinned to, as the upstream has removed the previous version (10) in favor of the latest (11). This is gonna probably happen every year. We could consider hosting our own mirror but not sure it's worth it for something that happens once a year. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a ## Testing - [X] QA'd all new/changed functionality manually Ran the job successfully on this branch: https://github.com/fleetdm/fleet/actions/runs/24358586742/job/71132009934 But it's a ticking clock; brew is gonna stop letting us install apps that don't pass gatekeeper in September: <img width="1025" height="245" alt="image" src="https://github.com/user-attachments/assets/5a35b31e-649c-46a0-bdad-3abef41a3e0c" /> Will open separate issue for this. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated Wine installation tooling to reference the latest Homebrew configuration definition, improving installation reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
b2e6162e51 |
Fix issue with GitOps incorrectly wiping policy stats (#43282)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #43273 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - Added new test for this case (policies without software automation being pushed by two different users), verified it fails on main and passes on this branch - [X] QA'd all new/changed functionality manually - [X] Verified that changing `webhooks_and_tickets_enabled` on a policy AND running gitops as another user doesn't wipe stats - [X] Verified that changing `query` on a policy and running gitops does wipe stats - [X] Verified that changing `query` on a policy and running gitops does wipe stats <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed an issue where policy stats were incorrectly reset during GitOps policy updates. Policy statistics now remain accurate when policies are re-applied without modification to installation or script configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
3ae98ee01d |
Clean up Gitops tests and add deprecation tests (#43039)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #40015 * Moves repeated empty mocks into a new `setupEmptyGitOpsMocks` method * Adds new "deprecation" tests: * In TestGitOpsFullGlobal, TestGitOpsFullTeam and TestGitOpsFullGlobalAndTeam tests "kitchen sink" with both new and deprecated keys * Added keys and checks to verify `setup_experience`, `apple_business_manager` and `volume_purchasing_program` configs * Consolidated map of deprecated -> new GitOps keys in one place |
||
|
Scott Gress
|
2790ba984e |
Update VSCode Typescript version setting (#43131)
This PR updates VSCode settings to use Fleet's installed version of Typescript (v4.7.4) for its language server (linting, autocomplete, etc.) instead of what's built in to VSCode (v6.0.2). As the two can drift, we end up with VSCode incorrectly highlighting certain syntax as incorrect. |
||
|
Scott Gress
|
1eabb85a5a |
Activate deprecation warnings (#41449)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40015 # Details Activates deprecation warnings for old API params and CLI args, updates tests that would generate warnings (except for tests explicitly designed to generate warnings). The expectation from here on is that Fleet UI usage should not generate any deprecation warnings in the server logs, nor should the output from `generate-gitops` generate any warnings when fed into `gitops`. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually - [X] clicked around in an mdm-enabled instance, turned setup experience features on and off, saw no server warnings - [X] did `fleetctl generate-gitops` on mdm-enabled instance, saw no server or cli warnings - [X] did `fleetctl gitops` on mdm-enabled instance, saw no server or cli warnings |
||
|
Scott Gress
|
cfb3667d4d |
Skip startup library for primo mode (#43019)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40791 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [ ] Added/updated automated tests We don't typically do new primo mode tests. As long as existing tests for startup library pass, we're good. - [X] QA'd all new/changed functionality manually - [X] tested that starting a new instance with `FLEET_PARTNERSHIPS_ENABLE_PRIMO=true` causes the starter library not to be applied (no fleets, labels or policies |
||
|
Scott Gress
|
854fa2af62 |
Cleanup docker publish (#42693)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42691 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a ## Testing - [ ] Added/updated automated tests - [X] QA'd all new/changed functionality manually - I ran the updated snapshot action on this branch and verified that it pushed the branch-tagged image, but not the SHA-tagged one. - I ran the cleanup script in dry-run mode and verified that it didn't expect to delete any non-sha-tagged images - I wasn't able to test the delete-image-on-branch-delete action for obvious reasons. - I haven't tested the cleanup script in non-dry-run mode... I could do on my personal dockerhub... <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Automated cleanup of Docker images when development branches are deleted to maintain registry hygiene. * New utility for managing and cleaning up legacy Docker image tags. * **Chores** * Enhanced Docker image tagging in snapshot builds with improved branch name handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
c4aa6f5529 |
Use fleetctl new templates for new instances (#42768)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41409 # Details This PR updates the `ApplyStarterLibrary` method and functionality to rely on the same templates and mechanisms as `fleetctl new`. The end result is that running `fleetctl new` and `fleetctl gitops` on a new instance should be a no-op; no changes should be made. Similarly, changing the templates in a Fleet release will automatically affect `fleetctl new` and `ApplyStarterLibrary` in the same exact way for that release. > Note that this moves the template files out of `fleetctl` and into their own shared package. This move comprises the majority of the file changes in the PR. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests Note that <img width="668" height="44" alt="image" src="https://github.com/user-attachments/assets/066cd566-f91d-4661-84fc-2aabbfce2ef9" /> will fail until the 4.83 Fleet docker image is published, since it's trying to push 4.83 config (including `exceptions`) to a 4.82 server. - [X] QA'd all new/changed functionality manually - [X] Created a new instance and validated that the fleets, policies and labels created matched the ones created by `fleetctl new` - [X] Ran `fleetctl new` and verified that it created the expected folders and files - [X] Ran `fleetctl gitops` with the files created by `fleetctl new` and verified that the instance was unchanged. - [X] Ran `fleetctl preview` successfully using a dev build of the Fleet server image (since it won't work against the latest published build, which doesn't support `exceptions`). Verified it shows the expected teams, policies and labels |
||
|
Scott Gress
|
29aa39a392 |
Update default fleet selected on dashboard and controls (#42688)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40317 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [ ] Added/updated automated tests With the current router we have in place, we can't really test `<Link>` elements, so our ability to make useful automated tests is pretty limited here. I extracted the fleet name sorting code into an exported function and added some tests for that. - [X] QA'd all new/changed functionality manually - [X] verified that when All Fleets is selected in dropdown, navigating to Controls switches to Workstations - [X] verified that when another fleet is selected in dropdown, navigating to Controls maintains that selection - [X] verified that when a fleet is selected in dropdown, navigating to the dashboard changes to All Fleets - [X] verified that when "Unassigned" is present in the fleets dropdown, it is at the bottom - [X] verified that when using a permalink to the dashboard with a fleet selected (e.g. `?fleet_id=1`), the correct fleet shows as selected |
||
|
Scott Gress
|
e62bdf17b6 |
Remove UI gating in GitOps mode for excepted entities (#42486)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42184 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [ ] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [ ] If paths of existing endpoints are modified without backwards compatibility, checked the frontend/CLI for any necessary changes ## Testing - [ ] Added/updated automated tests - [ ] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [ ] QA'd all new/changed functionality manually For unreleased bug fixes in a release candidate, one of: - [ ] Confirmed that the fix is not expected to adversely impact load test results - [ ] Alerted the release DRI if additional load testing is needed ## Database migrations - [ ] Checked schema for all modified table for columns that will auto-update timestamps during migration. - [ ] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. - [ ] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [ ] Verified that the setting is exported via `fleetctl generate-gitops` - [ ] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) - [ ] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - [ ] Verified that any relevant UI is disabled when GitOps mode is enabled ## fleetd/orbit/Fleet Desktop - [ ] Verified compatibility with the latest released version of Fleet (see [Must rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md)) - [ ] If the change applies to only one platform, confirmed that `runtime.GOOS` is used as needed to isolate changes - [ ] Verified that fleetd runs on macOS, Linux and Windows - [ ] Verified auto-update works from the released version of component to the new version (see [tools/tuf/test](../tools/tuf/test/README.md)) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added support for GitOps exceptions per entity type (labels, software, secrets), allowing specific areas to bypass GitOps mode restrictions when configured. * **Bug Fixes** * Improved GitOps mode behavior to properly respect per-entity-type exception settings across software, labels, and secrets management. * **Tests** * Extended test coverage for GitOps exception handling scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
07a8378a68 |
Implement FMA software policy automation (#42533)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #36751 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] Verified that `fleetctl generate-gitops` correctly outputs policies with `install_software.fleet_maintained_app_slug` populated when the policies have FMA automation - [X] Verified that running `fleetctl gitops` using files with `install_software.fleet_maintained_app_slug` creates/updates FMA policy automation correctly - [X] Verified no changes to the above for custom packages or VPP apps - [X] Verified that when software is excepted from GitOps, FMA policy automations still work (correctly validates FMAs exist before applying) ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [ ] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) checking on this - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - [X] Verified that any relevant UI is disabled when GitOps mode is enabled |
||
|
|
e54ea7b3ad |
Add GitOps exceptions UI to Change Management settings (#42348)
**Related issue:** Resolves #42182 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See <a href="https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files">Changes files</a> for more information. will add to last PR ## Testing - [X] Added/updated automated tests - [X] Added `ChangeManagement.tests.tsx` with unit/integration tests covering: - Exceptions checkboxes render correctly from config for new install (only Enroll secrets checked) and migrated instances (Labels and Enroll secrets checked) - Form save sends the correct `gitops.exceptions` payload via `configAPI.update` - Form validation shows error when GitOps mode is enabled but no repo URL is provided - Non-premium tier renders the premium feature message - [X] QA'd all new/changed functionality manually - [X] verified that Labels and Secrets are checked for pre-existing (migrated) instance - [X] verified that only Secrets is checked for new instance - [X] verified that changing the settings in the UI and saving persists the `gitops.exceptions` config as expected <img src="https://github.com/user-attachments/assets/095c538c-68aa-4179-b4b1-fd5878c0a2b0"> ## Summary by CodeRabbit * **New Features** * Added GitOps exceptions configuration in Change Management settings with toggles for Labels, Software, and Enroll Secrets, enabling granular control over exception flags. <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/fleetdm/fleet/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: sgress454 <553428+sgress454@users.noreply.github.com> |
||
|
Scott Gress
|
f1bad72003 |
Use new multiplatform keys on the front end (#41763)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41601 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Details This PR updates the front end to use the following renamed API keys: * bootstrap_package -> macos_bootstrap_package * manual_agent_install -> macos_manual_agent_install * enable_release_device_manually -> apple_ enable_release_device_manually * script -> macos_script * macos_setup -> setup_experience * macos_settings -> apple_settings * custom_settings -> configuration_profiles * macos_setup_assistant -> apple_setup_assistant It also ensures that consumers of the "get fleet config" API pull from the `.fleet` property rather than `.team`, so that they can use all of the newly renamed response fields. ## Summary by CodeRabbit * **Refactor** * Restructured Mobile Device Management configuration for Apple devices, reorganizing setup experience, bootstrap package, and device configuration field organization. * Updated filter terminology and query parameters throughout device management interfaces, improving how users filter and navigate Apple device settings. * Enhanced configuration field naming conventions for better clarity and maintainability across device management features. <!-- end of auto-generated comment: release notes by coderabbit.ai --> # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually --- Manual Test Plan Prerequisites - Fleet server with MDM enabled (macOS at minimum) - At least one macOS host enrolled in Fleet MDM - A team configured with setup experience settings --- 1. Manage Hosts — Filters - [X] macOS settings filter: - Go to Hosts > Manage Hosts - Filter by macOS settings status (e.g. Pending, Verified, Failed) - Verify the filter applies and hosts list updates - Check that the URL contains apple_settings=<status> - Copy the URL, paste it in a new tab — verify the filter is still applied - Manually edit the URL to use macos_settings=<status> instead — verify it still works (backward compat) - Clear the filter pill — verify both apple_settings and macos_settings are removed from the URL - [X] Bootstrap package filter: - Filter by bootstrap package status - Verify the URL contains macos_bootstrap_package=<status> - Manually edit the URL to use bootstrap_package=<status> — verify it still works - Clear the filter pill — verify both params are removed --- 2. Setup Experience (Controls Page) - [X] Bootstrap package: - Go to Controls > Setup experience for a team - Upload a bootstrap package — verify it appears in the table - Toggle the "manual agent install" advanced option on/off — verify it saves - Delete the bootstrap package — verify it's removed - In the bootstrap package table, click "View all hosts" link for a status row — verify it navigates to Manage Hosts with macos_bootstrap_package in the URL - [X] End user authentication: - Toggle end user authentication on/off for a team and for "No team" - Verify the toggle reflects the saved state after page reload - [X] Setup assistant: - Upload a setup assistant profile - Verify the "release device manually" toggle works for both a team and "No team" - [X] Install software: - Verify the "require all software" checkbox reflects the correct saved state for both team and "No team" |
||
|
Scott GressandAllen Houchins
|
7db99c7801 |
Fix test-packaging.yml action (#42570)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42573 Fixes failing test by replacing no-longer-supported `--no-quarantine` option with manually turning off quarantine for Wine. Successful run here: https://github.com/fleetdm/fleet/actions/runs/23661332211 --------- Co-authored-by: Allen Houchins <allenhouchins@mac.com> |
||
|
Scott Gress
|
6598b608b7 |
Enforce GitOps exceptions (#42191)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42180 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced GitOps exception handling for labels, secrets, and software with clearer enforcement and omission semantics. * Server-side prefetch of team software so omitted team software can preserve existing installers during validation. * Presence flags track whether top-level keys (labels, secrets, software) were provided versus omitted. * **Behavior Changes** * Omitted vs empty sections are now distinguished: omission can mean “no-op” or “delete-all” depending on exception settings. * GitOps YAML can define and manage labels directly; validations now reject YAML that includes keys marked as excepted. <!-- end of auto-generated comment: release notes by coderabbit.ai --> # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually * **Labels** - [ ] Validated that with label exceptions off, omitting `labels:` key from default.yml clears all global labels - [ ] Validated that with label exceptions off, omitting `labels:` key from a fleet .yml clears all labels for that fleet - [ ] Validated that with label exceptions off, setting empty `labels:` key from default.yml clears all global labels - [ ] Validated that with label exceptions off, setting empty `labels:` key from a fleet .yml clears all labels for that fleet - [ ] Validated that with label exceptions on, omitting `labels:` key from default .yml leaves existing global labels as-is - [ ] Validated that with label exceptions on, omitting `labels:` key from a fleet .yml leaves existing labels as-is - [ ] Validated that with label exceptions on, setting `labels:` key on default .yml generates an error - [ ] Validated that with label exceptions on, setting `labels:` key on a fleet .yml generates an error - [ ] Validated that with label exceptions on, a policy using `labels_include_any` referencing an existing label succeeds without `labels:` key - [ ] Validated that with label exceptions on, a query using `labels_include_any` referencing an existing label succeeds without `labels:` key - [ ] Validated that with label exceptions on, an MDM profile using `labels_include_any` referencing an existing label succeeds without `labels:` key - [ ] Validated that with label exceptions on, a software package using `labels_include_any` referencing an existing label succeeds without `labels:` key (requires software exceptions off) - [ ] Validated that with label exceptions on, an app store app using `labels_include_any` referencing an existing label succeeds without `labels:` key (requires software exceptions off) - [ ] Validated that with label exceptions on, a fleet maintained app using `labels_include_any` referencing an existing label succeeds without `labels:` key (requires software exceptions off) * **Secrets** - [ ] Validated that with secrets exceptions off, omitting `secrets:` key from default.yml clears all global secrets - [ ] Validated that with secrets exceptions off, omitting `secrets:` key from a fleet .yml clears all secrets for that fleet - [ ] Validated that with secrets exceptions on, omitting `secrets:` key from default .yml leaves existing global secrets as-is - [ ] Validated that with secrets exceptions on, omitting `secrets:` key from a fleet .yml leaves existing secrets as-is - [ ] Validated that with secrets exceptions on, setting `secrets:` key on default .yml generates an error - [ ] Validated that with secrets exceptions on, setting `secrets:` key on a fleet .yml generates an error * **Software** - [ ] Validated that with software exceptions off, omitting `software:` key from no-team.yml/unassigned.yml clears all software for "no team" - [ ] Validated that with software exceptions off, omitting `software:` key from a fleet .yml clears all software for that fleet - [ ] Validated that with software exceptions off, setting empty `software:` key on a fleet .yml clears all software for that fleet - [ ] Validated that with software exceptions off, setting empty `software:` key on no-team.yml/unassigned.yml clears all software for "no team - [ ] Validated that with software exceptions on, omitting `software:` key from a fleet .yml leaves existing software as-is - [ ] Validated that with software exceptions on, setting `software:` key on a fleet .yml generates an error - [ ] Validated that with software exceptions on, omitting `software:` key from no-team.yml/unassigned.yml leaves existing software as-is for "no team" - [ ] Validated that with software exceptions on, setting `software:` key on no-team.yml/unassigned.yml generates an error - [ ] Validated that with software exceptions on, a policy using `install_software.hash_sha256` referencing an existing package succeeds without `software:` key - [ ] Validated that with software exceptions on, a policy using `install_software.app_store_id` referencing an existing VPP app succeeds without `software:` key - [ ] Validated that with software exceptions on, a patch policy using `fleet_maintained_app_slug` referencing an existing FMA succeeds without `software:` key - [ ] Validated that with software exceptions on, `setup_experience.software` referencing existing software succeeds without `software:` key (server-side validation fallback) - [ ] Validated that with software exceptions on, omitting `software:` from no-team.yml/unassigned.yml preserves existing no-team software - [ ] Validated that with software exceptions on, a policy in no-team.yml/unassigned.yml using `install_software.hash_sha256` referencing existing no-team software succeeds without `software:` key For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results I don't think so. There is a bit of overhead when this feature is used since we have to fetch software from the server, but it would be done in a specific test, so even if there is an impact it should affect existing load testing, only new, specific tests. |
||
|
Scott Gress
|
83d1b02d0a |
Remove generate-gitops warning about categories (#42514)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #30453 All the work for this was already done in https://github.com/fleetdm/fleet/pull/32492. I verified that `generate-gitops` outputs the expected categories, and that `gitops` reads them back in correctly. This PR just removes the warning |
||
|
Scott Gress
|
34e0ace5f6 |
fix policy software vpp automations (#42400)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42399 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] Added a policy with a VPP software automation, confirmed that `generate-gitops` output an `app_store_id` for the policy - [X] Verified that FMA and custom package automations still output correctly from `generate-gitops` - [X] Verified that `fleetctl gitops` ingested the policies + automations correctly. |
||
|
Scott GressandNoah Talerman
|
249cb76be8 |
Update fleetctl new readme (#42236)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41345 Updates the README.md generated from `fleetctl new` to include instructions on how to deploy to GitHub / Gitlab. --------- Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> |
||
|
Scott GressandNoah Talerman
|
994843f330 |
Small tweaks to default.yml template for fleetctl new (#42239)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #41345 * Adds a missing comment delimiter * Adds required `organization_name` to `apple_business_manager` section * Adds required `location` to `volume_purchasing_program ` section * Adds Windows MDM section with `windows_enabled_and_configured`, required for the Windows OS update settings to work --------- Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com> |
||
|
Scott Gress
|
8ea6f338de |
Pin Localstack version (#42253)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42252 Pins the Localstack image to the last-known-good version (4.5) before they 🔪 'd the community edition and started requiring an auth token. I also added a "wait for localstack" as an initial debugging step, and left it in to catch similar future issues. It's probably redundant since there likely _is_ no future for Fleet and Localstack beyond this, but it take milliseconds and would catch any other weird Localstack failures so, why not. |
||
|
Scott Gress
|
deec6aa904 |
Add "exceptions" GitOps config (#42013)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #42008 # Details Step one in https://github.com/fleetdm/fleet/issues/40171. This PR adds a new `exceptions` subsection to the current GitOps config, with boolean keys for software, secrets and labels. For existing instances a migration is included to set labels and secrets to `true`. For new instances, only `secrets` will be `true`. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, will put changelog in when more functionality is implemented. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually (https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] ran migration and verified that app config had `gitops.exceptions` with `software: false, secrets: true, labels: true` - [X] created a new instance and verified that that app config had `gitops.exceptions` with `software: false, secrets: true, labels: false` - [X] verified that the PATCH /config API works and can update exceptions independently of other config ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. n/a - [X] Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects. n/a - [X] Ensured the correct collation is explicitly set for character columns (`COLLATE utf8mb4_unicode_ci`). n/a ## New Fleet configuration settings - [X] Setting(s) is/are explicitly excluded from GitOps these will not be set in GitOps, since they're _about_ how GitOps works. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * GitOps configuration now supports exception settings for granular resource control. Administrators can configure which specific resource types (labels, software, and secrets) are included in or excluded from GitOps mode operations. * **Improvements** * Improved GitOps configuration handling to preserve exception settings during partial updates and system migrations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
91362ba2ca |
Add fleetctl new command (#41909)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41345 # Details This PR: * Adds a new `fleetctl new` command which creates a starter GitOps repo file structure * Adds support for file globs for the `configuration_profiles:` key in GitOps, to support its use in the `fleetctl new` templates. This involved moving the `BaseItem` type and `SupportsFileInclude` interface into the `fleet` package so that the `MDMProfileSpec` type could implement the interface and do glob expansion. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] added unit and intg tests for globbing profiles - [ ] added tests for `fleetctl new` - [X] QA'd all new/changed functionality manually - [X] `fleetctl new` with no args prompted for org name and created a new `it-and-security` folder under current folder w/ correct files - [X] `fleetctl new --dir /tmp/testnew` created correct files under `/tmp/testnew` - [X] `fleetctl new --dir /tmp/testexisting --force` with an existing `/tmp/testexisting` folder created correct files under `/tmp/testexisting` - [X] `fleetctl new --org-name=foo` created correct files under `it-and-security` without prompting for org name - [X] `paths:` in `configuration_profiles` picks up multiple matching profiles - [X] `paths:` + `path:` in `configuration_profiles` will error if the same profile is picked up twice <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added `fleetctl new` command to initialize GitOps repository structure via CLI. * Added glob pattern support for `configuration_profiles` field, enabling flexible profile selection. * **Chores** * Updated CLI dependencies to support enhanced user interactions. * Removed legacy website generator configuration files. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott GressandCopilot Autofix powered by AI
|
40e91c0ece |
Allow hosts key to be empty for manual labels (#42022)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41672 # Details Updates GitOps label functionality so that omitting the `hosts:` key under a manual label will _not_ clear hosts from that label, but will instead preserve the existing membership. This allows users to manage manual hosts with an external system (via the labels API), while still managing the labels themselves in GitOps. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] verified that you can still add a manual label with `hosts:` - [X] verified that leaving `hosts:` off a manual label doesn't change the host assignment - [X] verified that putting `hosts:` with no value on a manual label clears the hosts - [X] verified that you can still add a dynamic label - [X] verified that generate-gitops still exports manual hosts --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Scott Gress
|
f093406e04 |
Fix get/create/update query response (#41966)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41603 # Details Fixes an issue where the nested `query` key in the get, create and update query API responses, which is the literal SQL query, was getting duplicated into a `report` key with the SQL. This was happening because our JSON field duplicator which adds the renamed version of deprecated keys to responses is intentionally naive; it doesn't account for cases where the parent and child structs have the same key with different meanings because that is... not ideal. In Fleet 5 we won't have this problem since it'll just be `report.query`, but for now the solution is to just hard-code a `Report` field onto the response structs, rather than over-complicate the duplicator code to account for different nesting levels. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] verified no `query.report` or `report.report` present in get query response - [X] verified no `query.report` or `report.report` present in create query response - [X] verified no `query.report` or `report.report` present in modify query response |
||
|
Scott GressandIan Littman
|
30632040b1 |
Fix addFleetMaintainedAppEndpoint to accept fleet_id param (#41805)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41771 # Details Solves two issues in 4.82: 1. The `fleet_id` param in `POST /software/fleet_maintained_apps` wasn't being read, causing all FMAs using that param to be added to fleet ID 0 (unassigned aka No Team) 2. We were logging deprecation warnings for body params even if the topic was turned off, meaning Fleet would generate deprecation warnings in certain cases that users wouldn't be able to fix. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests Added unit tests for the decoder since it's got one-off logic in it - [X] QA'd all new/changed functionality manually - [X] Added an FMA to a fleet successfully using `fleet_id` - [X] Added an FMA to a fleet successfully using `team_id` and saw deprecation warning - [X] Added an FMA to "Unassigned" successfully using `fleet_id=0` - [X] Added an FMA to "Unassigned" successfully using `team_id=0` - [X] Added an FMA to "Unassigned" successfully with no `fleet_id` or `team_id` param (this seems like a bug but it's existing behavior) --------- Co-authored-by: Ian Littman <iansltx@gmail.com> |
||
|
Scott Gress
|
01d13f5080 |
add keymap for new renames, and shallow duplication (#41682)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #41091 # Details This PR finishes the work of aliasing multi-platform keys by: * Added the renames to the list maintained by generate-gitops so that `fleetctl get` can use the new names * Updated the code that adds the new names to API and `fleetctl get` output to only add new nested keys under new parents, e.g. add `apple_settings.configuration_profiles`, but not `macos_settings.configuration_profiles`. The API key duplicator now runs through `RewriteDeprecatedKeys` which is a little heavier per-token, but for old keys we're doing less work so I think this ends up being slightly more performant than before, at least for large payloads. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, changelog for new keys added in previous PR ## Testing - [X] Added/updated automated tests updated tests for the duplicators - [X] QA'd all new/changed functionality manually - [X] `/config` and `/fleets` APIs now only return new keys under new parents - [X] `fleetctl get fleets` now returns new multiplatform keys |
||
|
Scott Gress
|
759c95100a |
Add aliases for more multi-platform setup experience fields (#41599)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41091 # Details Implements the following config key aliases: - [x] Add a second name for `bootstrap_package`: `macos_bootstrap_package` - Support `bootstrap_package` for backwards compatibility - [x] Add a second name for `manual_agent_install`: `macos_manual_agent_install` - Support `manual_agent_install` for backwards compatibility - [x] Add a second name for `enable_release_device_manually `: `apple_ enable_release_device_manually ` - Support `enable_release_device_manually` for backwards compatibility - [x] Add a second name for `script`: `macos_script` - Support `script` for backwards compatibility Also cleans up some error messages missed in previous alias PRs. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually ran gitops successfully with new keys |
||
|
Scott Gress
|
3e0552849b |
Delete dead code file team_scheduled_queries.ts (#41639)
Deletes a code file that's not referenced by anything and keeps causing
me merge conflicts.
JS linter and tests pass without it, which tells you everything you need
to know 🔪
|
||
|
Scott Gress
|
4e35de2ac3 |
Update fleetctl client urls and params (#41463)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41385 # Details This PR updates `fleetctl` to use the new API urls and params when communicating with Fleet server. This avoids deprecation warnings showing up on the server that users won't be able to fix. Most of the changes are straightforward `team_id` -> `fleet_id`. A couple of code changes have been pointed out. The most interesting is in icon URLs, which can be persisted in the database (so we'll need to do a migration in Fleet 5 if we want to drop support for `team_id`. Similarly the FMA download urls are briefly persisted in the db for the purpose of sending MDM commands. If we drop team_id support in Fleet 5 there could be a brief window where there are unprocessed commands in the db still with `team_id` in them, so we'll probably want to migrate those as well. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a - all internal ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [X] ran `fleetctl gitops` on main and saw a bunch of deprecation warnings, ran it on this branch and the warnings were gone 💨 - [X] same with `fleetctl generate-gitops` - [X] ran `fleetctl get` commands and verified that the new URLs and params were used - [X] ran `fleetctl apply` commands and verified that the new URLs and params were used |
||
|
Scott Gress
|
2d4e72ac7a |
Alias JIT Saml "team" attribute to FLEET_JIT_USER_ROLE_FLEET_<FLEET ID> (#41402)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40642 # Details This PR adds `FLEET_JIT_USER_ROLE_FLEET_` as an expected Saml attribute alongside `FLEET_JIT_USER_ROLE_TEAM_`. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually Tested with SimpleSAML SSO. - [x] Updated `users.php` to use both the new attribute and the old attribute for a user, and was able to log in with that user and see them created using JIT with the correct permissions |
||
|
Scott Gress
|
5c70d09081 |
Re-order post-4.82 migrations to fix CI test failure (#41624)
The [Test DB changes CI test](https://github.com/fleetdm/fleet/actions/workflows/test-db-changes.yml?query=branch%3Amain) has been failing on main since yesterday with: ``` ❌ fail: server/datastore/mysql/migrations/tables/20260218165545_FixMismatchedSoftwareTitles.go has an older timestamp than server/datastore/mysql/migrations/tables/20260218175704_FMAActiveInstallers.go this might cause problems if this change is merged please update the timestamp of server/datastore/mysql/migrations/tables/20260218165545_FixMismatchedSoftwareTitles.go Error: Process completed with exit code 1. ``` I think because some migrations were merged to main after the fleet 4.82 tag, but they came from a pre-4.82 tag branch. This PR moves the migrations to the earliest possible time _after_ all of the 4.82 migrations. |
||
|
Scott Gress
|
2686907dba |
Update API calls in front-end to use new, non-deprecated URLs and params (#41515)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41391 # Details This PR updates front-end API calls to use new URLs and API params, so that the front end doesn't cause deprecation warnings to appear on the server. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, should not be user-visible ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually The biggest risk here is not that we missed a spot that still causes a deprecation warning, but that we might inadvertently make a change that breaks the front end, for instance by sending `fleet_id` to a function that drops it silently and thus sends no ID to the server. Fortunately we use TypeScript in virtually every place affected by these changes, so the code would not compile if there were mismatches between the API expectation and what we're sending. Still, spot checking as many places as possible both for deprecation-warning leaks and loss of functionality is important. ## Summary by CodeRabbit * **Refactor** * Updated API nomenclature across the application to use "fleets" instead of "teams" and "reports" instead of "queries" in endpoint paths and request/response payloads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
18a60fc59a |
fix flaky test (#40916)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40915 Attempt to fix flaky MDM test. Authored by 🤖 . Stress test here: https://github.com/fleetdm/fleet/actions/runs/22647401633/job/65638800588 |
||
|
Scott Gress
|
b7a792d1cb |
Add fleet_id / fleet_name columns to CSV hosts export (#41446)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41074 # Details Fixes an issue where CSV export still has `team_name` and `team_id` columns, but not `fleet_name` or `fleet_id`. Unlike the API param and other renames, I took a manual approach here since it's just the two fields and isn't likely to expand. I added cleaning them up to my Fleet 5 punchlist. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [X] QA'd all new/changed functionality manually - [X] exported report from UI, saw both team_name and fleet_name - [X] exported report via API with no columns requested (so all columns returned), saw team_id, team_name, fleet_id and fleet_name |
||
|
Scott GressandIan Littman
|
2bf46b14ad |
Detect unknown keys in top-level GitOps settings (#41303)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41280 # Details Phase 2 of the "detect unknown keys in GitOps" work. The `org_settings` and `settings` top-level keys mainly shadow the `fleet.AppConfig` and `fleet.TeamConfig` types, but they have a couple of extra GitOps-only fields, so we add new GitOps-specific types for them (similar to what we already have for `GitOpsControls` and `GitOpsSoftware`. The `org_settings:` case is further complicated by the fact that its extra fields are themselves `any` types which we need to parse, so we add those to the `anyFieldTypes` registry in the validator to tell it what types to check them against. Also had to add some new logic to handle the GoogleCalendarAPI case which doesn't expose its keys as `json` tags at all, since we use a special method to obfuscate the values. I've tested this by routing the output from `fleetctl generate_gitops` back through `fleetctl gitops`, which is how I caught the `end_user_license_agreement` issue. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a - already added in previous PR ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually Did the `fleetctl generate-gitops` -> `fleetctl gitops` loop as mentioned above. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added support for managing secrets and certificate authorities through GitOps configuration * Improved detection of configuration errors with clear error messages when using unknown or misspelled settings keys, including suggestions for common typos * Enhanced error reporting for nested configuration files with precise location information <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Ian Littman <iansltx@gmail.com> |
||
|
Scott Gress
|
056e567bab |
Implement webhooks_and_tickets_enabled flag for policies in GitOps (#41183)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40627 # Details This PR updates the way we enable failed policy reporting (via webhook or ticket integration) for individual policies in GitOps. The existing method is to declare a `policy_ids` key underneath `failing_policies_webhook:` in either the global or a fleet .yml file, and specify a list of policy IDs to enable the automation for. This PR maintains this feature for backwards compatibility, and adds a new feature where you can set `webhook_and_tickets_enabled: true` key in the policy declaration itself. If _both_ these methods are used, the GitOps run will fail. **Implementation note:** Because we're keeping the old way of doing this until Fleet 5, I took the easy route and just translated the new way into the old way; that is, we gather up the list of policies with `webhook_and_tickets_enabled: true`, get their IDs and send that list to the server under the same config we did previously. This works fine and there's nothing _wrong_ with it but ideally this flag would work the same as other per-policy flags like `calendar_events_enabled` that are stored on the policy record. That requires a migration and more new code that we'd have to maintain alongside the existing code (or translate the old strategy to the new one). I'm taking the lower-touch path here. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [x] Verified that `generate-gitops` outputs the new `webhooks_and_tickets_enabled` flag instead of outputting `policy_ids` under `failing_policies_webhook` - [X] Verified that using the new flag in a fleet .yml file results in the specified policies being enabled in the "other" automations for policies (whether the webhook automation is enabled or not) - [X] Verified the same for a global .default.yml file - [X] Verified that using the old `failing_policies_webhook.policy_ids` a fleet .yml file results in the specified policies being enabled in the "other" automations for policies (whether the webhook automation is enabled or not) - [X] Verified the same for a global .default.yml file - [X] Verified that trying to use both `webhooks_and_tickets_enabled` and `failing_policies_webhook.policy_ids` at the same time results in an error. ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [x] Verified that the setting is exported via `fleetctl generate-gitops` - [ ] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) see https://github.com/fleetdm/fleet/issues/40627#issuecomment-4024988552 - [x] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - [x] Verified that any relevant UI is disabled when GitOps mode is enabled <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added configuration flag to enable webhooks and tickets for policies in GitOps settings. * System automatically resolves and assigns policy IDs when using the new flag. * **Tests** * Added comprehensive test coverage for webhook and ticket enablement in GitOps workflows, including conflict detection and policy ID assignment validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
63be71fd72 |
require controls on either global or no-team (#41350)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41307 # Details * Fixes a potential issue where running `fleetctl gitops` with only the global file, with no controls provided, could wipe out global controls that are provided in the "no team" file. * Fixes error message when controls are missing. # Checklist for submitter ## Testing - [X] Added/updated automated tests - [X] QA'd all new/changed functionality manually - [x] `fleetctl gitops -f /path/to/default.yml` without controls, gives `error: 'controls' must be set on global config, no-team.yml or unassigned.yml` - [x] `fleetctl gitops -f /path/to/default.yml` with empty controls works - [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/no-team.yml` without controls, gives `error: 'controls' must be set on global config or no-team.yml` - [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/unassigned.yml` without controls, gives `error: 'controls' must be set on global config or unassigned.yml` - [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/no-team.yml` with empty controls in no-team.yml works - [x] `fleetctl gitops -f /path/to/default.yml -f /path/to/unassigned.yml` with empty controls in unassigned.yml works - [x] `fleetctl gitops -f /path/to/no-team.yml` gives error `global config must be provided alongside no-team.yml` - [x] `fleetctl gitops -f /path/to/no-team.yml` gives error `global config must be provided alongside unassigned.yml` - [x] `fleetctl gitops -f /path/to/some-real-team.yml` with no controls works For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results |
||
|
Scott Gress
|
94d337c90a |
Fix flaky test: TestTriggerPollPicksUpQueuedRecord (#41155)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41154 🤖 fix to flaky test I encountered. Tagging @getvictor because he wrote the original test. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved test synchronization logic in the job polling test to more reliably detect and verify job completion. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
9715f75f9a |
Add glob support to more labels, policies and reports (#41141)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41006 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests Added tests for using path, paths and inline declaration for reports, policies and labels. - [X] QA'd all new/changed functionality manually - [x] tested that `path:` works for policies - [x] tested that `paths:` works for policies - [x] tested that incline declaration works for policies - [x] tested that `path:` works for reports - [x] tested that `paths:` works for reports - [x] tested that incline declaration works for reports - [x] tested that `path:` works for labels - [x] tested that `paths:` works for labels - [x] tested that incline declaration works for labels <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for glob patterns in path specifications within reports, labels, and policies configuration sections. * Enhanced validation and error handling for external file references. * Improved logging and error messages during configuration parsing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
9c4d5ce97e |
Make most GitOps top-level optional (#41138)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #41012 # Details This PR makes it allowable to leave out almost all top-level keys from GitOps files. The only required keys are _either_ `name:` (for a fleet settings file) or `org_settings:` (for a global settings file). Omitting a key is identical to supplying it with no value. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing - [X] Added/updated automated tests Updated the "missing all global keys test", and added some new tests to verify that omitting the key was the same as supplying it with an empty value. - [X] QA'd all new/changed functionality manually 1. Ran `fleetctl generate-gitops` to get a clean set of GitOps yml files 2. Removed all removable keys from default.yml and ran `fleetctl gitops` 3. Ran `fleetctl generate-gitops` again into a different dir 4. Ran `fleetctl gitops` with the original files to get back to original state 5. Cleared out all now-removable keys and replaced them with empty value (e.g. `reports:` with nothing under it) 6. Ran `fleetctl generate-gitops` again into a third dir 7. Compared the files from the second and third generate-gitops runs to verify that omitting the key had the same result as supplying it with an empty value 8. Did the above steps with a fleet (i.e. non-global) .yml file. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * GitOps files now support omitting top-level configuration keys instead of requiring them to be explicitly set to empty values. * org_settings is now required when team name is not specified. * **Tests** * Added integration tests validating behavior when omitting top-level keys in global and team-level GitOps configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Scott Gress
|
a8fa681467 |
Reorder columns on Host page (#41180)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #40489 # Detail This PR re-orders the column on the Manage Hosts page. No columns are added or removed. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. ## Testing Did my best to spot check with my human eyes that no code was changed, only moved. Also had 🤖 check it. - [X] QA'd all new/changed functionality manually <img width="1715" height="522" alt="image" src="https://github.com/user-attachments/assets/0df3081c-55dd-49cf-bc90-9a41114a36a3" /> <img width="1699" height="520" alt="image" src="https://github.com/user-attachments/assets/7bc050a7-18ae-4aa6-a74b-a459b5955be4" /> <img width="952" height="521" alt="image" src="https://github.com/user-attachments/assets/bf0d6d83-9e27-4ba7-af5d-887acf155e22" /> |
