<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40496
# Details
This is the first phase of an effort to detect unknown keys in GitOps
.yml files. In the regular `fleetctl gitops` case, it will fail when
unknown keys are detected. This behavior can be changed with a new
`--allow-unknown-keys` flag which will log the issues and continue.
In this first phase we are detecting unknown keys in _most_ GitOps
sections, other than the top-level `org_settings:` and `settings:`
sections which have more complicated typing. I will tackle those
separately as they require a bit more thought. Also ultimately I'd like
us to be doing this validation in a more top-down fashion in one place,
rather than spreading it across the code by doing it in each individual
section, but this is a good first step.
As a bonus, I invited my pal Mr. Levenshtein to the party so that we can
make suggestions when unknown keys are detected, like:
```
* unknown key "queyr" in "./lib/some-report.yml"; did you mean "query"?
```
> Note: the goal is to return as many validation errors as possible to
the user, so they don't have to keep running `fleetctl gitops` to get
the next error. I did _not_ update any other errors to stop returning
early, in an effort to keep this as low-touch as possible.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Tested this against existing it-and-security folder and one with
updated keys from https://github.com/fleetdm/fleet/pull/40959; no
unknown keys detected
- [X] Added unknown keys at various levels, GitOps errored with helpful
messages
- [X] Same as above but with `--allow-unknown-keys`; GitOps outputted
helpful messages but continued.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* GitOps runs now fail when unknown or misspelled keys are present in
configuration files.
* New CLI flag --allow-unknown-keys lets unknown keys be treated as
warnings instead of errors.
* Unknown-key messages include suggested valid key names to help correct
mistakes.
* **Tests**
* Expanded test coverage to validate unknown-key detection and the
allow-as-warning option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
## Summary
- `checkPermFile` in `pkg/secure/secure.go` now self-heals incorrect
file permissions via `os.Chmod` instead of returning a fatal error
- Fixes orbit crash-looping indefinitely when
`/opt/orbit/updates-metadata.json` has mode 755 instead of the expected
600
## Problem
Orbit refuses to start when `updates-metadata.json` has wrong
permissions (e.g. 755 instead of 600), entering an infinite restart loop
(`systemd` restart counter observed at 3447+). The manual workaround is
`chmod 600 /opt/orbit/updates-metadata.json`, but the root cause — an
external process changing file permissions — is intermittent and hard to
track.
The `checkPermFile` function in `pkg/secure/secure.go` was designed as a
security check, but its behavior of fatally erroring on any permission
mismatch causes a denial-of-service on the legitimate user. For
comparison, `checkPermPath` (the directory equivalent) already tolerates
permissions that are less permissive than expected.
## Fix
When `checkPermFile` detects a permission mismatch, it now attempts
`os.Chmod` to correct the permissions before proceeding. It only returns
an error if the chmod itself fails (e.g. insufficient privileges). This
preserves the security intent — files end up with correct permissions —
while making orbit resilient to external permission drift.
## Test plan
- [ ] `go test ./pkg/secure/ -v -run TestOpenFile` — verifies
self-healing behavior
- [ ] `go test ./pkg/secure/ -v -run TestMkdirAll` — unchanged, verifies
directory checks still work
- [ ] Manual: create `/opt/orbit/updates-metadata.json` with mode 755,
start orbit, confirm it self-heals and starts normally
---------
Co-authored-by: Bash Bandicoot <bash-bandicoot@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40488
# Details
Implements the renames requested in #40488:
- [X] Add a second name for `macos_setup`: `setup_experience`
- [X] Add a second name for `macos_settings`: `apple_settings`
- [X] Add a second name for `custom_settings`: `configuration_profiles`
- [X] Add a second name for `macos_setup_assistant`:
`apple_setup_assistant`
Prior names are deprecated and log warnings. This uses the same
`renameto` tags as previous aliases, and adds code in relevant sections
in gitops.go to run the existing "rename new to old keys" function so
that we can unmarshall into the existing structs (that still have their
`json` tags set to the old key names until Fleet 5).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Ran current it-and-security GitOps files successfully locally
(removing mdm stuff that wouldn't work for me locally, but wasn't
relevant to the updated keys
- [X] Run same files successfully after changing the deprecated key
names to their new aliases
- [X] Verified that new keys show up in API responses:
<img width="506" height="243" alt="image"
src="https://github.com/user-attachments/assets/db1eb522-a702-4d17-b313-81ca203632b6"
/>
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
n/a
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduces new configuration key aliases: apple_settings (macOS),
configuration_profiles (profiles for macOS/Windows/Android),
setup_experience (macOS setup), and apple_setup_assistant (macOS setup
assistant).
* Old configuration keys remain supported for backward compatibility;
tooling and generated controls will accept either the new or legacy
names.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
Fixes#41009
## Summary
- Scope `ValidatePackageIdentifiers` to only run when `$PACKAGE_ID` or
`$UPGRADE_CODE` template variables are present in the uninstall script
- Move `dmg`/`zip` early return before validation
- Switch from ASCII allowlist to shell metacharacter denylist, allowing
legitimate non-ASCII product names (e.g., `®`, parens) while still
blocking injection characters
## Test plan
- [x] Added unit tests for conditional validation (non-ASCII IDs
with/without template vars, dmg/zip bypass, upgrade code scoping)
- [x] Existing input tests still pass
- [x] Winget ingester tests unaffected
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40911
# Details
Updates some GitOps error messages to make them 1) use "report" instead
of query where applicable and 2) be more helpful by including filename
and path and not being confusing.
These IMO don't need to be cherry-picked to 4.82 since users won't be
getting deprecation warnings yet so the new error might actually be
_more_ confusing in this case, but I encountered them while working on
the "validate unknown keys" ticket and they looked really bad, so fixing
before I forget.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [X] Change "query name is required for each query" to "name is
required for each report in {filename} at {path}"
- [X] Change "query SQL query is required for each query" to "query is
required for each report in {filename} at {path}"
- [X] Change "query name must be in ASCII: {name}" to "name must be in
ASCII: {name} in {filename} at {path}"
- [X] Change "duplicate query names: {names}" to "duplicate report
names: {names}
Tested all in both main file and in a file included via `path:`
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40900
# Details
This PR makes the `secrets:` key under the top-level `org_settings` (for
default.yml) or `settings:` (for fleet .yml files) optional. Omitting
the key causes any enroll secrets present on the server to be retained.
There is more to the parent story that will require more design, but I
am getting this one out early because:
1. Our updated it-and-security files will not have `secrets:` and
2. This is not a breaking change, since currently omitting this key
results in a fatal error, _not_ the removal of all secrets (that
requires specifying an empty `secrets:` key)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
- [x] Using `secrets:` with correct syntax in `defaults.yml` updated
global secrets
- [x] Using `secrets:` with no value in `defaults.yml` removed all
global secrets
- [x] Omitting `secrets:` in `defaults.yml` retained all global secrets
- [x] Using `secrets:` with correct syntax in a fleet .yml file updated
that fleet's secrets
- [x] Using `secrets:` with no value in in a fleet .yml file removed
that fleet's secrets
- [x] Omitting `secrets:` in in a fleet .yml file retained that fleet's
secrets
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40108
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed double encoding of secret environment variables when configured
through GitOps, ensuring secrets are stored with proper escaping.
* **Tests**
* Added test coverage for configuration profile escaping to verify
proper handling of secret variables and API keys during GitOps
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40302
# Details
This PR adds support for a `paths:` key for scripts declared under
`controls:` in a GitOps fleet file. If supplied, `paths:` must contain a
"glob" expression (as [supported by the doublestar
package](https://github.com/bmatcuk/doublestar?tab=readme-ov-file#patterns)).
The existing `path:` key still works but may not contain glob
expressions. When a `paths:` key is encountered, we expand it and add
all matching valid (as in, `.sh` or `.ps1`) files to the set of script
files to process.
Subsequent PRs will add this functionality to other entities that use
`path:` (such as reports and policies).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
Tried with various combinations of `*` and `**` in gitops runs, and
mixing of `path:` and `paths:`
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40540
go-kit/log is no longer a direct dependency; moved kitlog adapter
required for some 3rd party libraries into its own package
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Modernized logging across the codebase: switched from legacy logging
wrappers to Go's standard slog, updated adapters, tests, tools, and
server components.
* Threaded the new slog logger through test utilities and tooling;
adjusted a small number of logging-related function/constructor
signatures to accept the new logger type (minor compatibility updates).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40540
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes present in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Updated internal logging infrastructure to improve consistency and
maintainability across the application.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #39314
**Related issue:** For #39238
# Details
This PR allows the use of "fleet" and "report" as spec "kinds",
deprecating the use of "team" and "query".
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
n/a
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
Deprecation logs assume `--enable-log-topics=deprecated-field-names` is
used in the command
- [ ] `fleetctl apply -f /path/to/spec` should add/update a fleet when
used with a spec containing `kind: fleet`
- [ ] `fleetctl apply -f /path/to/spec` should add/update a report when
used with a spec containing `kind: report`
- [ ] `fleetctl apply -f /path/to/spec` should add/update a fleet when
used with a spec containing `kind: team`, and log a deprecation warning
- [ ] `fleetctl apply -f /path/to/spec` should add/update a report when
used with a spec containing `kind: query`, , and log a deprecation
warning
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40564
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added OpenTelemetry instrumentation to the internal HTTP client to
provide enhanced observability through distributed tracing and metrics
collection for HTTP operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #40433
# Details
This PR updates `fleetctl gitops` and `fleetctl generate_gitops` to use
`unassigned.yml` in place of `no-team.yml`. The two files are utilized
identically, except that `unassigned.yml` expects the `name:` to be
`Unassigned` rather than `No team`.
Internally, we still map some things to the string "no team" before
sending to the back end so that we don't have to update back-end code
and make more spaghetti to clean up when we 🔪 No Team as a concept in
Fleet 5.
We do pass the filename into the main `DoGitOps` method, but both I and
Claude did our best to determine that it's not used in any way that
would break with this change.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
From test plan:
- [X] With a pre-existing GitOps folder w/ `no-team.yml`, run `fleetctl
gitops -f /path/to/no-team.yml --dry-run
--enable-log-topics=deprecated-field-names` and verify that everything
works as expected and you get the deprecation warning.
- [X] Do the above without `--dry-run` and verify via the UI and/or
`fleetctl generate-gitops` that the Fleet config is as expected.
- [X] Change `no-team.yml` to `unassigned.yml`, try a gitops run with
`fleetctl gitops -f /path/to/unassigned.yml --dry-run
--enable-log-topics=deprecated-field-names` and verify that you get an
error because the `name:` is still `No team`
- [X] Change the `name:` to `Unassigned`, repeat the run above and
verify that the output is the same as with `no-team.yml`, and that no
deprecation warning is listed.
- [X] Do the same as the above without `--dry-run` and verify that the
Fleet config is as expected.
- [X] Run `fleetctl generate-gitops` and verify that `unassigned.yml` is
output rather than `no-team.yml`, and any related files are under the
`lib/unassigned` folder rather than `lib/no-team`, and any paths inside
`unassigned.yml` (e.g. for scripts) are pointed at `lib/unassigned`.
## New Fleet configuration settings
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
@noahtalerman will work on this
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes included in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Consolidated and standardized internal logging infrastructure across
the application by adopting a unified logging package throughout the
codebase, replacing previous external logging dependencies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40122
# Details
* Adds deprecation warnings to `fleetctl apply`
* Adds alias conflict errors (i.e. using both new and deprecated keys in
the same spec) to `fleetctl apply`
* Adds logic around all deprecated field warnings to check the topic
first
* Disables deprecation warnings by default for `fleet serve`, `fleetctl
gitops` and `fleetctl apply`
* Enables deprecation warnings for dogfood via env var
To turn on warnings:
* In `fleet serve`, use either
`--logging_enable_topics=deprecated-field-names` or the
`FLEET_LOGGING_ENABLE_TOPICS=deprecated-field-names` env var
* In `fleetctl gitops` / `fleetctl apply` use either
`--enable-log-topics=deprecated-field-names` or
`FLEET_ENABLE_LOG_TOPICS=deprecated-field-names`
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
tested in `fleetctl apply`, `fleet serve` and `fleet gitops` that
warnings are suppressed by default and added when the appropriate env
var or CLI option is used
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40124
# Details
Implements the proposal in
https://docs.google.com/document/d/16qe6oVLKK25nA9GEIPR9Gw_IJ342_wlJRdnWEMmWdas/edit?tab=t.0#heading=h.nlw4agv1xs3g
Allows doing e.g.
```go
logger.WarnContext(logCtx, "The `team_id` param is deprecated, use `fleet_id` instead", "log_topic", "deprecated-field-names")
```
or
```go
if logging.TopicEnabled("deprecated-api-params") {
logging.WithLevel(ctx, slog.LevelWarn)
logging.WithExtras(
ctx,
"deprecated_param",
queryTagValue,
"deprecation_warning",
fmt.Sprintf("'%s' is deprecated, use '%s'", queryTagValue, renameTo),
)
}
```
Topics can be disabled at the app level, and enabled/disabled at the
command-line level.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
No logs have this in prod yet, but I added some manually in a branch and
verified that I could enable/disable them via CLI options and env vars,
including enabling topics that were disabled on the server. Tested for
both server and `fleetctl gitops`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added per-topic logging control to enable or disable logging for
specific topics via configuration and CLI flags.
* Added context-aware logging methods (ErrorContext, WarnContext,
InfoContext, DebugContext) to support contextual logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** For #39344
# Details
This PR builds on the previous PR
(https://github.com/fleetdm/fleet/pull/39847) which added `renameto`
tags to certain API parameters to mark them as deprecated. How this is
used:
### In requests
* When decoding requests, log a warning if a `json` or `query` param is
used that has a `renameto` tag, e.g. if a `team_id` param is sent but
the related struct has `renameto:"fleet_id"` in it.
* If the `renamedto` version (e.g. `fleet_id`) is sent in the request,
rewrite it to the deprecated name so that it can be unmarshalled into
the struct
* If both versions are sent (e.g. `team_id` AND `fleet_id`), throw an
error and quit
* URLs with deprecated terms have new aliases using `WithAltPaths` --
warning on using old URLSs a TODO that will be handled in a subsequent
PR.
### In responses
* Output _both_ the deprecated and new names for fields that have
`renameto` tags, so that we don't break existing workflows expecting the
old keys. Uses a shared `DuplicateJSONKeys` to do the duplication.
* Most API responses are handled in `EncodeCommonResponse`. Exceptions
are activities, failing policy webhooks and the streaming "list hosts"
endpoints which call the function directly.
### In fleetctl
* Similar to requests, log warnings when deprecated keys are used and
rewrite the new keys internally so that they can be unmarshalled.
* For `fleetctl get` and `fleetctl generate-gitops`, _only_ output the
new names
* The set of keys to replace is hardcoded in `fleetctl` rather than
being dynamically generated as it is for API endpoints. Given the
mixture of typed and untyped data and the level of nesting, dynamic map
generation was very fragile and error-prone.
### Performance considerations
* The biggest performance hit is the addition of the JSON key rewriter
to the request pipeline. The rewriter buffers the entire request into
memory before eventually passing it to the decoder than unmarshals the
data into structs. I tried implementing this as a true streaming
rewriter but encountered issues where the request would hang if the
downstream reader (the decoder) encountered any errors. It's possible we
could implement this in a streaming fashion if we replace our [current
request
decoder](https://github.com/fleetdm/fleet/blob/da43bf8371695382c4af0972d5da456c6b94bdaf/server/service/endpoint_utils.go#L108)
with the v2 version, which is a bigger change requiring more thoughtful
discussion in the engineering team. As it stands, memory usage for
requests with deprecated fields will double while the request is being
decoded.
* The "alias rules" used to determine the old and new key names are
cached per struct type and for most endpoints are generated on server
start, so no performance impact is expected.
* Some `fleetctl` commands may have an extra unmarshal/marshal step but
as these are user-initiated and not performed in tight loops, the impact
should be minimal.
### TODO
* Log deprecation warnings when old URLs like "/fleet/teams" are used
* Update API fields that the front-end uses to avoid deprecation
warnings
* Update `fleetctl apply` to accept/return `kind: fleet` rather than
`kind: team`
* Find/update any fleet server config vars with old language
* Update all error messages that use old language
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
- [X] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [X] QA'd all new/changed functionality manually
* Clicking around the front-end, no broken pages due to request
ingestion errors or bad responses
* Looking in network tab to verify that responses have both the old and
new keys
* Running `fleetctl generate-gitops` and verifying that the output looks
correct and can be ingested by `fleetctl gitops`
* Running `fleetctl get` and `fleetctl apply`
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
This PR changes 3 things.
1. Validate `admin_url` + all URLs for HTTPS/non-private
2. Add custom `DialContext` hook in fleethttp.NewClient(), this is
needed for DNS-rebinding protection at connection time
3. Validate Smallstep SCEP challenge endpoint
# **IMPORTANT**
There are two validations occurring.
1. `CheckURLForSSRF`
2. `SSRFDialContext`
## Why?
`CheckURLForSSRF` checks the hostname. It resolves DNS, validates the
ip, and then returns an error to the user. It protects certificate
authority create/update API endpoints. But then
`GetSmallstepSCEPChallenge` calls `http.NewRequest(http.MethodPost,
ca.ChallengeURL, ...)` with the original hostname
This is where `SSRFDialContext` comes into play. It fires when an actual
HTTP request is attempted. Meaning Fleet would first build the request,
encode the body, set up TLS, etc., before being blocked at the dial.
`CheckURLForSSRF` stops the operation before any of that work happens.
`SSRFDialContext` protects the actual challenge fetch that happens later
at enrollment time. They're not always called together. The dial-time
check is the only thing protecting the enrollment request and DNS
rebinding.
## Should we remove `CheckURLForSSRF`
This is debatable and I don't have a strong opinion. Removing
`CheckURLForSSRF` would still provide the same protection. However, it
would return a generic connection error from the HTTP client which would
make it slightly hard to diagnose why it is broken.
## What's next
I implemented this for certificate authorities. I am sure there are
other places in the code base that take user submitted urls and could
also use this check. That is outside the scope of this particular PR.
But worthy to investigate in the near future.
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Security**
* Added SSRF protections for validating external URLs and blocking
private/IP-metadata ranges; dev mode can bypass checks for local testing
* **New Features**
* Introduced an SSRF-protected HTTP transport and an option to supply a
custom transport per client
* **Tests**
* Added comprehensive tests covering SSRF validation, dialing behavior,
and resolution edge cases
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39272
Changes file already added on another subtask
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#39265
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
Currently in Fleet Device Management, there is no support for Apple
iPods.
Eventhough iPods are considered vintage by Apple already, we still use
them and I know that in various companies they are still used as a low
cost device within the company. (eg. shops/warehouses to look up stock
levels)
Currently, enrolling an iPod through ABM, results in the device being
recognised as a Mac device.
With this PR, I'd like to add support for iPods, similar functionality
as iPhones to Fleet, simply as iOS device, which works fine. Considering
that all commands are the same (if available) and considering iPods
aren't updated anymore, I don't think we need to explicitly mention it,
perhaps just in docs, and add them to a separate category than iPhones.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [ ] Added/updated automated tests
- I have not added automated tests since it'd basically be a 1:1 copy of
iPhone tests
- [x] Manual QA for all new/changed functionality
> Follows up on discussion from #27263 with @noahtalerman
Manual QA:
- adding an iPod in ABM results in the device being recognised as iOS
<img width="1754" alt="overview"
src="https://github.com/user-attachments/assets/7681c613-2b34-489a-8b94-10eff8977e19"
/>
<img width="1766" alt="detail-abm"
src="https://github.com/user-attachments/assets/f88c8e84-e55f-4c5f-8998-8b6697b57abc"
/>
- after enrolling the iPod through setup, it is correctly synced with
Fleet and all commands are possible. (tried Restart, Rename device, push
apps)
<img width="1766" alt="ipod-post-sync"
src="https://github.com/user-attachments/assets/7668942e-b110-4c38-a448-b6027419507c"
/>
- enrollment video (can be uploaded if needed)
- manual enrollment works fine too (using Enroll url)
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Resolves#38484. This includes a CI job change to make sure we don't
introduce any more env vars that don't get proxied (and thus turned off
outside `--dev`).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
Manual QA touched hot paths, but did _not_ manually test every
FLEET_DEV_* environment variable change.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Centralized dev-mode environment management for consistent FLEET_DEV_*
handling and test-friendly overrides.
* Dev-mode allows targeted overrides for certain dev-only configuration
when running with --dev.
* **Chores**
* Migrated environment access to the centralized dev-mode helper across
the codebase.
* Added CI checks to enforce proper usage of FLEET_DEV_* variables.
* **Documentation**
* Added guidance on dev-mode environment variable rules and overrides.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38777
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#37710
Adds some logic to try to use the name/version/identifier from a bundle
that exactly matches the bundle identifier from the root pkg-info
element, if found. The loop continues even if that is found since
packageIDSet needs to be built.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
Tested against all installers from the google drive folder, prisma
access browser is the only package with changed metadata.
Before:
```
File: Prisma Access Browser-144.6.10.59-b070a3e0.pkg
- Name: 'PrismaAccessBrowserSoftwareUpdateAgent'
- Bundle Identifier: 'com.paloaltonetworks.PrismaAccessBrowserUpdater.Agent'
- Package IDs: 'com.paloaltonetworks.PrismaAccessBrowserUpdater,com.paloaltonetworks.PrismaAccessBrowserUpdater.Agent,com.talon-sec.Work,com.talon-sec.Work.framework,com.talon-sec.Work.framework.AlertNotificationService,com.talon-sec.Work.helper,com.talon-sec.Work.helper.plugin,com.talon-sec.Work.helper.renderer,org.sparkle-project.Downloader,org.sparkle-project.InstallerLauncher,org.sparkle-project.Sparkle,org.sparkle-project.Sparkle.Updater'
- Version: 144.6.10
```
After:
```
File: Prisma Access Browser-144.6.10.59-b070a3e0.pkg
- Name: 'Prisma Access Browser'
- Bundle Identifier: 'com.talon-sec.Work'
- Package IDs: 'com.paloaltonetworks.PrismaAccessBrowserUpdater,com.paloaltonetworks.PrismaAccessBrowserUpdater.Agent,com.talon-sec.Work,com.talon-sec.Work.framework,com.talon-sec.Work.framework.AlertNotificationService,com.talon-sec.Work.helper,com.talon-sec.Work.helper.plugin,com.talon-sec.Work.helper.renderer,org.sparkle-project.Downloader,org.sparkle-project.InstallerLauncher,org.sparkle-project.Sparkle,org.sparkle-project.Sparkle.Updater'
- Version: 144.6.10.59
```
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#37806
This PR creates an activity bounded context and moves the following HTTP
endpoint (including the full vertical slice) there:
`/api/latest/fleet/activities`
NONE of the other activity functionality is moved! This is an
incremental approach starting with just 1 API/service endpoint.
A significant part of this PR is tests. This feature is now receiving
significantly more unit/integration test coverage than before.
Also, this PR does not remove the `ListActivities` datastore method in
the legacy code. That will be done in the follow up PR (part 2 of 2).
This refactoring effort also uncovered an activity/user authorization
issue: https://fleetdm.slack.com/archives/C02A8BRABB5/p1768582236611479
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Activity listing API now available with query filtering, date-range
filtering, and type-based filtering
* Pagination support for activity results with cursor-based and
offset-based options
* Configurable sorting by creation date or activity ID in ascending or
descending order
* Automatic enrichment of activity records with actor user details
(name, email, avatar)
* Role-based access controls applied to activity visibility based on
user permissions
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#13698
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves oversight on initial bug fix:
https://github.com/fleetdm/fleet/issues/32920#issuecomment-3716712957
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#32920
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
Resolves#36909.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34376
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## New Fleet configuration settings
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35308
Changes file covered by merge for #35307
Changes Windows MDM enrollment and unenrollment logic to no longer send
notifications to orbit and as such, require customers to enroll through
Settings or Autopilot. When a new enrollment is detected, processes the
enrollment type(auto=autopilot, manual=settings app) as specified and
maps it to the user who performed the enrollment
This is unfortunately all done in the query processing code which is
where we currently process these enrollments. I didn't see a better way
to do it without significant rework that was unclear in scope and the
new logic is simple enough it didn't feel like the right time to do
that.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
**Related issue:** Resolves#35460, #35462
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added certificate templates for managing Android device certificates
at global and team levels
* Introduced API endpoints to create, list, retrieve, and delete
certificate templates
* Enabled GitOps workflow support for certificate template
specifications
* Implemented automatic variable substitution in certificate subjects
for host identifiers
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Scott Gress <scottmgress@gmail.com>
Co-authored-by: Scott Gress <scott@fleetdm.com>
