**Related issue:** Resolves#25574
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
---
## Summary
- Adds a new `splunk` log plugin that sends osquery logs directly to
Splunk's HTTP Event Collector (HEC) endpoint
- Eliminates the need for middleware like AWS Firehose when using Splunk
as a log destination
- Follows the same pattern as existing log destinations (Firehose, Kafka
REST, NATS, etc.)
- Includes `insecure_skip_verify` option for environments with
self-signed TLS certs
## UI changes
Follows the same pattern as the NATS log destination PR (#36527) --
adding "Splunk" to the display name, tooltip, and TypeScript type union.
No new components, pages, or styles.
### Manage automations modal -- "Log destination: Splunk"
<img width="822" height="527" alt="image"
src="https://github.com/user-attachments/assets/2533207f-fa95-4364-8ee0-3c39cd3e8e4d"
/>
### Query details page -- "Log destination: Splunk"
<img width="1905" height="662" alt="image"
src="https://github.com/user-attachments/assets/069a5005-f95c-4562-a819-fd8bdcc349f7"
/>
### Tooltip on hover
<img width="639" height="348" alt="image"
src="https://github.com/user-attachments/assets/809a47a6-b82a-4f45-b731-77b2d2c87947"
/>
### Edit query form -- "sent to your log destination: Splunk"
<img width="451" height="814" alt="image"
src="https://github.com/user-attachments/assets/b78b9a57-1f0c-4413-8b7c-654de1fd40a2"
/>
### Save new query modal -- "sent to your log destination: Splunk"
<img width="536" height="698" alt="image"
src="https://github.com/user-attachments/assets/d0a0ab01-66fe-4d63-9190-9c5e840e456d"
/>
---
### How it works
The Splunk writer (`server/logging/splunk.go`) implements the
`fleet.JSONLogger` interface. On startup it performs a health check
against the HEC `/services/collector/health` endpoint. On each `Write()`
call, it wraps each log entry in Splunk's HEC event format (adding
`time`, `index`, `source`, `sourcetype`), batches them up to 1 MB, and
POSTs to `/services/collector/event` with the `Authorization: Splunk
<token>` header. If a batch exceeds 1 MB it flushes and starts a new
one. Events over 1 MB are dropped with a log warning. Transient errors
(HTTP 503) are retried with exponential backoff (up to 8 retries).
### Configuration
```yaml
osquery:
status_log_plugin: splunk
result_log_plugin: splunk
splunk:
url: https://splunk.example.com:8088
token: <HEC token>
index: main
source: fleet
source_type: fleet:json
insecure_skip_verify: false # set true for self-signed certs
```
Or via environment variables:
```
FLEET_OSQUERY_STATUS_LOG_PLUGIN=splunk
FLEET_OSQUERY_RESULT_LOG_PLUGIN=splunk
FLEET_SPLUNK_URL=https://splunk.example.com:8088
FLEET_SPLUNK_TOKEN=<HEC token>
FLEET_SPLUNK_INDEX=main
FLEET_SPLUNK_SOURCE=fleet
FLEET_SPLUNK_SOURCE_TYPE=fleet:json
```
### Files changed
- `server/logging/splunk.go` -- Splunk HEC log writer with batching,
retry, and health check
- `server/logging/splunk_test.go` -- 9 unit tests
- `server/logging/splunk_integration_test.go` -- 3 integration tests
against real Splunk (gated by env var)
- `server/logging/logging.go` -- Added `SplunkConfig` and `case
"splunk"` to factory
- `server/config/config.go` -- Added `SplunkConfig` struct and config
flags
- `cmd/fleet/logging.go` -- Wired Splunk config into logging builder
- `server/fleet/app.go` -- Added `SplunkConfig` type for API responses
(excludes token)
- `server/service/service_appconfig.go` -- Added `case "splunk"` to
logging plugin validation
- `frontend/interfaces/config.ts` -- Added `"splunk"` to LogDestination
type
-
`frontend/components/LogDestinationIndicator/LogDestinationIndicator.tsx`
-- Added Splunk display name and tooltip
- `docs/Configuration/fleet-server-configuration.md` -- Splunk config
documentation
- `docs/Get started/FAQ.md` -- Updated plugin list
- `articles/log-destinations.md` -- Updated Splunk section with native
HEC docs
- `changes/25574-splunk-log-destination` -- Change file
## Test plan
### Unit tests (9 tests)
- [x] `TestSplunkWrite` -- sends 3 events, verifies HEC format, auth
header, index/source/sourcetype
- [x] `TestSplunkWriteEmpty` -- empty logs don't trigger HTTP request
- [x] `TestSplunkServerError` -- HEC 403 propagates as error
- [x] `TestSplunkHealthCheckFailure` -- constructor fails on bad health
- [x] `TestSplunkRecordTooBig` -- oversized events (>1MB) are dropped,
normal events still sent
- [x] `TestSplunkSplitBatchBySize` -- logs exceeding 1MB batch limit are
split into multiple requests
- [x] `TestSplunkRetryOnServiceUnavailable` -- 503 retried with backoff,
succeeds on 3rd attempt
- [x] `TestSplunkRetryExhausted` -- after 9 attempts (1 + 8 retries)
returns error
- [x] `TestSplunkMissingConfig` -- empty URL/token returns descriptive
error
### Integration tests (3 tests, gated by `SPLUNK_INTEGRATION_TEST=1`)
- [x] `TestSplunkIntegration` -- 3 events sent via writer, queried back
from Splunk REST API
- [x] `TestSplunkIntegrationBatch` -- 100 events in one Write(), all
confirmed indexed
- [x] `TestSplunkIntegrationBadToken` -- bad token Write() returns 403
### End-to-end test (macOS ARM64, real osquery agent)
1. Started Splunk Enterprise, MySQL, Redis via Docker
2. Built Fleet server from this branch with
`--osquery_status_log_plugin=splunk`
3. Set up Fleet, enrolled a real osquery 5.23.0 agent on this MacBook
4. **83 real osquery status log events indexed in Splunk** with correct
source/sourcetype/index
5. Each event contained full osquery data (`hostIdentifier`,
`host_uuid`, `calendarTime`, `severity`, `message`, `decorations`)
### Splunk showing real osquery events from Fleet
<img width="1910" height="861" alt="image"
src="https://github.com/user-attachments/assets/192490bf-d594-4424-a3e3-a18306892873"
/>
Generated with [Claude Code](https://claude.ai/code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added native Splunk HEC logging destination for status, result, and
audit logs.
* Updated the log destination UI to display **Splunk** with a dedicated
tooltip.
* Added Splunk HEC configuration (URL/token/index/source/source type)
including TLS verification control.
* **Bug Fixes**
* Improved log delivery with batching, retries for temporary HTTP
failures, and safeguards for oversized events.
* **Tests**
* Added unit tests and optional integration tests covering routing,
batching, retries, and error scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Relates to #48549
The S3 carve cleanup (server/datastore/s3, run by the
cleanups_then_aggregation cron) advanced ListObjectsV2 pagination using
the response's ContinuationToken — an echo of the request token —
instead of NextContinuationToken. On any bucket with more than one page
of objects this looped forever, hanging the entire serial cleanup cron
and stalling every cleanup/aggregation job ordered after it.
Replace the bucket-listing reconciliation with a direct HeadObject probe
per carve, which is exact and independent of listing order or object
counts:
- Only carves older than 24h with a completed upload are reconciled
(mirrors the MySQL carve store's floor; skips in-flight multipart
uploads). A carve is expired only on a definitive not-found; transient
or other probe errors leave it for a future run, so a carve whose object
still exists is never expired.
- Probes run with bounded concurrency; expirations are written in one
batched, retryable UPDATE (new ExpireCarves datastore method) rather
than one per carve.
- The number of carves reconciled per run is capped so a large backlog
drains across runs without any single run making unbounded S3 requests.
Add S3-carve-store-only server settings (the MySQL carve store is
unaffected):
- s3.carves_cleanup_disabled — skip reconciliation entirely
- s3.carves_cleanup_max_per_run — per-run cap (default 1000)
- s3.carves_cleanup_concurrency — concurrent probes (default 32)
Also log the expired count per run and fix the test bucket cleanup
helper to paginate. Adds unit tests (transient-error safety, partial
failure, concurrency) and a MySQL integration test for ExpireCarves.
This part of the document still references `macos_setup`. Should this be
changed to `setup_experience`?
May break older references, which should be updated.
**Related issue:** Resolves#42288
# Summary
This PR adds support for configuring an optional SES sender domain.
When the SES email backend is enabled, Fleet can now use a configured
sender domain for the `From` address instead of always deriving the
domain from `server.server_url`. If the setting is not provided, Fleet
keeps the existing behavior.
# Impact
This gives self-hosted operators a server-side SES configuration option
for email sending without changing UI-managed SMTP settings.
# Root cause
The SES sender path only generated `do-not-reply@<server host>` from the
Fleet server URL, so there was no way to override the sender domain
through server configuration.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [x] Setting(s) is/are explicitly excluded from GitOps
## Testing
- [x] `go test -tags full,fts5,netgo ./server/mail -run
'Test_(getFromSES|sesSender_SendEmail)$'`
- [x] `go test -tags full,fts5,netgo ./server/config -run
'TestConfig(SESSenderDomain|Roundtrip)$'`
- [x] `go test -tags full,fts5,netgo ./server/service -run
'TestService_EmailConfig$'`
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional SES sender domain configuration. Users can specify a
custom domain for the email "From" address via config or environment
variable; when unset it falls back to the server hostname.
* **Tests**
* Added and expanded tests to verify sender-domain precedence,
From-header generation, and related error cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/43811?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Added an optional HTTP-level pre-auth middleware (enabled using the
FLEET_OSQUERY_ALLOW_BODY_AUTH_FALLBACK server config) that validates
incoming osquery requests based on `Authorization: NodeKey <node_key>`
header.
For the following issue:
- https://github.com/fleetdm/fleet/issues/41419
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated tooling and documentation wording across scripts, utilities,
and infrastructure files to reflect the product terminology change from
“teams” to “fleets” (references now note “fleets” and indicate the
former “teams” naming).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Fix typo in the agent configuration documentation.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
@noahtalerman: For the following bug:
- https://github.com/fleetdm/fleet/issues/43160
## Summary
- Condense and relocate the S3 `endpoint_url` override warning from the
general `## S3` section header into the individual `endpoint_url`
setting sections
- The primary warning now lives in
`s3_software_installers_endpoint_url`; the `s3_carves_endpoint_url` and
`packaging_s3_endpoint_url` sections cross-reference it
- Condense region-discovery notes similarly (primary in
`s3_software_installers_region`, cross-references elsewhere)
- Condense the Reference Architectures S3 section to a single paragraph
## Changes
**`docs/Configuration/fleet-server-configuration.md`:**
- Removed the large callout block under `## S3`
- Added concise warning to `s3_software_installers_endpoint_url` with
link to [#42349](https://github.com/fleetdm/fleet/issues/42349)
- Added concise note to `s3_software_installers_region`
- Updated `s3_carves_endpoint_url`, `s3_carves_region`,
`packaging_s3_endpoint_url`, and `packaging_s3_region` with short
cross-references
**`docs/Deploy/Reference-Architectures.md`:**
- Condensed the `S3 configuration for AWS deployments` section to a
single paragraph with link to the config reference
## Related
- Relates to #42349
---
Built for
[ntalerman](https://fleetdm.slack.com/archives/D0AEA6U4SM9/p1775051868003859?thread_ts=1775051234.724799&cid=D0AEA6U4SM9)
by [Kilo for Slack](https://kilo.ai/features/slack-integration)
---------
Co-authored-by: kiloconnect[bot] <240665456+kiloconnect[bot]@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38013
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
See
https://drive.google.com/file/d/1zeFNLuf_rT5FWzDiYyL2_hbIBW2neba-/view?usp=drive_link
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* GitOps variables in JSON configuration profiles (Apple DDM
declarations and Android profiles) are now automatically escaped for
JSON special characters, ensuring proper handling of sensitive values.
* **Tests**
* Added JSON configuration profile escaping validation to the enterprise
GitOps integration test suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The glob/ `paths:` feature was added across PRs #40799 (scripts,
configuration_profiles) and #41141 (labels, policies, reports) but the
customer-facing YAML reference at `docs/Configuration/yaml-files.md` was
never updated to document it.
This adds documentation for:
- New `path:` vs `paths:` reference section explaining the difference
between singular (literal file) and plural (glob pattern)
- Characters to avoid in filenames when using `path:` (`*`, `?`, `[`,
`{`)
- `scripts` section: `paths:` support with `.sh`/`.ps1` filter note
- `apple_settings`/`windows_settings` section: `path:` vs `paths:` wit
examples
- `android_settings` section: `paths:` support note
- `labels`, `policies`, `reports` sections: `paths:` support notes
- `paths:` glob examples added to the controls YAML example for both
scripts, apple_settings, and windows_settings
Discovered via customer-sonet whose Windows profile filenames use
brackets as a CSP naming convention (e.g.
`[AllowSpotlightCollection].xml`), which are rejected by `path:`
validation as glob metacharacters.
See related bug report for the false positive on literal filenames.
https://github.com/fleetdm/fleet/issues/43598
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#43598
Co-authored-by: Rachael Shaw <r@rachael.wtf>
- @noahtalerman: We decided to stop calling the settings experimental
and just warn in the docs what happens if you turn it on. That way we’re
not calling them “experimental” which feels unsafe. They're not
experimental; they're just deliberately allowing custom profiles.
## Summary
- Clarifies in the MySQL docs intro that read replica configuration is
fully independent — no values are inherited from the primary config
- Adds explicit callout that `mysql_read_replica_region` must be set
separately when using IAM authentication
- Adds note to `mysql_tls_config` explaining that this setting is
typically not needed for RDS IAM auth since Fleet uses bundled RDS CA
certificates (including GovCloud regions)
These changes address real-world customer confusion where a GovCloud
deployment failed because `FLEET_MYSQL_READ_REPLICA_REGION` was not set
independently of `FLEET_MYSQL_REGION`.
Note: related to #39832
## Testing
Documentation-only change — no code impact.
