74 Commits
Author SHA1 Message Date
Lucas Manuel Rodriguez 24d0fa71a1 Remove unused prometheus from main docker compose (#50053)
It's not used by CI or tests. Probably a leftover from when we tested
some Prometheus integration manually.

While I was at it fixed some typos on the `writing.md` file.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Removed the Prometheus monitoring service from the local development
environment.
* Removed the associated metrics scraping configuration for the
application.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-31 10:11:36 -03:00
Lucas Manuel Rodriguez c37f7463ad Remove unused cadvisor from dev/test docker-compose.yml (#49432)
Reasons to delete:
- We are not using this docker image.
- It's polluting the `docker compose up` logs with errors, e.g.:
```
cadvisor-1            | E0716 17:14:28.835089       1 manager.go:1116] Failed to create existing container: /docker/9ac0ef6fad63613bf90dbfb80e4fcf7affc4a378bbf1c2680f9fa0a587db783f: failed to identify the read-write layer ID for container "9ac0ef6fad63613bf90dbfb80e4fcf7affc4a378bbf1c2680f9fa0a587db783f". - open /var/lib/docker/image/overlayfs/layerdb/mounts/9ac0ef6fad63613bf90dbfb80e4fcf7affc4a378bbf1c2680f9fa0a587db783f/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.839077       1 manager.go:1116] Failed to create existing container: /docker/891666c60086c1a695860bf5ea4a1eaf4ada94d902ee7392ddfab1bfd7ed5876: failed to identify the read-write layer ID for container "891666c60086c1a695860bf5ea4a1eaf4ada94d902ee7392ddfab1bfd7ed5876". - open /var/lib/docker/image/overlayfs/layerdb/mounts/891666c60086c1a695860bf5ea4a1eaf4ada94d902ee7392ddfab1bfd7ed5876/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.841743       1 manager.go:1116] Failed to create existing container: /docker/bbfd8df39c02cec5d4ad665f4bb23faa5fc374efd7cb1ae0ff4864474c8a1b63: failed to identify the read-write layer ID for container "bbfd8df39c02cec5d4ad665f4bb23faa5fc374efd7cb1ae0ff4864474c8a1b63". - open /var/lib/docker/image/overlayfs/layerdb/mounts/bbfd8df39c02cec5d4ad665f4bb23faa5fc374efd7cb1ae0ff4864474c8a1b63/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.844137       1 manager.go:1116] Failed to create existing container: /docker/62cad7aa07068e9ad14319a170622c4658008fd394347aa60e84f79727a03d29: failed to identify the read-write layer ID for container "62cad7aa07068e9ad14319a170622c4658008fd394347aa60e84f79727a03d29". - open /var/lib/docker/image/overlayfs/layerdb/mounts/62cad7aa07068e9ad14319a170622c4658008fd394347aa60e84f79727a03d29/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.846207       1 manager.go:1116] Failed to create existing container: /docker/6fc30f822f6ccf81a192b5d385b30f5ed02f162df1c9823774405e0517be3cfe: failed to identify the read-write layer ID for container "6fc30f822f6ccf81a192b5d385b30f5ed02f162df1c9823774405e0517be3cfe". - open /var/lib/docker/image/overlayfs/layerdb/mounts/6fc30f822f6ccf81a192b5d385b30f5ed02f162df1c9823774405e0517be3cfe/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.849253       1 manager.go:1116] Failed to create existing container: /docker/a7cec691d2ba6dc8cb8b034729957e6d9bc6dca06813953288d4430cf8d82d28: failed to identify the read-write layer ID for container "a7cec691d2ba6dc8cb8b034729957e6d9bc6dca06813953288d4430cf8d82d28". - open /var/lib/docker/image/overlayfs/layerdb/mounts/a7cec691d2ba6dc8cb8b034729957e6d9bc6dca06813953288d4430cf8d82d28/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.851095       1 manager.go:1116] Failed to create existing container: /docker/160d97ff57081d3439843b5e2fd6bea00ef3ee744de51d01be1f6c0c90dec97d: failed to identify the read-write layer ID for container "160d97ff57081d3439843b5e2fd6bea00ef3ee744de51d01be1f6c0c90dec97d". - open /var/lib/docker/image/overlayfs/layerdb/mounts/160d97ff57081d3439843b5e2fd6bea00ef3ee744de51d01be1f6c0c90dec97d/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.853019       1 manager.go:1116] Failed to create existing container: /docker/9f5f2f98be4da013911de372e808e33ae93b8f5cb2219adb681f31ee3c1497ed: failed to identify the read-write layer ID for container "9f5f2f98be4da013911de372e808e33ae93b8f5cb2219adb681f31ee3c1497ed". - open /var/lib/docker/image/overlayfs/layerdb/mounts/9f5f2f98be4da013911de372e808e33ae93b8f5cb2219adb681f31ee3c1497ed/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.854765       1 manager.go:1116] Failed to create existing container: /docker/c637f0bf6708caeec3fe3b9b610204e84b8b4529cf770c09d6ca2e82f5b681e3: failed to identify the read-write layer ID for container "c637f0bf6708caeec3fe3b9b610204e84b8b4529cf770c09d6ca2e82f5b681e3". - open /var/lib/docker/image/overlayfs/layerdb/mounts/c637f0bf6708caeec3fe3b9b610204e84b8b4529cf770c09d6ca2e82f5b681e3/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.856713       1 manager.go:1116] Failed to create existing container: /docker/74953fdfba3c534515113bedef7f17ce01a7feef27ec22000d52ff4f98a538e9: failed to identify the read-write layer ID for container "74953fdfba3c534515113bedef7f17ce01a7feef27ec22000d52ff4f98a538e9". - open /var/lib/docker/image/overlayfs/layerdb/mounts/74953fdfba3c534515113bedef7f17ce01a7feef27ec22000d52ff4f98a538e9/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.858256       1 manager.go:1116] Failed to create existing container: /docker/96e468826a675094cf686619e9560a5ed0e089bbca8f3c9c41a5d432d7e7e235: failed to identify the read-write layer ID for container "96e468826a675094cf686619e9560a5ed0e089bbca8f3c9c41a5d432d7e7e235". - open /var/lib/docker/image/overlayfs/layerdb/mounts/96e468826a675094cf686619e9560a5ed0e089bbca8f3c9c41a5d432d7e7e235/mount-id: no such file or directory
cadvisor-1            | E0716 17:14:28.859877       1 manager.go:1116] Failed to create existing container: /docker/785d4800c3249c155f6b921ef554f49851920e5d6a295e1d2ef12fb85a327935: failed to identify the read-write layer ID for container "785d4800c3249c155f6b921ef554f49851920e5d6a295e1d2ef12fb85a327935". - open /var/lib/docker/image/overlayfs/layerdb/mounts/785d4800c3249c155f6b921ef554f49851920e5d6a295e1d2ef12fb85a327935/mount-id: no such file or directory
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Removed the cAdvisor monitoring service from the Docker Compose
configuration.
* The application no longer starts cAdvisor or exposes its monitoring
endpoint by default.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-16 17:06:00 -03:00
Steven Palmesano 22c732a249 Update build instructions (#46339)
Small updates to our build instructions that I ran into while testing.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated macOS M1 setup instructions with simplified platform
configuration guidance for MySQL containers.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/fleetdm/fleet/pull/46339?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-22 09:43:59 -05:00
Tim Lee 60cd943643 Add Redis 8.x/Valkey CI test matrix and optional Valkey loadtests (#47180) 2026-06-12 07:26:22 -06:00
Victor Lyuboslavsky 84b16a08f8 Set Redis explicitly to 6.2 for clarity (it was already aliased to latest 6.2) (#44501)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #43928 



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated Redis image version to 6.2 across Docker Compose
configurations for improved stability and compatibility.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-30 11:29:05 -05:00
Victor Lyuboslavsky 7ba762ebec Set MySQL binary logs expiration to save disk space in dev (#43024)
Manually tested.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated database container configurations to manage log retention.
Binary logs will now automatically expire after 24 hours in local
development environments, helping manage disk space usage during testing
and development workflows.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-06 08:57:23 -05:00
Victor Lyuboslavsky ea22c8087b Bind docker ports to 127.0.0.1 (#42232)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #42226

When doing dev in a remote environment, like a public cloud VM, don't
expose ports to the public.
This is a contributor security improvement.

The localstack fail is present on main, and was not caused by this
change:
https://github.com/fleetdm/fleet/actions/runs/23439965808/job/68187858627

# Checklist for submitter

## Testing

- [x] QA'd all new/changed functionality manually


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Docker Compose configuration updated across multiple services (Redis,
MySQL, mail, monitoring, and storage services) to restrict port bindings
to localhost only instead of all network interfaces.
* Documentation Docker Compose examples updated to reflect
localhost-only port binding for core services.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-03-23 12:30:23 -05:00
Scott Gress 8ea6f338de Pin Localstack version (#42253)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #42252 

Pins the Localstack image to the last-known-good version (4.5) before
they 🔪 'd the community edition and started requiring an auth token. I
also added a "wait for localstack" as an initial debugging step, and
left it in to catch similar future issues. It's probably redundant since
there likely _is_ no future for Fleet and Localstack beyond this, but it
take milliseconds and would catch any other weird Localstack failures
so, why not.
2026-03-23 12:11:55 -05:00
Victor Lyuboslavsky 8c2ad7f901 Run multiple independent Fleet dev servers in parallel (#41865)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #41848 

Docs updates: https://github.com/fleetdm/fleet/pull/41868/changes

# Checklist for submitter

- changes not needed since this is a dev environment and test issue

## Testing

- [x] QA'd all new/changed functionality manually

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Enhanced test infrastructure to support environment-variable-based
configuration for SAML, mail, database, and S3 services, enabling more
flexible and dynamic test setups.

* **Chores**
* Updated Docker Compose configuration to use environment variables for
service ports, allowing runtime customization while maintaining backward
compatibility with default values.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-03-18 13:58:58 -05:00
Ian Littman 4a4e55efc0 Bump RustFS dependency to latest version (#40843)
This is just a `fleetctl preview` deps + docker-compose deps bump.
Tested both.

## Testing

- [x] QA'd all new/changed functionality manually
2026-03-11 19:17:48 -05:00
Ian Littman 7d4acdc5c4 Bump supported MySQL versions (#40892)
Fixes #40975.

8.0.32 (was running in Aurora managed cloud at the time) -> 8.0.39 (what
we're running now) 8.0.36 -> 8.0.44 (latest 8.0.x version supported by
Aurora; holding off on 8.0.45 until Aurora supports it) 8.4.7 -> 8.4.8
9.5.0 -> 9.6.0

Also bumped the supported Aurora version from 3.07.0 to 3.08.2 to match
what we're running in managed cloud right now

Fleet might work on older patch versions but we'll no longer dev/test on
them. MySQL 9.x not testing previous minor versions matches with our
previous approach for that version.

Since these are all patch/minor bumps (and the overnight build cases are
patch bumps/are covered by AWS envs) automated testing should be
sufficient here.
2026-03-04 12:25:20 -06:00
Ian Littman ec39dc8c06 Bump to latest release of RustFS docker-compose dependency (#38469)
Validated reads/writes locally on standard docker-compose.
2026-01-19 11:20:46 -06:00
0a9537d2e3 Update RustFS Docker image to 1.0.0-alpha.73 (#37099)
**Related issue:** Resolves #

## Description

Bumps RustFS image version from `1.0.0-alpha.72` to `1.0.0-alpha.73` in
docker-compose configurations.

**Files updated:**
- `docker-compose.yml` - root development environment
- `tools/osquery/in-a-box/docker-compose.yml` - Fleet preview
environment

RustFS provides S3-compatible object storage for file carving and
software installer features in development/testing environments.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes

## Testing

- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)

- [ ] QA'd all new/changed functionality manually

For unreleased bug fixes in a release candidate, one of:

- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed

## Database migrations

- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).

## New Fleet configuration settings

- [ ] Setting(s) is/are explicitly excluded from GitOps

If you didn't check the box above, follow this checklist for
GitOps-enabled settings:

- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled

## fleetd/orbit/Fleet Desktop

- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))

<!-- START COPILOT ORIGINAL PROMPT -->



<details>

<summary>Original prompt</summary>

> ## Summary
> Update RustFS Docker image references from version `1.0.0-alpha.72` to
the latest version `1.0.0-alpha.73` in all docker-compose configuration
files.
> 
> ## Files to Update
> Based on the code search, the following files need to be updated:
> 
> 1. `docker-compose.yml` (root level)
> 2. `docs/solutions/docker-compose/docker-compose.yml`
> 3. `tools/osquery/in-a-box/docker-compose.yml`
> 
> ## Changes Required
> For each file, update the RustFS image reference:
> - **FROM:** `rustfs/rustfs:1.0.0-alpha.72`
> - **TO:** `rustfs/rustfs:1.0.0-alpha.73`
> 
> ## Context
> RustFS is used as an S3-compatible object storage backend in Fleet's
development and testing environments. Keeping the version up-to-date
ensures we have the latest bug fixes and improvements from the RustFS
project.
> 
> ## Verification
> After making these changes, verify that:
> 1. All docker-compose files can start successfully
> 2. The S3-compatible storage functionality works as expected
> 3. File carving and software installer storage features continue to
work properly


</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

*This pull request was created as a result of the following prompt from
Copilot chat.*
> ## Summary
> Update RustFS Docker image references from version `1.0.0-alpha.72` to
the latest version `1.0.0-alpha.73` in all docker-compose configuration
files.
> 
> ## Files to Update
> Based on the code search, the following files need to be updated:
> 
> 1. `docker-compose.yml` (root level)
> 2. `docs/solutions/docker-compose/docker-compose.yml`
> 3. `tools/osquery/in-a-box/docker-compose.yml`
> 
> ## Changes Required
> For each file, update the RustFS image reference:
> - **FROM:** `rustfs/rustfs:1.0.0-alpha.72`
> - **TO:** `rustfs/rustfs:1.0.0-alpha.73`
> 
> ## Context
> RustFS is used as an S3-compatible object storage backend in Fleet's
development and testing environments. Keeping the version up-to-date
ensures we have the latest bug fixes and improvements from the RustFS
project.
> 
> ## Verification
> After making these changes, verify that:
> 1. All docker-compose files can start successfully
> 2. The S3-compatible storage functionality works as expected
> 3. File carving and software installer storage features continue to
work properly

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in
our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: iansltx <472804+iansltx@users.noreply.github.com>
Co-authored-by: Ian Littman <iansltx@gmail.com>
2025-12-11 10:59:46 -06:00
Ian Littman fe2a9a867e Swap minio to rustfs (#36851)
Resolves #36909.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.

## Testing

- [x] Added/updated automated tests

- [x] QA'd all new/changed functionality manually
2025-12-10 10:03:48 -06:00
Ian LittmanandNoah Talerman f91aa591b0 Target Redis 6 everywhere rather than a mix of 5 and 6 (#35373)
Redis 5 has been EOL for a few years, and didn't get updates for the
latest high-severity CVEs. We're already using 6 in most places
(fleetctl preview, recommended reference architectures, managed cloud
environments) so it's safe to set 6 as the new minimum.

Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
2025-11-10 17:05:44 -06:00
Jonathan RudenbergandScott Gress 48760fec58 Add support for reading private_key from AWS Secrets Manager (#31134)
Adds support for reading server `private_key` from AWS Secrets Manager.
Combined with #31075, this should allow removing all common sensitive
secrets from the environment/config (if I missed any let me know). This
works with localstack for local development (set
`AWS_ENDPOINT_URL=$LOCALSTACK_URL`, `AWS_ACCESS_KEY_ID=test`, and
`AWS_SECRET_ACCESS_KEY=test`).

I did not include config options for `AWS_ACCESS_KEY_ID` and
`AWS_SECRET_ACCESS_KEY` because they are a bad practice vs role
credentials and defeat the purpose of this feature which is to remove
secrets from the environment/config.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Scott Gress <scott@fleetdm.com>
2025-09-09 16:56:35 -05:00
Victor Lyuboslavsky 98954e8113 Make MySQL test port configurable (#31782)
Fixes #31781 

Only test-related changes.

I found this useful for agentic AI workflows. For example, you have an
AI agent debugging/rerunning a test. Meanwhile, you can spin up another
`mysql_test` instance in another workarea and work there in parallel.
2025-08-12 15:45:43 +02:00
Lucas Manuel Rodriguez 404f0d3ac0 Migrate from aws-sdk-go v1 to v2 (#30308)
#29482

[Migrate to the AWS SDK for Go
v2](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html)
documents how to migrate codebases.

QA on features that use AWS SDK Go:
- Bootstrap package:
  - upload:  
  - download: 
  - cleanup: 
- Software (upload, download, installation, etc.) 
  - Cloudfront: Luckly, this feature was already using aws-sdk-go-v2.
- Carves 
- Logging:
	- Firehose 
	- Kinesis 
- Lambda  (tested result logs to a lambda function on our AWS Dogfood
account)
- Email:
	- Amazon SES TODO ⚠️ (this is what Dogfood uses and a few customers)
- We cannot easily test locally, we can use dogfood or load testing
(AWS) environments.

---

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
2025-06-30 17:45:39 -03:00
Lucas Manuel Rodriguez 148d914f01 Optimize software_titles query to use indexes (#25722)
For #25160.

Measured improvement by splitting the MySQL query into two queries to
use the indexes more efficiently:
- ~8s vs ~100ms for ~30k entries in software_titles for ~1.7k incoming
software without bundle_identifier (linux software).
- ~1.64s vs ~2ms for 25k entries in software_titles and ~500 incoming
new software with bundle_identifier + ~200 new software without
bundle_identifier (macOS software).

---

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [X] Manual QA for all new/changed functionality
2025-01-23 15:48:21 -03:00
Lucas Manuel Rodriguez 963cc7e22c Automatic install custom packages (#25021)
#24385

Some docs change here: https://github.com/fleetdm/fleet/pull/25026.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-12-27 15:10:28 -03:00
Lucas Manuel Rodriguez f6b2ec8793 Remove unused toxiproxy service (#21554)
Everyone running this thing by default when doing `docker compose up`
does not make sense. I added it and used once a few years ago.
2024-08-26 13:28:51 -03:00
Dante Catalfamo 5e1a3d03ae MySQL 8.0 Migration (#20225)
#17249
2024-07-22 16:27:36 -04:00
Lucas Manuel Rodriguez 28ca463d13 iOS/iPadOS as platforms/labels (#20126)
#19963 

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Manual QA for all new/changed functionality

---

# API changes for dashboard UI changes

## Main dashboard page

`GET /api/latest/fleet/host_summary?low_disk_space=32` (see
`ios`/`ipados` platforms and `iOS`/`iPadOS` labels)
```json
{
  "totals_hosts_count": 9,
  "online_count": 0,
  "offline_count": 9,
  "mia_count": 0,
  "missing_30_days_count": 0,
  "new_count": 0,
  "all_linux_count": 2,
  "low_disk_space_count": 3,
  "builtin_labels": [
    {
      "id": 1,
      "name": "macOS 14+ (Sonoma+)",
      "description": "macOS hosts with version 14 and above",
      "label_type": "builtin"
    },
    {
      "id": 7,
      "name": "All Hosts",
      "description": "All hosts which have enrolled in Fleet",
      "label_type": "builtin"
    },
    {
      "id": 8,
      "name": "macOS",
      "description": "All macOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 9,
      "name": "Ubuntu Linux",
      "description": "All Ubuntu hosts",
      "label_type": "builtin"
    },
    {
      "id": 10,
      "name": "CentOS Linux",
      "description": "All CentOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 11,
      "name": "MS Windows",
      "description": "All Windows hosts",
      "label_type": "builtin"
    },
    {
      "id": 12,
      "name": "Red Hat Linux",
      "description": "All Red Hat Enterprise Linux hosts",
      "label_type": "builtin"
    },
    {
      "id": 13,
      "name": "All Linux",
      "description": "All Linux distributions",
      "label_type": "builtin"
    },
    {
      "id": 14,
      "name": "chrome",
      "description": "All Chrome hosts",
      "label_type": "builtin"
    },
    {
      "id": 15,
      "name": "iOS",
      "description": "All iOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 16,
      "name": "iPadOS",
      "description": "All iPadOS hosts",
      "label_type": "builtin"
    }
  ],
  "platforms": [
    {
      "platform": "darwin",
      "hosts_count": 3
    },
    {
      "platform": "ios",
      "hosts_count": 1
    },
    {
      "platform": "ipados",
      "hosts_count": 1
    },
    {
      "platform": "rhel",
      "hosts_count": 1
    },
    {
      "platform": "ubuntu",
      "hosts_count": 1
    },
    {
      "platform": "windows",
      "hosts_count": 2
    }
  ]
}
```

## After selecting a platform

`GET /api/latest/fleet/host_summary?platform=ios&low_disk_space=100`
(similar with `ipados`)
```json
{
  "totals_hosts_count": 1,
  "online_count": 0,
  "offline_count": 1,
  "mia_count": 0,
  "missing_30_days_count": 0,
  "new_count": 0,
  "all_linux_count": 0,
  "low_disk_space_count": 1,
  "builtin_labels": [
    {
      "id": 1,
      "name": "macOS 14+ (Sonoma+)",
      "description": "macOS hosts with version 14 and above",
      "label_type": "builtin"
    },
    {
      "id": 7,
      "name": "All Hosts",
      "description": "All hosts which have enrolled in Fleet",
      "label_type": "builtin"
    },
    {
      "id": 8,
      "name": "macOS",
      "description": "All macOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 9,
      "name": "Ubuntu Linux",
      "description": "All Ubuntu hosts",
      "label_type": "builtin"
    },
    {
      "id": 10,
      "name": "CentOS Linux",
      "description": "All CentOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 11,
      "name": "MS Windows",
      "description": "All Windows hosts",
      "label_type": "builtin"
    },
    {
      "id": 12,
      "name": "Red Hat Linux",
      "description": "All Red Hat Enterprise Linux hosts",
      "label_type": "builtin"
    },
    {
      "id": 13,
      "name": "All Linux",
      "description": "All Linux distributions",
      "label_type": "builtin"
    },
    {
      "id": 14,
      "name": "chrome",
      "description": "All Chrome hosts",
      "label_type": "builtin"
    },
    {
      "id": 15,
      "name": "iOS",
      "description": "All iOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 16,
      "name": "iPadOS",
      "description": "All iPadOS hosts",
      "label_type": "builtin"
    }
  ],
  "platforms": [
    {
      "platform": "ios",
      "hosts_count": 1
    }
  ]
}
```

### To populate list of MDM solutions of a selected platform

`GET /api/latest/fleet/hosts/summary/mdm\?platform=ios` (similar with
`ipados`)

```json
{
  "counts_updated_at": "2024-06-27T21:56:45Z",
  "mobile_device_management_enrollment_status": {
    "enrolled_manual_hosts_count": 0,
    "enrolled_automated_hosts_count": 1,
    "pending_hosts_count": 0,
    "unenrolled_hosts_count": 0,
    "hosts_count": 1
  },
  "mobile_device_management_solution": [
    {
      "id": 1,
      "name": "Fleet",
      "server_url": "https://lucas-fleet.ngrok.app/mdm/apple/mdm",
      "hosts_count": 1
    }
  ]
}
```

### To populate OS versions of a selected platform

`GET /api/latest/fleet/os_versions?platform=ipados` (similar with `ios`)
```json
{
  "meta": {
    "has_next_results": false,
    "has_previous_results": false
  },
  "count": 1,
  "counts_updated_at": "2024-06-27T21:36:12Z",
  "os_versions": [
    {
      "os_version_id": 7,
      "hosts_count": 1,
      "name": "iPadOS 17.5.1",
      "name_only": "iPadOS",
      "version": "17.5.1",
      "platform": "ipados",
      "vulnerabilities": []
    }
  ]
}
```

## Filtering hosts by the two new `iOS`/`iPadOS` labels

Works the same as with other labels.
2024-07-08 18:05:29 -03:00
Victor Lyuboslavsky f761827850 Move CalculateAggregatedPerfStatsPercentiles reads to the replica (#19206)
Move CalculateAggregatedPerfStatsPercentiles reads to the replica
#18838

I manually tested the aggregated query stats change by using a read
replica.
https://github.com/fleetdm/fleet/blob/main/tools/mysql-replica-testing/README.md
- But set `master_delay=0` due to issue
https://github.com/fleetdm/fleet/issues/19272
- Run a saved query as a live query, and see that its stats in
`aggregated_stats` table are updated.


# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-05-31 07:08:31 -05:00
Lucas Manuel Rodriguez b7aac2cfab Fix bundle_identifier for common software in osquery-perf (#19023)
bundle_identifier should be unique for the common software (should not
have hostname name in it).
2024-05-24 09:26:42 -03:00
Lucas Manuel Rodriguez 1afb015f6c Test DB migrations with Percona XtraDB MySQL server 5.7.25 (#16320)
#15881

This PR adds a script to test DB migrations with Percona XtraDB 5.7.25.

PS: To run this test before we merge this PR to `main` you will need to
change step 2 (`Make sure to be on latest main`), instead of `main` use
this branch `15881-test-migrations-with-percona`.
2024-01-29 14:21:37 -03:00
Tim Lee 0557f10ac5 14729 smtp settings validation for TLS (#15029) 2023-11-21 11:48:21 -07:00
Lucas Manuel RodriguezandTim Lee 9facf144dc Schedule queries on hosts even if automations off (given we now store results) (#14411)
This is ready to go.

---------

Co-authored-by: Tim Lee <timlee@fleetdm.com>
2023-10-11 15:20:06 -03:00
Luke Heath 79a9451952 Set MySQL table-definition-cache to 2048 (#13864) 2023-09-11 15:04:43 -05:00
a23d208b1d gate DEP enrollment behind SSO when configured (#11309)
#10739

Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
2023-04-27 09:43:20 -03:00
Martin Angers 9aab3d628c Move Redis cluster docker yml to separate file (#11162) 2023-04-12 15:14:28 -04:00
Lucas Manuel Rodriguez 40265d0e6f Fix SMTP e-mail send when SMTP server has credentials (#10758)
#9609

This PR also fixes #10777.

The issue is: We were using `svc.AppConfig` instead of
`svc.ds.AppConfig` to retrieve the SMTP credentials.
`svc.AppConfig` obfuscates credentials, whereas `svc.ds.AppConfig` does
not.
To help prevent this from happening again I've renamed `svc.AppConfig`
to `svc.AppConfigObfuscated`.
I've also added a new test SMTP server
(https://github.com/axllent/mailpit) that supports Basic Authentication
and tests that make use of it to catch these kind of bugs (the tests are
executed when running `go test` with `MAIL_TEST=1`).

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-03-28 15:23:15 -03:00
Lucas Manuel Rodriguez 78df7b6f70 Add guide on how to simulate slow connections to redis and mysql (#9140)
* Add guide on how to simulate slow connections to redis and mysql

* Add pageOrderInSection
2023-01-03 17:29:40 -03:00
9191f4ce66 Add Apple MDM functionality (#7940)
* WIP

* Adding DEP functionality to Fleet

* Better organize additional MDM code

* Add cmdr.py and amend API paths

* Fix lint

* Add demo file

* Fix demo.md

* go mod tidy

* Add munki setup to Fleet

* Add diagram to demo.md

* Add fixes

* Update TODOs and demo.md

* Fix cmdr.py and add TODO

* Add endpoints to demo.md

* Add more Munki PoC/demo stuff

* WIP

* Remove proposals from PoC

* Replace prepare commands with fleetctl commands

* Update demo.md with current state

* Remove config field

* Amend demo

* Remove Munki setup from MVP-Dogfood

* Update demo.md

* Add apple mdm commands (#7769)

* fleetctl enqueue mdm command

* fix deps

* Fix build

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>

* Add command to upload installers

* go mod tidy

* fix subcommands help

There is a bug in urfave/cli where help text is not generated properly when subcommands
are nested too deep.

* Add support for installing apps

* Add a way to list enrolled devices

* Add dep listing

* Rearrange endpoints

* Move DEP routine to schedule

* Define paths globally

* Add a way to list enrollments and installers

* Parse device-ids as comma-separated string

* Remove unused types

* Add simple commands and nest under enqueue-command

* Fix simple commands

* Add help to enqueue-command

* merge apple_mdm database

* Fix commands

* update nanomdm

* Split nanomdm and nanodep schemas

* Set 512 MB in memory for upload

* Remove empty file

* Amend profile

* Add sample commands

* Add delete installers and fix bug in DEP profile assigning

* Add dogfood.md deployment guide

* Update schema.sql

* Dump schema with MySQL 5

* Set default value for authenticate_at

* add tokens to enrollment profiles

When a device downloads an MDM enrollment profile, verify the token passed
as a query parameter. This ensures untrusted devices don't enroll with
our MDM server.

- Rename enrollments to enrollment profiles. Enrollments is used by nano
  to refer to devices that are enrolled with MDM
- Rename endpoint /api/<version>/fleet/mdm/apple/enrollments to ../enrollmentprofiles
- Generate a token for authentication when creating an enrollment profile
- Return unauthorized if token is invalid when downloading an enrollment profile from /api/mdm/apple/enroll?token=

* remove mdm apple server url

* update docs

* make dump-test-schema

* Update nanomdm with missing prefix table

* Add docs and simplify changes

* Add changes file

* Add method docs

* Fix compile and revert prepare.go changes

* Revert migration status check change

* Amend comments

* Add more docs

* Clarify storage of installers

* Remove TODO

* Remove unused

* update dogfood.md

* remove cmdr.py

* Add authorization tests

* Add TODO comment

* use kitlog for nano logging

* Add yaml tags

* Remove unused flag

* Remove changes file

* Only run DEP routine if MDM is enabled

* Add docs to all new exported types

* Add docs

* more nano logging changes

* Fix unintentional removal

* more nano logging changes

* Fix compile test

* Use string for configs and fix config test

* Add docs and amend changes

* revert changes to basicAuthHandler

* remove exported BasicAuthHandler

* rename rego authz type

* Add more information to dep list

* add db tag

* update deps

* Fix schema

* Remove unimplemented

Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
2022-10-05 19:53:54 -03:00
Zach Wasserman db7d1c5bf5 Strongly discourage use of MariaDB (#7699) 2022-09-23 11:18:34 -07:00
Lucas Manuel Rodriguez 142f19f286 Turn mysql event scheduler off (#6640) 2022-07-13 13:50:58 -03:00
Lucas Manuel Rodriguez 22382c757c Add better support for M1 to docker-compose.yml and add fleet-dev target to Makefile (#6432)
* Add better support for M1 to docker-compose.yml and add fleet-dev target

* Amend comment
2022-06-29 14:38:23 -03:00
Lucas Manuel Rodriguez fc7650c4f8 Fix mock tests set test name (#6345)
* Set mock test on live query mocked tests

* Use MySQL 5.7 as default in docker-compose.yml
2022-06-23 11:34:52 -03:00
Zach Wasserman 26eae438f6 Document supported MySQL versions (#5421)
- Pin versions in development and CI.
2022-04-27 16:21:16 -07:00
Martin Angers fd0cd153ce Fix SSO paths to always use /v1/ instead of /latest/ (#5246) 2022-04-20 12:46:45 -04:00
Martin Angers 90b15071a4 Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
Zach Wasserman 35523017c9 Make migrations compatible with GTID replication (#2615)
* Make migrations compatible with GTID replication

Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).

Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.

* Update docker-compose formatting

* if exists
2021-10-21 07:46:02 -03:00
Martin Angers eda6d60be8 Add replicas to the redis docker-compose setup. (#2511) 2021-10-18 08:33:26 -04:00
Benjamin Edwards 16c5823692 add support for minio backend file carving (#2448)
* add support for minio backend file carving
* add changes file
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2021-10-12 15:32:06 -04:00
Martin Angers 9a0871a2f1 Address issues related to Redis Cluster support (#1885)
Closes #1847 .
2021-09-01 16:32:57 -04:00
Benjamin Edwards c18214be37 add logging settings to config api response (#1467)
- add docker-compose file for locally testing aws dependencies
- update firehose & kinesis configs to optionally supply endpoint url override
- serialize `logging` field in appconfig api response
2021-07-30 11:45:49 -04:00
Tomas Touceda a6cff7ea89 Migrate all mysql tests to the new form (#1408)
* Migrate all mysql tests to the new form

* Only dump sql if MYSQL_TEST is on

* Removing parallel until we get rid of this code

* Move TestMain to an actual _test file

* A little experiment with tmpfs to speed up the db

* Let's make sure the dump.sql file is also in ram
2021-07-19 18:20:31 -03:00
Zach Wasserman 8be9068c1c Fix migration for MariaDB compatibility (#1280)
Give unique names to each new foreign key in the migrations.

Fixes #1279
2021-07-01 15:59:31 -07:00
Tomas Touceda 8e2db1fecf For platform on mysql so it runs in Apple M1 macbooks (#1274) 2021-07-01 13:09:51 -03:00
Zach Wasserman 8c97b36764 Set up Cypress testing for Teams/Tiers (#1005)
- Update names/roles of users in `make e2e-setup`.
- Update test SSO user info.
- Add Cypress commands for seeding users/Teams.
- Stub Cypress tests for team/tier matrix.
2021-06-09 11:56:59 -07:00