Lucas Manuel Rodriguez
bec3b0dc2a
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #47722 The issue was from a customer running `GET /api/v1/fleet/hosts?device_mapping=true&page=1&per_page=100&query=<ADDRESS>%40example.com` on a script in a for loop. This change reduces the impact of the API on such workflows. Results from my local load test: EXPLAIN ANALYZE: ``` ┌───────────────────────────────────┬────────────────┬─────────────┬─────────────────────────────────────────────┐ │ │ optimizer cost │ actual time │ device_mapping aggregation │ ├───────────────────────────────────┼────────────────┼─────────────┼─────────────────────────────────────────────┤ │ Old (derived-table GROUP BY join) │ ~23,179 │ ~73 ms │ materialized dm derived table, cost ~7,125 │ ├───────────────────────────────────┼────────────────┼─────────────┼─────────────────────────────────────────────┤ │ New (correlated subquery) │ ~1,260 │ ~25 ms │ Aggregate … loops=1 (only the returned row) │ └───────────────────────────────────┴────────────────┴─────────────┴─────────────────────────────────────────────┘ ``` Tests with 10k hosts: ``` ┌───────────────────────────────────┬────────────┬───────────────┬───────┐ │ dataset │ OLD (main) │ NEW (this PR) │ ratio │ ├───────────────────────────────────┼────────────┼───────────────┼───────┤ │ 10k hosts × 3 emails (30k rows) │ 4.6s │ 1.1s │ ~4× │ ├───────────────────────────────────┼────────────┼───────────────┼───────┤ │ 10k hosts × 30 emails (300k rows) │ 35.9s │ 1.2s │ ~30× │ └───────────────────────────────────┴────────────┴───────────────┴───────┘ ``` # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually ## What & why `GET /api/v1/fleet/hosts?device_mapping=true&page=1&per_page=100&query=<email>` caused high MySQL **reader** load on instances with ~10k hosts. Each page load ran an expensive aggregation over the entire `host_emails` table even though only ~100 rows are returned. **Root cause:** with `device_mapping=true`, `applyHostFilters` added a `LEFT JOIN` on a derived table with `GROUP BY host_id` over `host_emails`. Because of the `GROUP BY`, MySQL must fully materialize that derived table (aggregating every row for all hosts) before the outer `WHERE`/`LIMIT 100` can be applied, so the full cost is paid on every page request regardless of result size. `CountHosts` reused the same options, materializing the aggregation a **second** time per page load. **Fixes (both in `server/datastore/mysql/hosts.go`):** 1. Replaced the derived-table join with a correlated subquery in the `SELECT` list (only when `opt.DeviceMapping`), so it is evaluated only for the rows actually returned, each as an indexed lookup on `idx_host_emails_host_id_email`. This matches the existing `host_additional` pattern in the same query. 2. Set `opt.DeviceMapping = false` in `CountHosts` — the column is never selected for counting — mirroring the existing `opt.DisableIssues` handling. ## Notes - The composite index `idx_host_emails_host_id_email (host_id, email)` already exists, so the correlated subquery resolves via an indexed lookup per returned row. - `TestHosts` (full suite) passes, including `HostDeviceMapping`, `CustomHostDeviceMapping`, and `IDPHostDeviceMapping` (the last two verify the `custom_*` → `custom` and `idp` → `mdm_idp_accounts` source translation still works through the new subquery). - Recommend validating with `EXPLAIN ANALYZE` on a ~10k-host dataset before/after, per the issue. I did not have access to such a dataset. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance** * Improved host list responsiveness when using search filters alongside device mapping. * Reduced database load during host listing by retrieving device mapping more efficiently per host. * Improved host counting speed by avoiding device-mapping evaluation for count queries. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
News · Report a bug · Docs · Why open source? · Art
Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.
What's it for?
Fleet gives you a single system to secure and maintain all your computing devices over the air. You can do MDM, patch stuff, deploy software, and verify anything, all from one place, across every OS your organization uses.
Fleet works directly with data and events from the native operating system, down to the bare metal. Strong diagnostics let you investigate errors on end-user devices and collect accurate audit evidence in minutes.
Get started
You can try Fleet out for yourself, or grab time with one of the maintainers to chat.
Is it any good?
Fleet is used in production by IT and security teams managing thousands of devices. Many deployments support tens of thousands of hosts, and a few large organizations manage 400,000 or more.
Supported platforms
- Linux (all distros)
- macOS
- Windows
- Chromebooks
- iOS and Android (BYOD or corporate-owned)
- Amazon Web Services (AWS)
- Google Cloud (GCP)
- Azure (Microsoft cloud)
- Data centers
- Containers (kube, etc)
- Linux-based IoT devices
Infrastructure as code
Manage your fleet with GitOps, or use the GUI, REST API, webhook events, and the fleetctl command-line tool.
Linux support
First-class support for all major distros. Linux gets the same attention and visibility as macOS and Windows.
Visibility and compliance
Fleet can report on hundreds of attributes across your devices and ships with CIS benchmarks for macOS and Windows and comprehensive operating system, hardware, and software data. Check out the table reference documentation to see what's available.
Open by design
Fleet is open source and transparent about what it can and can't see. End users can verify exactly how the agent works and what data their company collects. Fleet collects only the data needed to manage and secure devices, not private activity like keystrokes, emails, or webcams.
Good neighbors
Ready-to-use, enterprise-friendly integrations exist for Snowflake, Splunk, GitHub Actions, Vanta, Elastic Jira, Zendesk, and more. Fleet also works with tools such as Munki, Chef, Puppet, Ansible, CrowdStrike, and SentinelOne.
Lighter than air
Fleet is lightweight and modular. You can use it for MDM without using it for security, and vice versa. You can turn off features you are not using.
Free as in free
The free version of Fleet will always be free. Fleet is independently backed and actively maintained with the help of many amazing contributors.
Longevity
The company behind Fleet is founded (and majority-owned) by true believers in open source. The company's business model is influenced by GitLab (NYSE: GTLB), with great investors, happy customers, and the capacity to become profitable at any time.
Fleet Device Management's company handbook is public and open source. You can read about the history of Fleet and our commitment to improving the product.
Chat
The Fleet community is full of kind and helpful people. Whether or not you are a paying customer, if you need help, just reach out.
Contributing 
Contributions are welcome, whether you answer questions on Slack / GitHub / LinkedIn, improve the documentation or website, write a tutorial, give a talk at a conference or local meetup, give an interview on a podcast, troubleshoot reported issues, or submit a patch. The Fleet code of conduct is on GitHub.
License
The free version of Fleet is available under the MIT license. The commercial license is also designed to allow contributions to paid features for users whose employment agreements allow them to contribute to open source projects. (See LICENSE.md for details.)
Fleet is built on osquery, nanoMDM, Nudge, and swiftDialog.