Sharon KatzandClaude Opus 4.6
6ba04b0d20
**Related issue:** Resolves #48697 ## Summary The hourly `UpdateQueryAggregatedStats` cron job currently walks **every query ID** in the `queries` table and runs 5 expensive percentile-calculation queries per query against `scheduled_query_stats`, plus 1 INSERT/UPDATE to store results. Most queries have no execution data at all (they are saved queries, live-only, or de-scheduled), so this work is pure waste. This PR changes the cron to only process queries that actually have execution data, by querying `scheduled_query_stats` directly instead of the `queries` table. The now-unused `walkIdsInTable` helper function is also removed. ### How the calculations work `CalculateAggregatedPerfStatsPercentiles` computes performance statistics for each query that has been scheduled and executed by hosts. For each qualifying query ID, it runs these operations against the read replica: 1. **P50 user_time** -- Calculates the median (50th percentile) of per-host average user-mode CPU time. The query groups `scheduled_query_stats` rows by `host_id`, computes `SUM(user_time) / SUM(executions)` per host, sorts them, then picks the row at position `FLOOR(total_rows * 0.5) + 1` using a `@rownum` session variable. 2. **P95 user_time** -- Same calculation but picks the 95th percentile row (`FLOOR(total_rows * 0.95) + 1`). 3. **P50 system_time** -- Same percentile calculation for kernel/system CPU time. 4. **P95 system_time** -- 95th percentile of system CPU time. 5. **Total executions** -- `SELECT COALESCE(SUM(executions), 0) FROM scheduled_query_stats WHERE scheduled_query_id = ?` 6. **INSERT/UPDATE** -- Writes the JSON result (`user_time_p50`, `user_time_p95`, `system_time_p50`, `system_time_p95`, `total_executions`) into the `aggregated_stats` table via `INSERT ... ON DUPLICATE KEY UPDATE`. ### What changed **Before:** `SELECT id FROM queries` -- walks every query (200-400+ in a typical deployment). **After:** `SELECT DISTINCT scheduled_query_id FROM scheduled_query_stats WHERE executions > 0` -- walks only queries that have actual execution data (typically 10-20). ### Benchmark results (MySQL 8.0, 300 queries seeded, only 15 with stats) | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | Avg time per cron run | 3.36s | 0.28s | **12.2x faster** | | DB operations per run | 1,800 | 90 | **95% fewer** | | DB operations per day | 43,200 | 2,160 | **41,040 eliminated** | | `aggregated_stats` rows written | 300 (285 empty) | 15 (all meaningful) | Less table bloat | | Correctness | baseline | byte-identical JSON | **Zero regression** | At 500+ queries the current approach **drops MySQL connections** (`unexpected EOF` / `invalid connection`) because the cursor is held open across thousands of heavy serial queries. The optimized version handles any scale trivially. ### Impact analysis Verified safe across all consumers: all query endpoints use `LEFT JOIN aggregated_stats` (NULL-safe for missing rows), the frontend explicitly handles null stats as "Undetermined", live query stats (`service_campaigns.go`) call `CalculateAggregatedPerfStatsPercentiles` directly and are unaffected, and query deletion already cleans up both `scheduled_query_stats` and `aggregated_stats` rows. # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/` - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements) - [x] Added/updated automated tests - [x] Confirmed that the fix is not expected to adversely impact load test results Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
News · Report a bug · Docs · Why open source? · Art
Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.
What's it for?
Fleet gives you a single system to secure and maintain all your computing devices over the air. You can do MDM, patch stuff, deploy software, and verify anything, all from one place, across every OS your organization uses.
Fleet works directly with data and events from the native operating system, down to the bare metal. Strong diagnostics let you investigate errors on end-user devices and collect accurate audit evidence in minutes.
Get started
You can try Fleet out for yourself, or grab time with one of the maintainers to chat.
Is it any good?
Fleet is used in production by IT and security teams managing thousands of devices. Many deployments support tens of thousands of hosts, and a few large organizations manage 400,000 or more.
Supported platforms
- Linux (all distros)
- macOS
- Windows
- Chromebooks
- iOS and Android (BYOD or corporate-owned)
- Amazon Web Services (AWS)
- Google Cloud (GCP)
- Azure (Microsoft cloud)
- Data centers
- Containers (kube, etc)
- Linux-based IoT devices
Infrastructure as code
Manage your fleet with GitOps, or use the GUI, REST API, webhook events, and the fleetctl command-line tool.
Linux support
First-class support for all major distros. Linux gets the same attention and visibility as macOS and Windows.
Visibility and compliance
Fleet can report on hundreds of attributes across your devices and ships with CIS benchmarks for macOS and Windows and comprehensive operating system, hardware, and software data. Check out the table reference documentation to see what's available.
Open by design
Fleet is open source and transparent about what it can and can't see. End users can verify exactly how the agent works and what data their company collects. Fleet collects only the data needed to manage and secure devices, not private activity like keystrokes, emails, or webcams.
Good neighbors
Ready-to-use, enterprise-friendly integrations exist for Snowflake, Splunk, GitHub Actions, Vanta, Elastic Jira, Zendesk, and more. Fleet also works with tools such as Munki, Chef, Puppet, Ansible, CrowdStrike, and SentinelOne.
Lighter than air
Fleet is lightweight and modular. You can use it for MDM without using it for security, and vice versa. You can turn off features you are not using.
Free as in free
The free version of Fleet will always be free. Fleet is independently backed and actively maintained with the help of many amazing contributors.
Longevity
The company behind Fleet is founded (and majority-owned) by true believers in open source. The company's business model is influenced by GitLab (NYSE: GTLB), with great investors, happy customers, and the capacity to become profitable at any time.
Fleet Device Management's company handbook is public and open source. You can read about the history of Fleet and our commitment to improving the product.
Chat
The Fleet community is full of kind and helpful people. Whether or not you are a paying customer, if you need help, just reach out.
Contributing 
Contributions are welcome, whether you answer questions on Slack / GitHub / LinkedIn, improve the documentation or website, write a tutorial, give a talk at a conference or local meetup, give an interview on a podcast, troubleshoot reported issues, or submit a patch. The Fleet code of conduct is on GitHub.
License
The free version of Fleet is available under the MIT license. The commercial license is also designed to allow contributions to paid features for users whose employment agreements allow them to contribute to open source projects. (See LICENSE.md for details.)
Fleet is built on osquery, nanoMDM, Nudge, and swiftDialog.