Sharon Katzandcopilot-swe-agent[bot]
5c127e5fe4
**Related issue:** Resolves #49805, Resolves #48719 # Checklist for submitter If some of the following don't apply, delete the relevant line. - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [x] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually --- ## Context A customer (~2,500 hosts, v4.89.1) had their DB writer slammed with `DELETE FROM host_software_installed_paths` statements carrying 30,000+ IDs each. These never completed, required repeated manual intervention, and the table grew from 14.5M to 14.8M rows in 2 days. This is #49805. While investigating, Victor linked #48719, a related `software_titles` INSERT lock convoy issue seen in load tests. Both are in the same software ingestion code path (`server/datastore/mysql/software.go`), so this PR fixes both. ## Root cause ### #49805: Unbatched DELETEs on `host_software_installed_paths` When a host's software changes, Fleet computes a delta and deletes stale rows from `host_software_installed_paths`. The function `deleteHostSoftwareInstalledPaths()` issued a **single** `DELETE FROM host_software_installed_paths WHERE id IN (?)` with all IDs expanded by `sqlx.In()`. With 30,000+ IDs and 14.8M rows in the table, these massive statements held row locks for minutes, timed out, and never completed. On the next agent check-in, the same (or larger) DELETE was retried, creating a feedback loop where the table grew unboundedly. Notably, the INSERT function for the same table (`insertHostSoftwareInstalledPaths`) already batched at 500 rows. The DELETE simply lacked the same treatment. ### #48719: INSERT IGNORE lock convoys on `software_titles` (related) When a host reports software that Fleet hasn't seen before, `preInsertSoftwareInventory()` runs `INSERT IGNORE INTO software_titles (...)` inside a `withRetryTxx` transaction. For homogeneous fleets (many hosts sharing the same software catalog, typical for imaged corporate Windows machines), hundreds of concurrent goroutines try to INSERT IGNORE the same title rows simultaneously. Even though `INSERT IGNORE` is a no-op when the row already exists, InnoDB still acquires row/gap locks on the unique index for the duration of the enclosing transaction. With many goroutines holding or waiting on the same index locks, the DB enters a "lock convoy" where sessions serialize on locks they don't actually need. In load tests (40 Fleet instances, 100K hosts, 141 identical Windows software items), this produced 690 average active sessions on the writer and 85s fleet-wide p99. The existing read-first check (`getIncomingSoftwareChecksumsToExistingTitles`) prevents the convoy at steady state. But on cold start (empty `software_titles`, e.g. after cleanup purges orphaned titles), the check finds nothing and all goroutines race to INSERT the same titles. ## How I reproduced it Started MySQL via `docker compose up -d mysql_test`, created a git worktree. ### #49805 `TestHostSoftwareInstalledPathsDeleteExplosion`: Created a host with 500 software items and installed paths, then replaced all software with an entirely new set. This triggers `deleteHostSoftwareInstalledPaths()` with all 500 old IDs in a single unbatched DELETE statement. At 500 IDs the local test completes quickly, but the structure confirms the problem: at 30K+ IDs on production Aurora with 14M rows, these never finish. ### #48719 `TestSoftwareTitlesInsertIgnoreLockConvoy`: Created 50 hosts, each reporting 100 identical software items (simulating a homogeneous fleet). Used a barrier to release all 50 goroutines simultaneously, then measured two phases: 1. **Cold start** (empty `software_titles`): All 50 hosts concurrently call `ds.UpdateHostSoftware()`. 2. **Steady state** (titles exist): Same 50 hosts re-ingest. **Before fix:** | Metric | Cold start | Steady state | |--------|-----------|-------------| | Wall time | 3.0s | 38ms | | Avg per-host | 1,981ms | 29ms | | **Convoy factor** | **79x** | | The 79x slowdown confirms the lock convoy. ## How I fixed it ### #49805: Batch the DELETE at 500 Changed `deleteHostSoftwareInstalledPaths()` from a single `DELETE ... WHERE id IN (all IDs)` to a loop that processes 500 IDs per batch, matching the existing INSERT batching pattern in the same file. ### #48719: Three-layer defense against lock convoys **Layer 1 - Move title INSERT IGNORE outside the transaction.** Previously, `INSERT IGNORE INTO software_titles` ran inside `withRetryTxx`, so locks were held for the full transaction duration. Now each title INSERT is executed via `ds.writer(ctx).ExecContext()` outside any transaction, auto-committing independently and holding locks for microseconds. **Layer 2 - singleflight per title key.** Added a `singleflight.Group` on the `Datastore` struct. For each title, only one goroutine actually executes the INSERT; concurrent goroutines wait and share the result. **Layer 3 - In-process cache (`sync.Map`).** After a title is inserted, its key is stored in `knownSoftwareTitleKeys`. Subsequent ingestions check the cache first and skip the INSERT entirely. `CleanupSoftwareTitles` clears the cache when it deletes orphaned titles. The three layers work together: the cache handles the common case (title already known), singleflight handles the cold-start race (only one INSERT per title), and auto-commit ensures even the winning INSERT holds locks for microseconds. ## How I tested that it works ### New reproduction tests - `TestSoftwareTitlesInsertIgnoreLockConvoy`: 50 concurrent hosts, 100 identical software items. Measures cold-start convoy factor and verifies all 100 titles are created. - `TestHostSoftwareInstalledPathsDeleteExplosion`: Full software replacement path with 500 items per host, including concurrent hosts. ### Existing test suite Ran all existing software tests including: - `UpdateHostSoftware`, `UpdateHostSoftwareDeadlock`, `PreInsertSoftwareInventory` - `SoftwareTitleUpgradeCodeDriftMatch`, `UpdateHostSoftwareSameBundleIDDifferentNames` - `CleanupSoftwareTitles` (validates cache invalidation works correctly) - `SaveHost`, `SyncHostsSoftware`, and ~80 other subtests All pass. ### After-fix measurements | Metric | Before fix | After fix | |--------|-----------|-----------| | Cold-start wall (50 hosts) | ~3.0s | ~1.4s | | Cold-start avg per-host | ~1,981ms | ~594ms | | Steady-state wall | ~38ms | ~7ms | | Titles created correctly | 100/100 | 100/100 | The remaining cold-start time is from other pipeline operations (`INSERT IGNORE INTO software`, host_software linking), not from `software_titles`. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Performance Improvements** - Improved software inventory ingestion under large, concurrent workloads, including more efficient handling of repeated software-title inserts. - Reduced lock contention when many devices report the same titles at the same time. - Batched deletions of installed software-path records to speed up large updates. - **Bug Fixes** - Ensured deterministic, collation-safe software-title deduplication to prevent incorrect or stale title mapping. - Strengthened orphan cleanup behavior so caches are cleared when orphan titles are removed. - **Tests** - Added stress/regression tests for software-title insert contention and large installed-path delete workloads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2 lines
308 B
Plaintext
2 lines
308 B
Plaintext
- Improved software ingestion performance at scale: batched `host_software_installed_paths` deletes (previously unbounded single statements) and eliminated `software_titles` INSERT lock convoys during concurrent ingestion by moving title inserts outside the main transaction with singleflight deduplication.
|