**Related issue:** Resolves#49805, Resolves#48719
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
---
## Context
A customer (~2,500 hosts, v4.89.1) had their DB writer slammed with
`DELETE FROM host_software_installed_paths` statements carrying 30,000+
IDs each. These never completed, required repeated manual intervention,
and the table grew from 14.5M to 14.8M rows in 2 days. This is #49805.
While investigating, Victor linked #48719, a related `software_titles`
INSERT lock convoy issue seen in load tests. Both are in the same
software ingestion code path (`server/datastore/mysql/software.go`), so
this PR fixes both.
## Root cause
### #49805: Unbatched DELETEs on `host_software_installed_paths`
When a host's software changes, Fleet computes a delta and deletes stale
rows from `host_software_installed_paths`. The function
`deleteHostSoftwareInstalledPaths()` issued a **single** `DELETE FROM
host_software_installed_paths WHERE id IN (?)` with all IDs expanded by
`sqlx.In()`. With 30,000+ IDs and 14.8M rows in the table, these massive
statements held row locks for minutes, timed out, and never completed.
On the next agent check-in, the same (or larger) DELETE was retried,
creating a feedback loop where the table grew unboundedly.
Notably, the INSERT function for the same table
(`insertHostSoftwareInstalledPaths`) already batched at 500 rows. The
DELETE simply lacked the same treatment.
### #48719: INSERT IGNORE lock convoys on `software_titles` (related)
When a host reports software that Fleet hasn't seen before,
`preInsertSoftwareInventory()` runs `INSERT IGNORE INTO software_titles
(...)` inside a `withRetryTxx` transaction. For homogeneous fleets (many
hosts sharing the same software catalog, typical for imaged corporate
Windows machines), hundreds of concurrent goroutines try to INSERT
IGNORE the same title rows simultaneously.
Even though `INSERT IGNORE` is a no-op when the row already exists,
InnoDB still acquires row/gap locks on the unique index for the duration
of the enclosing transaction. With many goroutines holding or waiting on
the same index locks, the DB enters a "lock convoy" where sessions
serialize on locks they don't actually need. In load tests (40 Fleet
instances, 100K hosts, 141 identical Windows software items), this
produced 690 average active sessions on the writer and 85s fleet-wide
p99.
The existing read-first check
(`getIncomingSoftwareChecksumsToExistingTitles`) prevents the convoy at
steady state. But on cold start (empty `software_titles`, e.g. after
cleanup purges orphaned titles), the check finds nothing and all
goroutines race to INSERT the same titles.
## How I reproduced it
Started MySQL via `docker compose up -d mysql_test`, created a git
worktree.
### #49805
`TestHostSoftwareInstalledPathsDeleteExplosion`: Created a host with 500
software items and installed paths, then replaced all software with an
entirely new set. This triggers `deleteHostSoftwareInstalledPaths()`
with all 500 old IDs in a single unbatched DELETE statement. At 500 IDs
the local test completes quickly, but the structure confirms the
problem: at 30K+ IDs on production Aurora with 14M rows, these never
finish.
### #48719
`TestSoftwareTitlesInsertIgnoreLockConvoy`: Created 50 hosts, each
reporting 100 identical software items (simulating a homogeneous fleet).
Used a barrier to release all 50 goroutines simultaneously, then
measured two phases:
1. **Cold start** (empty `software_titles`): All 50 hosts concurrently
call `ds.UpdateHostSoftware()`.
2. **Steady state** (titles exist): Same 50 hosts re-ingest.
**Before fix:**
| Metric | Cold start | Steady state |
|--------|-----------|-------------|
| Wall time | 3.0s | 38ms |
| Avg per-host | 1,981ms | 29ms |
| **Convoy factor** | **79x** | |
The 79x slowdown confirms the lock convoy.
## How I fixed it
### #49805: Batch the DELETE at 500
Changed `deleteHostSoftwareInstalledPaths()` from a single `DELETE ...
WHERE id IN (all IDs)` to a loop that processes 500 IDs per batch,
matching the existing INSERT batching pattern in the same file.
### #48719: Three-layer defense against lock convoys
**Layer 1 - Move title INSERT IGNORE outside the transaction.**
Previously, `INSERT IGNORE INTO software_titles` ran inside
`withRetryTxx`, so locks were held for the full transaction duration.
Now each title INSERT is executed via `ds.writer(ctx).ExecContext()`
outside any transaction, auto-committing independently and holding locks
for microseconds.
**Layer 2 - singleflight per title key.** Added a `singleflight.Group`
on the `Datastore` struct. For each title, only one goroutine actually
executes the INSERT; concurrent goroutines wait and share the result.
**Layer 3 - In-process cache (`sync.Map`).** After a title is inserted,
its key is stored in `knownSoftwareTitleKeys`. Subsequent ingestions
check the cache first and skip the INSERT entirely.
`CleanupSoftwareTitles` clears the cache when it deletes orphaned
titles.
The three layers work together: the cache handles the common case (title
already known), singleflight handles the cold-start race (only one
INSERT per title), and auto-commit ensures even the winning INSERT holds
locks for microseconds.
## How I tested that it works
### New reproduction tests
- `TestSoftwareTitlesInsertIgnoreLockConvoy`: 50 concurrent hosts, 100
identical software items. Measures cold-start convoy factor and verifies
all 100 titles are created.
- `TestHostSoftwareInstalledPathsDeleteExplosion`: Full software
replacement path with 500 items per host, including concurrent hosts.
### Existing test suite
Ran all existing software tests including:
- `UpdateHostSoftware`, `UpdateHostSoftwareDeadlock`,
`PreInsertSoftwareInventory`
- `SoftwareTitleUpgradeCodeDriftMatch`,
`UpdateHostSoftwareSameBundleIDDifferentNames`
- `CleanupSoftwareTitles` (validates cache invalidation works correctly)
- `SaveHost`, `SyncHostsSoftware`, and ~80 other subtests
All pass.
### After-fix measurements
| Metric | Before fix | After fix |
|--------|-----------|-----------|
| Cold-start wall (50 hosts) | ~3.0s | ~1.4s |
| Cold-start avg per-host | ~1,981ms | ~594ms |
| Steady-state wall | ~38ms | ~7ms |
| Titles created correctly | 100/100 | 100/100 |
The remaining cold-start time is from other pipeline operations (`INSERT
IGNORE INTO software`, host_software linking), not from
`software_titles`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance Improvements**
- Improved software inventory ingestion under large, concurrent
workloads, including more efficient handling of repeated software-title
inserts.
- Reduced lock contention when many devices report the same titles at
the same time.
- Batched deletions of installed software-path records to speed up large
updates.
- **Bug Fixes**
- Ensured deterministic, collation-safe software-title deduplication to
prevent incorrect or stale title mapping.
- Strengthened orphan cleanup behavior so caches are cleared when orphan
titles are removed.
- **Tests**
- Added stress/regression tests for software-title insert contention and
large installed-path delete workloads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
