Files

T

Andrey Kizimenko 6430e6132f Add 4.89.0 baseline and migration loadtest results (#49168 )

## Summary

Adds the **4.89.0** load test metrics artifacts, following the existing
`tools/loadtest/metrics/runs/` conventions.

### Baseline — `runs/baseline/489loadtest/`
18h run on a fresh RC instance (no data).
- `489loadtest-2026-07-10-210313Z-18h.json` / `.md`

### Migration — `runs/migration/487to489mig/`
4.87 → 4.89 migration, before vs. after.
- `487to489mig-2026-07-10-162812Z-1h.{json,md}` — pre-migration (1h
window)
- `487to489mig-2026-07-10-192738Z-130m.{json,md}` — post-migration (130m
window)

## Results

| Run | Fleet errors | ALB 5xx | Abnormal stops | Threshold checks |
|-----|:---:|:---:|:---:|---|
| Baseline (18h) | 0 | 0 | 0 | RDS Writer Deadlocks avg 0.13
(occasional, retried) |
| Migration pre (1h) | 0 | 0 | 0 | ✅ all within thresholds |
| Migration post (130m) | 0 | 0 | 0 | RDS Writer Deadlocks avg 0.08
(occasional, retried) |

The only flag is a near-zero average of occasional RDS writer deadlocks
(MySQL retries these); everything else is within expected range and
holds steady across the migration. Full per-metric comparison
(`compare-metrics.sh`) is posted on the release QA issue.

Data only — no code changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
  * Added baseline and migration load-test performance reports.
* Included infrastructure, database, Redis, load balancer, network,
container health, error, and SQL performance metrics.
* Added reports covering multiple test durations and migration intervals
for improved performance analysis.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-07-10 17:00:36 -05:00

runs

Add 4.89.0 baseline and migration loadtest results (#49168 )

2026-07-10 17:00:36 -05:00

collect-metrics.sh

…

compare-metrics.sh

…

README.md

…

README.md

Load test metrics

Collect and compare AWS CloudWatch metrics for Fleet load test environments. Use these scripts to capture a point-in-time synopsis of a running load test and to diff runs against each other to catch regressions release-over-release.

collect-metrics.sh — discovers a load test environment's AWS resources from its Terraform workspace name, pulls CloudWatch metrics averaged over a lookback interval, and writes a .json data file plus a human-readable .md synopsis (with threshold alerts).
compare-metrics.sh — diffs two or more runs side by side and flags deltas as ok / WARN / ALERT.

Requirements

AWS CLI v2, authenticated against the account hosting the load test environment.
jq

Collecting metrics

# 3h lookback (default) for the "486loadtest" workspace
./collect-metrics.sh --workspace 486loadtest

# 1h lookback, filed under the mdm category
./collect-metrics.sh --workspace 483applemdm --interval 1h --category mdm

Key flags (--help for the full list):

Flag	Meaning
`-w, --workspace`	Terraform workspace name (required). AWS resource names are derived from it.
`-i, --interval`	Lookback window: `<N>h`, `<N>m`, or a bare integer (hours). Default `3h`.
`-c, --category`	File the run under a category: `baseline` \| `migration` \| `mdm`.
`-o, --output`	Override the output file path.
`-r, --region`	AWS region. Default `us-east-2`.

Output lands in runs/[<category>/]<workspace>/<workspace>-<timestamp>-<interval>.json alongside a matching .md synopsis.

The --workspace value is not free-form — collect-metrics.sh derives AWS resource names from it (fleet-<ws>-backend, fleetdm-<ws>-mysql, fleet-<ws>-redis, …), so it must match the actual Terraform workspace.

Comparing runs

# Compare the 2 most recent runs across all categories
./compare-metrics.sh

# Last 4 baseline releases, one run per workspace
./compare-metrics.sh --filter loadtest --depth 4 --unique

# Two specific files
./compare-metrics.sh runs/baseline/485loadtest/485*.json runs/baseline/486loadtest/486*.json

compare-metrics.sh searches runs/ recursively, so category subfolders are included automatically. The --filter flag matches on the workspace name, which — thanks to the naming conventions below — doubles as a category selector (--filter loadtest, --filter mig).

Run organization

Historical runs live under runs/, grouped by what the load test exercised:

Category	`runs/` subfolder	Workspace naming convention	Examples
Baseline — per-release branch load test	`runs/baseline/`	`<version>loadtest`	`486loadtest`
Migration — n-1 → n schema migration	`runs/migration/`	`<n-1>to<n>mig`	`485to486mig`
MDM — platform-specific MDM load test	`runs/mdm/`	`<version><platform>` / `<platform>-release`	`483applemdm`, `486-windows`

The category subfolder is purely for human organization; the scripts don't depend on it. Keeping workspace names to these conventions is what makes --filter a reliable category selector.

Submitting results

After a load test, commit the run so the history stays useful for future comparisons:

Collect with the right category so the files land in the correct folder, e.g. ./collect-metrics.sh --workspace 486loadtest --category baseline.
Commit both the .json (data) and .md (synopsis) for the run under runs/<category>/<workspace>/. For multi-step tests (e.g. MDM), a short REPORT.md summarizing the runs is welcome too.
Open a PR against main with the new files. Keep it to the run artifacts — no script changes.