Fix S3 file carve cleanup hang and rework reconciliation

Relates to #48549

The S3 carve cleanup (server/datastore/s3, run by the
cleanups_then_aggregation cron) advanced ListObjectsV2 pagination using
the response's ContinuationToken — an echo of the request token —
instead of NextContinuationToken. On any bucket with more than one page
of objects this looped forever, hanging the entire serial cleanup cron
and stalling every cleanup/aggregation job ordered after it.

Replace the bucket-listing reconciliation with a direct HeadObject probe
per carve, which is exact and independent of listing order or object
counts:

- Only carves older than 24h with a completed upload are reconciled
(mirrors the MySQL carve store's floor; skips in-flight multipart
uploads). A carve is expired only on a definitive not-found; transient
or other probe errors leave it for a future run, so a carve whose object
still exists is never expired.
- Probes run with bounded concurrency; expirations are written in one
batched, retryable UPDATE (new ExpireCarves datastore method) rather
than one per carve.
- The number of carves reconciled per run is capped so a large backlog
drains across runs without any single run making unbounded S3 requests.

Add S3-carve-store-only server settings (the MySQL carve store is
unaffected):
- s3.carves_cleanup_disabled       — skip reconciliation entirely
- s3.carves_cleanup_max_per_run    — per-run cap (default 1000)
- s3.carves_cleanup_concurrency    — concurrent probes (default 32)

Also log the expired count per run and fix the test bucket cleanup
helper to paginate. Adds unit tests (transient-error safety, partial
failure, concurrency) and a MySQL integration test for ExpireCarves.
This commit is contained in:
Juan Fernandez
2026-07-01 14:00:59 -04:00
committed by GitHub
parent a88354c5e3
commit e8f26ec4ef
12 changed files with 646 additions and 118 deletions
@@ -2975,6 +2975,51 @@ On GCE, GKE, or Cloud Run, ADC typically resolves to the runtime workload identi
carves_force_s3_path_style: false
```
### s3_carves_cleanup_disabled
When `true`, the S3 carve store skips the periodic reconciliation that marks carves whose S3
object no longer exists as expired. Set this if you rely solely on the bucket lifecycle policy
to remove carve objects and do not need the `expired` flag reconciled. This applies only to the
S3 carve store; it has no effect when carves are stored in MySQL.
- Default value: false
- Environment variable: `FLEET_S3_CARVES_CLEANUP_DISABLED`
- Config file format:
```yaml
s3:
carves_cleanup_disabled: true
```
### s3_carves_cleanup_max_per_run
The maximum number of carves the S3 cleanup reconciles per run, which also bounds
the number of S3 `HeadObject` requests a single run makes. A larger carve backlog
is drained across subsequent runs. Raise this to drain a large backlog faster, at
the cost of more work per run; lower it to reduce each run's impact on the shared
cleanup schedule.
- Default value: 1000
- Environment variable: `FLEET_S3_CARVES_CLEANUP_MAX_PER_RUN`
- Config file format:
```yaml
s3:
carves_cleanup_max_per_run: 1000
```
### s3_carves_cleanup_concurrency
The number of concurrent S3 `HeadObject` probes the carve cleanup performs. Kept
modest by default to stay well under S3's per-prefix request rate; lower it if you
observe throttling, or raise it to speed up a backlog's probe phase.
- Default value: 32
- Environment variable: `FLEET_S3_CARVES_CLEANUP_CONCURRENCY`
- Config file format:
```yaml
s3:
carves_cleanup_concurrency: 32
```
### s3_carves_region
> Same region-discovery behavior as [`s3_software_installers_region`](#s3_software_installers_region). Set this explicitly to avoid defaulting to `us-east-1`.