Relates to #48549
The S3 carve cleanup (server/datastore/s3, run by the
cleanups_then_aggregation cron) advanced ListObjectsV2 pagination using
the response's ContinuationToken — an echo of the request token —
instead of NextContinuationToken. On any bucket with more than one page
of objects this looped forever, hanging the entire serial cleanup cron
and stalling every cleanup/aggregation job ordered after it.
Replace the bucket-listing reconciliation with a direct HeadObject probe
per carve, which is exact and independent of listing order or object
counts:
- Only carves older than 24h with a completed upload are reconciled
(mirrors the MySQL carve store's floor; skips in-flight multipart
uploads). A carve is expired only on a definitive not-found; transient
or other probe errors leave it for a future run, so a carve whose object
still exists is never expired.
- Probes run with bounded concurrency; expirations are written in one
batched, retryable UPDATE (new ExpireCarves datastore method) rather
than one per carve.
- The number of carves reconciled per run is capped so a large backlog
drains across runs without any single run making unbounded S3 requests.
Add S3-carve-store-only server settings (the MySQL carve store is
unaffected):
- s3.carves_cleanup_disabled — skip reconciliation entirely
- s3.carves_cleanup_max_per_run — per-run cap (default 1000)
- s3.carves_cleanup_concurrency — concurrent probes (default 32)
Also log the expired count per run and fix the test bucket cleanup
helper to paginate. Adds unit tests (transient-error safety, partial
failure, concurrency) and a MySQL integration test for ExpireCarves.
Juan Fernandez
e8f26ec4ef