Fix S3 file carve cleanup hang and rework reconciliation
Relates to #48549 The S3 carve cleanup (server/datastore/s3, run by the cleanups_then_aggregation cron) advanced ListObjectsV2 pagination using the response's ContinuationToken — an echo of the request token — instead of NextContinuationToken. On any bucket with more than one page of objects this looped forever, hanging the entire serial cleanup cron and stalling every cleanup/aggregation job ordered after it. Replace the bucket-listing reconciliation with a direct HeadObject probe per carve, which is exact and independent of listing order or object counts: - Only carves older than 24h with a completed upload are reconciled (mirrors the MySQL carve store's floor; skips in-flight multipart uploads). A carve is expired only on a definitive not-found; transient or other probe errors leave it for a future run, so a carve whose object still exists is never expired. - Probes run with bounded concurrency; expirations are written in one batched, retryable UPDATE (new ExpireCarves datastore method) rather than one per carve. - The number of carves reconciled per run is capped so a large backlog drains across runs without any single run making unbounded S3 requests. Add S3-carve-store-only server settings (the MySQL carve store is unaffected): - s3.carves_cleanup_disabled — skip reconciliation entirely - s3.carves_cleanup_max_per_run — per-run cap (default 1000) - s3.carves_cleanup_concurrency — concurrent probes (default 32) Also log the expired count per run and fix the test bucket cleanup helper to paginate. Adds unit tests (transient-error safety, partial failure, concurrency) and a MySQL integration test for ExpireCarves.
This commit is contained in:
@@ -2975,6 +2975,51 @@ On GCE, GKE, or Cloud Run, ADC typically resolves to the runtime workload identi
|
||||
carves_force_s3_path_style: false
|
||||
```
|
||||
|
||||
### s3_carves_cleanup_disabled
|
||||
|
||||
When `true`, the S3 carve store skips the periodic reconciliation that marks carves whose S3
|
||||
object no longer exists as expired. Set this if you rely solely on the bucket lifecycle policy
|
||||
to remove carve objects and do not need the `expired` flag reconciled. This applies only to the
|
||||
S3 carve store; it has no effect when carves are stored in MySQL.
|
||||
|
||||
- Default value: false
|
||||
- Environment variable: `FLEET_S3_CARVES_CLEANUP_DISABLED`
|
||||
- Config file format:
|
||||
```yaml
|
||||
s3:
|
||||
carves_cleanup_disabled: true
|
||||
```
|
||||
|
||||
### s3_carves_cleanup_max_per_run
|
||||
|
||||
The maximum number of carves the S3 cleanup reconciles per run, which also bounds
|
||||
the number of S3 `HeadObject` requests a single run makes. A larger carve backlog
|
||||
is drained across subsequent runs. Raise this to drain a large backlog faster, at
|
||||
the cost of more work per run; lower it to reduce each run's impact on the shared
|
||||
cleanup schedule.
|
||||
|
||||
- Default value: 1000
|
||||
- Environment variable: `FLEET_S3_CARVES_CLEANUP_MAX_PER_RUN`
|
||||
- Config file format:
|
||||
```yaml
|
||||
s3:
|
||||
carves_cleanup_max_per_run: 1000
|
||||
```
|
||||
|
||||
### s3_carves_cleanup_concurrency
|
||||
|
||||
The number of concurrent S3 `HeadObject` probes the carve cleanup performs. Kept
|
||||
modest by default to stay well under S3's per-prefix request rate; lower it if you
|
||||
observe throttling, or raise it to speed up a backlog's probe phase.
|
||||
|
||||
- Default value: 32
|
||||
- Environment variable: `FLEET_S3_CARVES_CLEANUP_CONCURRENCY`
|
||||
- Config file format:
|
||||
```yaml
|
||||
s3:
|
||||
carves_cleanup_concurrency: 32
|
||||
```
|
||||
|
||||
### s3_carves_region
|
||||
|
||||
> Same region-discovery behavior as [`s3_software_installers_region`](#s3_software_installers_region). Set this explicitly to avoid defaulting to `us-east-1`.
|
||||
|
||||
Reference in New Issue
Block a user