Files
fleet/changes
Nico a335b3e6d4 Fix VPP API retry recursion causing server OOM (#46659)
<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #46656

`server/mdm/apple/vpp.do` retried transient Apple errors by **calling
itself recursively**, with the rate-limit branch nesting `retry.Do`
inside `retry.Do`.

This change replaces the recursion with a single retry loop (respecting
the prior 1 initial attempt + 3 retries), closes each response before
retrying, honors Apple's `Retry-After` capped at 30s so that a
multi-minute value can't block a synchronous request, and threads
`context` through the VPP calls so the backoff is cancellable. The retry
timings are otherwise unchanged from before.

Following @sgress454 suggestion, I considered routing this through the
shared `retry.Do` helper (a single attempt wrapped in `retry.Do` + an
error filter) but figured out that:
- retry.Do` owns its own wait schedule and its error filter returns an
outcome enum rather than a duration, so it can't honor Apple's
per-response `Retry-After` value.
- also, I'd have to change the `retry` package to receive an extra `ctx`
param so that the backoff is context-aware (which IMHO is more blast
radius than this incident fix should carry).

# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/`.
- [x] Timeouts are implemented and retries are limited to avoid infinite
loops

## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

**What was verified.** The new automated test cannot run against `main`
(the fix changes the VPP function signatures and adds the retry knobs),
so to confirm the actual failure mode I checked out `main` and ran a
small repro that drives the VPP client against an Apple endpoint that
always returns the rate-limit error. On `main`, the call **never
returns** — `do()` recurses without bound — and the repro times out:

```
--- FAIL: TestReproUnboundedRecursionOnMain (10.00s)
    zz_repro_main_test.go:30: AssociateAssets did NOT return within 10s — unbounded retry recursion in do() on main
FAIL
FAIL	github.com/fleetdm/fleet/v4/server/mdm/apple/vpp	10.642s
```

On this branch the same scenario returns a bounded error promptly. That
behavior is covered by the new `TestDoRetryIsBoundedAndNonRecursive`
(bounded rate-limit retries, `Retry-After` honored-but-capped, and
context cancellation), and the full `server/mdm/apple/vpp` package
passes.
**I did not perform an end-to-end QA against a live Apple endpoint**.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Fixed a server out-of-memory crash that occurred when Apple VPP API
repeatedly returned transient errors during VPP operations, including
app installs, user registration, and license seat releases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-02 16:18:02 -03:00
..