[orbit/packaging] delay restart for in-band pkg upgrade on Linux (#31286)

Admins _should_ be upgrading orbit/osqueryd/fleet-desktop using TUF, but
there's no official path for pushing updates to the orbit environment
file (`/etc/default/orbit`).

Knowing that this file is installed by fleet-osquery, I naively pushed a
new fleet-osquery package to a user's machine, thinking that would be
fine installing over the existing package. Instead it actually broke
orbit entirely on the host, rendering it unreachable until the user
manually reinstalled the package. This is because the pre-removal script
unconditionally stops orbit even if it's being upgraded, and when orbit
is stopped it terminates any processes underneath it, including the
ongoing package installation.

To workaround this, we replace the simple
`systemctl restart orbit.service` with a check for the `INSTALLER_PATH`
environment variable that orbit sets during software installations. If
the variable is present, `systemd-run` is used to schedule the service
restart 60 seconds in the future, which is assumed to be more than
enough time for the package manager to finish and exit. Unfortunately,
this bugfix cannot be made retroactive, because the broken version of
the prerm script is called before the new package starts to be
unpacked/installed.

Although there are other ways of doing anything an administrator might
be trying to accomplish by pushing a new fleet-osquery package, bricking
an endpoint simply by pushing this package through the Software page
seems like a pretty massive footgun that is easy to protect against.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for safe in-band upgrades of DEB and RPM packages
generated by the `fleetctl package` command via the Software page, after
an initial manual update.

* **Documentation**
* Added a note clarifying the new upgrade process and the need for a
one-time manual update before using in-band upgrades.

* **Bug Fixes**
* Improved upgrade scripts to prevent the Orbit service from stopping
unexpectedly during package upgrades, ensuring smoother and safer
updates.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dan Fuhry <dan@fuhry.com>

This commit is contained in:

Dan Fuhry

2025-07-28 15:55:44 +02:00

committed by

GitHub

parent b5b6d3c51e

commit 6e227b6eb5

2 changed files with 50 additions and 5 deletions

orbit/changes/31286-package-upgrade-fix

View File

@@ -0,0 +1,3 @@
 - DEB and RPM packages generated by `fleetctl package` will now be safe to upgrade in-band through the Software page.
   - Note that the package will need to be updated out-of-band once, because the pre-removal script from previously-generated packages is called upon an upgrade. The old pre-removal script stopped Orbit unconditionally.
   - In other words, fleet-osquery can safely be updated through the Software page only _after_ a new package generated with this version of fleetctl has been installed through other means.