aff440236e
**Related issue:** Ref #34797 Ref #42675 ## Problem When a software installer spec has no `hash_sha256`, Fleet re-downloads the package, re-extracts metadata, and re-upserts the DB on every GitOps run, even if the upstream file hasn't changed. For deployments with 50+ URL-only packages across multiple teams, this wastes bandwidth and processing time on every run. ## Solution By default, use etags to avoid unnecessary downloads: 1. First run: Fleet downloads the package normally and stores the server's ETag header 2. Subsequent runs: Fleet sends a conditional GET with `If-None-Match`. If the server returns 304 Not Modified, Fleet skips the download, metadata extraction, S3 upload, and DB upsert entirely Opt-out with `always_download:true`, meaning packages continue to be downloaded and re-processed on every run, same as today. No UI changes needed. ```yaml url: https://nvidia.gpcloudservice.com/global-protect/getmsi.esp?version=64&platform=windows always_download: true install_script: path: install.ps1 ``` ### Why conditional GET instead of HEAD Fleet team [analysis of 276 maintained apps](https://github.com/fleetdm/fleet/pull/42216#issuecomment-4105430061) showed 7 apps where HEAD requests fail (405, 403, timeout) but GET works for all. Conditional GET eliminates that failure class: if the server doesn't support conditional requests, it returns 200 with the full body, same as today. ### Why opt-in 5 of 276 apps (1.8%) have stale ETags (content changes but ETag stays the same), caused by CDN caching artifacts (CloudFront, Cloudflare, nginx inode-based ETags). The `cache` key lets users opt in per package for URLs where they've verified ETag behavior is correct. Validation rejects `always_download: true` when hash_sha256` is set ## Changes - New YAML field: `cache` (bool, package-level) - New migration: `http_etag` VARCHAR(512) column (explicit `utf8mb4_unicode_ci` collation) + composite index `(global_or_team_id, url(255))` on `software_installers` - New datastore method: `GetInstallerByTeamAndURL` - `downloadURLFn` accepts optional `If-None-Match` header, returns 304 as `(resp, nil, nil)` with `http.NoBody` - ETag validated per RFC 7232 (ASCII printable only, no control chars, max 512 bytes) at both write and read time - Cache skipped for `.ipa` packages (multi-platform extraInstallers) - TempFileReader and HTTP response leak prevention on download retry - Docs updated in `yaml-files.md` ## What doesn't change - Packages with `hash_sha256`: existing hash-based skip, untouched - FMA packages: FMA version cache, untouched - Packages with `always_download: true`: identical to current behavior - Fleet UI: no changes ## Test plan Automated testing: - [x] 16 unit tests for `validETag` - [x] 8 unit tests for conditional GET behavior (304, 200, 403, 500, weak ETag, S3 multipart, no ETag) - [x] MySQL integration test for `GetInstallerByTeamAndURL` - [x] All 23 existing `TestSoftwareInstallers` datastore tests pass - [x] All existing service tests pass Manual testing: - [x] E2E: 86 packages across 6 CDN patterns, second apply shows 51 conditional hits (304) - [x] @sgress454 used a local fileserver tool to test w/ a new instance and dummy packages <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * ETag-based conditional downloads to skip unchanged remote installer files. * New always_download flag to force full re-downloads. * **Tests** * Added integration and unit tests covering conditional GETs, ETag validation, retries, edge cases, and payload behavior. * **Chores** * Persist HTTP ETag and related metadata; DB migration and index to speed installer lookups. * Added installer lookup by team+URL to support conditional download flow. * **Bug Fix** * Rejects using always_download together with an explicit SHA256 in uploads. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Scott Gress <scott@fleetdm.com> Co-authored-by: Scott Gress <scott@pigandcow.com> Co-authored-by: Ian Littman <iansltx@gmail.com>
News · Report a bug · Handbook · Why open source? · Art
Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.
What's it for?
Organizations like Fastly and Gusto use Fleet for vulnerability reporting, detection engineering, device management (MDM), device health monitoring, posture-based access control, managing unused software licenses, and more.
Explore data
To see what kind of data you can use Fleet to gather, check out the table reference documentation.
Out-of-the-box policies
Fleet includes out-of-the box support for all CIS benchmarks for macOS and Windows, as well as many simpler queries.
Take as much or as little as you need for your organization.
Supported platforms
Here are the platforms Fleet currently supports:
- Linux (all distros)
- macOS
- Windows
- Chromebooks
- Amazon Web Services (AWS)
- Google Cloud (GCP)
- Azure (Microsoft cloud)
- Data centers
- Containers (kube, etc)
- Linux-based IoT devices
Lighter than air
Fleet is lightweight and modular. You can use it for security without using it for MDM, and vice versa. You can turn off features you are not using.
Openness
Fleet is dedicated to flexibility, accessibility, and clarity. We think everyone can contribute and that tools should be as easy as possible for everyone to understand.
Good neighbors
Fleet has no ambition to replace all of your other tools. (Though it might replace some, if you want it to.) Ready-to-use, enterprise-friendly integrations exist for Snowflake, Splunk, GitHub Actions, Vanta, Elastic Jira, Zendesk, and more.
Fleet plays well with Munki, Chef, Puppet, and Ansible, as well as with security tools like Crowdstrike and SentinelOne. For example, you can use the free version of Fleet to quickly report on what hosts are actually running your EDR agent.
Free as in free
The free version of Fleet will always be free. Fleet is independently backed and actively maintained with the help of many amazing contributors.
Longevity
The company behind Fleet is founded (and majority-owned) by true believers in open source. The company's business model is influenced by GitLab (NYSE: GTLB), with great investors, happy customers, and the capacity to become profitable at any time.
In keeping with Fleet's value of openness, Fleet Device Management's company handbook is public and open source. You can read about the history of Fleet and osquery and our commitment to improving the product.
Is it any good?
Fleet is used in production by IT and security teams with thousands of laptops and servers. Many deployments support tens of thousands of hosts, and a few large organizations manage deployments as large as 400,000+ hosts.
Chat
Please join us in MacAdmins Slack or in osquery Slack.
The Fleet community is full of kind and helpful people. Whether or not you are a paying customer, if you need help, just ask.
Contributing 
The landscape of cybersecurity and IT is too complex. Let's open it up.
Contributions are welcome, whether you answer questions on Slack / GitHub / StackOverflow / LinkedIn / Twitter, improve the documentation or website, write a tutorial, give a talk at a conference or local meetup, give an interview on a podcast, troubleshoot reported issues, or submit a patch. The Fleet code of conduct is on GitHub.
What's next?
To see what Fleet can do, head over to fleetdm.com and try it out for yourself, grab time with one of the maintainers to discuss, or visit the docs and roll it out to your organization.
Production deployment
Fleet is simple enough to spin up for yourself. Or you can have us host it for you. Premium features are available either way.
Documentation
Complete documentation for Fleet can be found at https://fleetdm.com/docs.
License
The free version of Fleet is available under the MIT license. The commercial license is also designed to allow contributions to paid features for users whose employment agreements allow them to contribute to open source projects. (See LICENSE.md for details.)
Fleet is built on osquery, nanoMDM, Nudge, and swiftDialog.