Scott Gress
d7fa35e417
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** Resolves #45715 # Details This PR refactors the way the charts module stores historical data to use the [roaring bitmap](https://github.com/RoaringBitmap/roaring) package instead of saving raw bitmaps. See [this blurb](https://github.com/RoaringBitmap/roaring#how-does-roaring-compares-with-the-alternatives) to learn how roaring compresses data, but TL;DR for our purposes it represents a huge improvement especially for larger deployments where host ID numbers may be very large. In testing, some data was reduced 96%. The majority of the changes in this PR are straight swapping of types from `[]byte` to `*roaring.Bitmap` in vars and function signatures, and updating the internals of our bit math helpers to use roaring methods instead of native AND and OR methods. I've tried to comment on all functional changes. Since the charts have been shipped already, so there will be data in the wild in the prior "dense" format, the code still handles dense bitmaps on _read_, but will always _write_ roaring bitmaps. The majority of the data will therefore have turned over within 30 days on its own, but I plan on a follow-up PR that will transform open rows when the cron runs so that we should be guaranteed to turn over completely within 30 days. # Checklist for submitter If some of the following don't apply, delete the relevant line. - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [X] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. ## Testing - [X] Added/updated automated tests - Tests updated to accommodate the new format, and existing unchanged tests act as proof against regression - [X] QA'd all new/changed functionality manually - Using a tool that dumps the `host_scd_data` rows data into a JSON file (with the keys being entity_id+data and the values being host IDs on that date), compared the data from main branch and this and confirmed they're identical - With a host count of ~9000, some of which have IDs of over 1,000,000, the data storage requirements were: * 82,558,976 bytes for dense * 2,867,200 for roaring (a 96% decrease) For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - should hugely improve - [X] Alerted the release DRI if additional load testing is needed ## Database migrations - [X] Checked schema for all modified table for columns that will auto-update timestamps during migration. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented roaring bitmaps in historical data collection to optimize bitmap handling for chart data aggregation * Added encoding support to bitmap storage schema for flexible data representation <!-- end of auto-generated comment: release notes by coderabbit.ai -->
News · Report a bug · Docs · Why open source? · Art
Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.
What's it for?
Fleet gives you a single system to secure and maintain all your computing devices over the air. You can do MDM, patch stuff, deploy software, and verify anything, all from one place, across every OS your organization uses.
Fleet works directly with data and events from the native operating system, down to the bare metal. Strong diagnostics let you investigate errors on end-user devices and collect accurate audit evidence in minutes.
Get started
You can try Fleet out for yourself, or grab time with one of the maintainers to chat.
Is it any good?
Fleet is used in production by IT and security teams managing thousands of devices. Many deployments support tens of thousands of hosts, and a few large organizations manage 400,000 or more.
Supported platforms
- Linux (all distros)
- macOS
- Windows
- Chromebooks
- iOS and Android (BYOD or corporate-owned)
- Amazon Web Services (AWS)
- Google Cloud (GCP)
- Azure (Microsoft cloud)
- Data centers
- Containers (kube, etc)
- Linux-based IoT devices
Infrastructure as code
Manage your fleet with GitOps, or use the GUI, REST API, webhook events, and the fleetctl command-line tool.
Linux support
First-class support for all major distros. Linux gets the same attention and visibility as macOS and Windows.
Visibility and compliance
Fleet can report on hundreds of attributes across your devices and ships with CIS benchmarks for macOS and Windows and comprehensive operating system, hardware, and software data. Check out the table reference documentation to see what's available.
Open by design
Fleet is open source and transparent about what it can and can't see. End users can verify exactly how the agent works and what data their company collects. Fleet collects only the data needed to manage and secure devices, not private activity like keystrokes, emails, or webcams.
Good neighbors
Ready-to-use, enterprise-friendly integrations exist for Snowflake, Splunk, GitHub Actions, Vanta, Elastic Jira, Zendesk, and more. Fleet also works with tools such as Munki, Chef, Puppet, Ansible, CrowdStrike, and SentinelOne.
Lighter than air
Fleet is lightweight and modular. You can use it for MDM without using it for security, and vice versa. You can turn off features you are not using.
Free as in free
The free version of Fleet will always be free. Fleet is independently backed and actively maintained with the help of many amazing contributors.
Longevity
The company behind Fleet is founded (and majority-owned) by true believers in open source. The company's business model is influenced by GitLab (NYSE: GTLB), with great investors, happy customers, and the capacity to become profitable at any time.
Fleet Device Management's company handbook is public and open source. You can read about the history of Fleet and our commitment to improving the product.
Chat
The Fleet community is full of kind and helpful people. Whether or not you are a paying customer, if you need help, just reach out.
Contributing 
Contributions are welcome, whether you answer questions on Slack / GitHub / LinkedIn, improve the documentation or website, write a tutorial, give a talk at a conference or local meetup, give an interview on a podcast, troubleshoot reported issues, or submit a patch. The Fleet code of conduct is on GitHub.
License
The free version of Fleet is available under the MIT license. The commercial license is also designed to allow contributions to paid features for users whose employment agreements allow them to contribute to open source projects. (See LICENSE.md for details.)
Fleet is built on osquery, nanoMDM, Nudge, and swiftDialog.