For #25160.
Measured improvement by splitting the MySQL query into two queries to
use the indexes more efficiently:
- ~8s vs ~100ms for ~30k entries in software_titles for ~1.7k incoming
software without bundle_identifier (linux software).
- ~1.64s vs ~2ms for 25k entries in software_titles and ~500 incoming
new software with bundle_identifier + ~200 new software without
bundle_identifier (macOS software).
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [X] Manual QA for all new/changed functionality
# Overview
This PR adds support for remote YARA queries to osquery-perf, so that
remote YARA queries can be load-tested.
# Details
The existing `runLiveQuery()` is updated to branch off into different
query running functions based on the content of the query. If the query
contains `from yara` and `sigurl`, then the new `runLiveYaraQuery()`
function is run which makes a request to the Fleet "get yara rules" API
before returning an appropriate response. Otherwise, the new
`RunLiveMockQuery()` function is run which includes the previous logic
for sending a mock response.
# Testing
I don't see any automated testing for osquery-perf, but I manually
tested in the following way:
1. Started osquery-perf with `go run agent.go`
2. Ran a live query on the new host using
```
SELECT * FROM yara where sigurl="https://localhost:8080/api/osquery/yara/rule1.yar"
```
and verified that the result was as-expected:
<img width="1642" alt="image"
src="https://github.com/user-attachments/assets/a2c9cacf-e28d-409c-8e83-1c82809b89c0"
/>
I also used a log in Fleet to verify that the "get yara rules" API was
really being called.
3. Ran another live query on the host using:
```
SELECT * FROM system_info"
```
and verified that the result was as expected:
<img width="1665" alt="image"
src="https://github.com/user-attachments/assets/a8d35389-f193-4902-badf-200d760fdf46"
/>
I also tested that sending a `sigurl` with the wrong host returns a
`live yara query failed because sigurl host did not match server
address` error
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
#22810
# Demo
[](http://www.youtube.com/watch?v=le71QQ92suc)
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Orbit runs on macOS, Linux and Windows. Check if the orbit
feature/bugfix should only apply to one platform (`runtime.GOOS`).
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Part 2 of https://github.com/fleetdm/fleet/issues/19447
- iOS and iPadOS user-installed apps are loaded into Fleet
- Added an additional identifier into software_titles table to
differentiate between iOS/iPadOS apps
- Updated nano queue timestamp precision
Note: TestIntegrationsMDM/TestVPPApps fails when run as part of the
suite, but passes standalone. I'd like to proceed with merging this PR,
and figure out the issue next week.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Roberto Dip <rroperzh@gmail.com>
#19789
When osquery returns a serial number of -1 (default value), we keep the
existing serial number in the database.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
## Addresses #19127
- Also replace all ocurrences of "comma separated" with
"comma-separated"
- [x] Changes file added for user-visible changes in `changes/`
- [x] `SELECT *` is avoided, SQL injection is prevented (using
placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#18119
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
Sample on how to simulate 50 iPads and 50 iPhones:
```sh
go run ./cmd/osquery-perf -host_count 100 -os_templates iphone_14.6.tmpl:50,ipad_13.18.tmpl:50 -mdm_scep_challenge <...>
```
@xpkoala came across this issue while performing a load test for the
calendar backoff feature with rolling.
It changed our baseline mainly while performing the hosts enrollment
during load tests
This was discovered by @xpkoala while performing a load test for the
calendar backoff feature.
Some enroll requests were failing due to enrolling hosts too fast (`-var
loadtest_containers` from `0` to `40` at once), and osquery-perf had a
bug in the enroll request where the `bytes.Buffer` was being incorrectly
reused thus sending an empty body on the enroll retries, getting 400s
from Fleet due to `Expected JSON Body`:
```
2024/04/11 18:57:49 request failed: 400
```
#17361#17148
In GET fleet/hosts/:id response, added the following fields:
- orbit_version
- `orbit_version == null` means this agent is not an orbit agent
- fleet_desktop_version
- `fleet_desktop_version == null` means this agent is not an orbit agent
or it is an older version which is not collecting the desktop version
- `fleet_desktop_version == ""` means this agent is an orbit agent but
does not have fleet desktop
- scripts_enabled
- `scripts_enabled == null` means this agent is not an orbit agent or it
is an older version which is not collecting scripts_enabled
In orbit_info table, added the following fields:
- desktop_version
- scripts_enabled
Updated docs for orbit_info PR:
https://github.com/fleetdm/fleet/pull/18135
Updated API docs: https://github.com/fleetdm/fleet/pull/17814
MDM lock/unlock/wipe error messages are not part of this PR. They will
be in a separate PR.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Fixes from the changes added to #17003.
- 50% failure for the software query was not realistic (changing to 5%).
- 50% failure for the VS Code query was also not realistic (changing to
5%).
- Renamed a wrongly named variable.
#15565
Replace the use of the isFederated registry key with a keys that check
for AAD (Azure Active Directory, now Entra ID)
Federated enrollment (`isFederated`) seems to be when windows uses a
Discovery MDM endpoint to get its policy and management endpoint
configuration. This is always the case when a client is enrolled with
fleet, so installations always show up as automatic.
It's being replaced by a different key, `AADResourceID`, which appears
to identify the resource that controls the automated deployment. In my
tests it only appears to be populated when the computer is enrolled
through automated deployments. This key appears on both Windows 10 and
11.
There is a similar key, `AADTenantID`, which appears to identify the
client (tenant) to the Azure cloud. I haven't seen this ID in our
systems, so it is likely exclusively used in Azure. Both this key and
`AADResourceID` seem to always be set at the same time, so we only
check for the `AADResourceID`.
I've also added documentation on the registry keys I've analyzed for future reference.
#17003
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
This is a bug in the generation of results of scheduled queries in
osquery-perf.
It seems the bug has been around since we added scheduled query result
support in osquery-perf.
PS: In my Fleet downtime tests for
https://github.com/fleetdm/fleet/issues/16423 I probably missed this
because I was using a high frequency queries (150s intervals IIRC).
Orbit changes for #16423.
Should also fix#16326 (in case of network errors).
Orbit will log the following every 5 minutes:
```
2024-02-20T14:27:40-03:00 INF network error error="Post \"https://localhost:8080/api/fleet/orbit/config\": dial tcp [::1]:8080: connect: connection refused"
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
#16423
- We need to consume less memory while Fleet is down (otherwise we might
bring the load test environment down, local tests showed that in 30m I
got to 600 MB of memory usage) so am generating the results when needed
only.
- Also fixing the release of fasthttp requests and responses (according
to their docs).
#15703
`wall_time_ms` (not `wall_time`) coming from osquery will now be put
into `wall_time` in our DB.
Hence, `wall_time` coming from Fleet API will now be in milliseconds and
not seconds
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Manual QA for all new/changed functionality
These are the osquery-perf changes for the load test to be performed in
#16423.
- Adding buffering of results when they fail to be sent to the server
(when Fleet is offline/down).
- I'm changing many `a.waitingDo(req, res)` to just do one request and
if it fails it will retry on the next interval, which is what osquery
does on
`/api/osquery/config`/`/api/osquery/distributed/read`/`/api/osquery/distributed/write`/`/api/osquery/log`.
- I'm reducing the logs as they will accumulate considerably while Fleet
is down for 30m with 140k hosts.
- Changing from `fmt.Printfs` to `log.Printfs` for consistency.
- Adding more fine grained stats (to know which endpoints are failing)
- Adding a mode to disable Fleet Desktop to simulate what the customer
is running.
- Adding `logger_tls_max_lines` flag that simulates the osquery setting.
Added `gigs_total_disk_space` to host endpoint responses.
- Now user will know host disk size even when disk is full
(percent_disk_space_available==0)
#15058
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
📺 Loom explaining the PR:
https://www.loom.com/share/8c6ec0c362014910931f183d68307525?sid=7de51e6f-a59d-4d8f-b06b-f703f1692f17#467
Live Queries now collect stats. Stats are collected for saved queries
present in the Queries tab.
- After running a live query, user will see updated stats in Queries
tab.
- Query stats on Host Details page will no longer be cleared after
host/agent reboots.
- Query stats are now deleted when query is deleted.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
This PR introduces the following fixes:
- Fixes a race condition by protecting the `scheduledQueries` and
`scheduledQueryData` fields with a `sync.Mutex` (*).
- Adds some more information about request counts for /log, /config and
/distributed/read requests (and uptime).
- Fixes the resource leaks around creating time.Ticks on every for loop
iteration.
(*) Sample of the race condition when running with `-race`:
```
==================
WARNING: DATA RACE
Read at 0x00c000604800 by goroutine 20:
main.(*agent).runLoop()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:525 +0x71b
main.main.func2()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1737 +0x4e
Previous write at 0x00c000604800 by goroutine 40:
main.(*agent).config()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:915 +0xb30
main.(*agent).runLoop.func2()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:512 +0x37
```
This is safe to merge during freeze as this is an osquery-perf only
change.
```
panic: runtime error: index out of range [1] with length 1
goroutine 14 [running]:
main.(*agent).config(0xc0001e5b80)
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:847 +0xb74
main.(*agent).runLoop(0xc0001e5b80, 0x0?, 0x0?)
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:463 +0x4d2
created by main.main in goroutine 1
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1666 +0x1285
```
#13527
(Adding @mna to double check the changes in the async implementation of
policy result storage)
This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml
# Leave running for some time
fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```
After applying CIS policies previous to these changes:
After applying these changes and applying the same policies:
These osquery-perf changes were useful to test/troubleshoot #10957.
It basically allows osquery-perf to simulate no results and/or failures
when running live queries on the devices.
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Lucas Manuel Rodriguez
