Commit Graph
7920 Commits
Author SHA1 Message Date
Lucas Manuel Rodriguez 6415f062c6 Reduce size of DistributedQueryResult to improve live query performance (#11882)
This was found while working on #10957.

When running a live query, a lot of unused host data is stored in Redis
and sent on every live query result message via websockets. The frontend
and fleetctl just need `id`, `hostname` and `display_name`. (This
becomes worse every time we add new fields to the `Host` struct.)

Sample of one websocket message result when running `SELECT * from
osquery_info;`:

size in `main`: 2234 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":57,\"host\":
{\"created_at\":\"2023-05-22T12:14:11Z\",\"updated_at\":\"2023-05-23T12:31:51Z\",
\"software_updated_at\":\"0001-01-01T00:00:00Z\",\"id\":106,\"detail_updated_at\":\"2023-05-23T11:50:04Z\",
\"label_updated_at\":\"2023-05-23T11:50:04Z\",\"policy_updated_at\":\"1970-01-02T00:00:00Z\",
\"last_enrolled_at\":\"2023-05-22T12:14:12Z\",
\"seen_time\":\"2023-05-23T09:52:23.876311-03:00\",\"refetch_requested\":false,
\"hostname\":\"lucass-macbook-pro.local\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"platform\":\"darwin\",\"osquery_version\":\"5.8.2\",\"os_version\":\"macOS 13.3.1\",\"build\":\"22E261\",\"platform_like\":\"darwin\",\"code_name\":\"\",
\"uptime\":91125000000000,\"memory\":34359738368,\"cpu_type\":\"x86_64h\",\"cpu_subtype\":\"Intel x86-64h Haswell\",\"cpu_brand\":\"Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz\",\"cpu_physical_cores\":4,\"cpu_logical_cores\":8,\"hardware_vendor\":\"Apple Inc.\",\"hardware_model\":\"MacBookPro16,2\",\"hardware_version\":\"1.0\",
\"hardware_serial\":\"0DPQR4HMD1FZ\",
\"computer_name\":\"Lucas’s MacBook Pro\",\"public_ip\":\"\",
\"primary_ip\":\"192.168.0.230\",\"primary_mac\":\"68:2f:67:8e:b6:1f\",
\"distributed_interval\":1,\"config_tls_refresh\":60,\"logger_tls_period\":10,\"team_id\":null,
\"pack_stats\":null,\"team_name\":null,
\"gigs_disk_space_available\":386.23,\"percent_disk_space_available\":40,
\"issues\":{\"total_issues_count\":0,\"failing_policies_count\":0},
\"mdm\":{\"enrollment_status\":null,\"server_url\":null,\"name\":\"\",\"encryption_key_available\":false},
\"status\":\"online\",\"display_text\":\"lucass-macbook-pro.local\",\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"b7ee9363a7c686e76e99ffb122e9c5241a791e69\",\"config_valid\":\"1\",
\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",\"start_time\":\"1684757652\",
\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",
\"version\":\"5.8.2\",\"watcher\":\"8364\"}],\"error\":null}}"]
```

vs. size of the message result on this branch: 675 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":59,
\"host\":{\"id\":106,\"hostname\":\"lucass-macbook-pro.local\",
\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"f80dee827635db39077a458243379b3ad63311fd\",
\"config_valid\":\"1\",\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",
\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",
\"start_time\":\"1684757652\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"version\":\"5.8.2\",
\"watcher\":\"8364\"}]}}"]
```

Manual tests included running with an old fleetctl running with a new
fleet server, and vice-versa, a new fleetctl running against an old
fleet server.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-05-25 08:11:53 -03:00
Mike McNeil f68054652a keep it simple (#11955)
the last sentence read weird, so just cutting it
2023-05-24 23:58:01 -05:00
Mike McNeil 141467878b Update README.md (#11954)
.
2023-05-24 23:46:19 -05:00
EricandMike Thomas 02bb987b22 Website: Update pricing page (#11910)
Changes:
- Updated the pricing page to match the latest wireframes.
- Updated the features in `pricing-features-table.yml`
- Updated `build-static-content.js` to remove support for the Ultimate
tier in `pricing-features-table.yml`

---------

Co-authored-by: Mike Thomas <mthomas@fleetdm.com>
2023-05-25 12:40:19 +09:00
Juan Fernandez 012a633866 Fixed query used when aggregating WinOS stats (#11840)
Aggregated stats on the dashboard page were showing the wrong WinOS version.
2023-05-24 22:11:43 -04:00
gillespi314 3e3880b62d Modify GET /mdm/apple/profiles API endpoint to return empty array instead of null when results set is empty (#11904) 2023-05-24 18:56:39 -05:00
Luke Heath 8d47cbca4d Add QA section to website request template (#11871) 2023-05-25 08:35:08 +09:00
Luke Heath 9055564b0c Prepare v4.32.0 (#11908) 2023-05-24 17:31:20 -05:00
Mike McNeil 35d3764f3a Add Jo for autoapprovals in issue templates (#11950) 2023-05-24 17:00:09 -05:00
Luke Heath 39ab012a35 Add vertical padding to file paths (#11949) 2023-05-24 16:37:18 -05:00
Luke Heath fae007b50d Show file path for all host types (#11946) 2023-05-24 15:22:06 -05:00
Luke Heath bfae379262 Remove small padding on file path cell (#11944) 2023-05-24 15:13:05 -05:00
Juan Fernandez 2249d171bc Return DB results even if nothing has changed (#11936) 2023-05-24 14:05:45 -05:00
Lucas Manuel Rodriguez 9f7383b9ff Filter out "null" string in installed_path from osquery (#11931) 2023-05-24 12:28:20 -05:00
Gabriel Hernandez e500a4f65d Fix icon not being aligned properly on Host details/ My device page. (#11919) 2023-05-24 09:10:07 -05:00
Zach Wasserman 048de9a002 Move all workstations to macOS 13.4.0 (#11918) 2023-05-23 20:42:08 -07:00
Rachael ShawandMike McNeil 9cd094a22a Add rachaelshaw to CODEOWNERS for docs (#11793)
+ Added myself to `/docs/` as discussed with @mikermcneil 
+ Also to the specific doc sections for the REST API and API for
contributors because they were separated out (let me know if you'd
rather I remove myself from those, don't mean to get grabby!)
+ Fixed a typo 

.

---------

Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
2023-05-23 21:46:41 -05:00
Mike McNeil 37a976adf7 Update custom.js (#11914) 2023-05-23 21:46:24 -05:00
Zach Wasserman 97464ce045 Add customer data deletion policy (#11911) 2023-05-23 19:00:56 -07:00
Jacob Shandling c733aca3f2 UI: Multiple team oberver plus bug (#11897) 2023-05-23 18:10:52 -05:00
gillespi314 78635b310c Improve webhook URL validation in manage software automations UI (#11909) 2023-05-23 18:00:45 -05:00
Luke Heath 162b709eb2 Adjust bold weight (#11905) 2023-05-23 16:24:01 -05:00
Noah Talerman c0ea5801da Update fleetdm.com/pricing (#11902)
- Self-managed autoupdate registry already exists. Docs are here: https://fleetdm.com/docs/deploying/fleetctl-agent-updates
2023-05-23 13:18:41 -07:00
Luke Heath 846ee18cb3 Add example workflow (#11893) 2023-05-23 13:52:21 -05:00
gillespi314 259d4fa1ac Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
Noah Talerman 75ad1ad06d Docs: remove issuer_uri from end_user_authentication (#11639)
- `issuer_uri` isn't required for this feature and won't included in the
UI.
2023-05-23 10:59:28 -07:00
Roberto Dip e57f90fbac fix display issues with screenshot in MDM migration flow (#11866)
For #11858, I reproduced the issue by running a local server behind
ngrok, _with the exact_ same path as the one in the website:
`https://server-url/images/permanent/mdm-migration-screenshot-768x180@2x.png`

I tried multiple combinations, but at the end, removing the `@` made the
trick. My guess is that's something to do with the markdown parser
library used by swiftDIalog.

I also removed a rogue `\` that was being displayed.
2023-05-23 14:29:42 -03:00
Roberto Dip 04ef6f13f8 fix typescript type (#11887) 2023-05-23 11:47:39 -05:00
gillespi314 62e8e9e36a Fix unreleased bugs related to policies page URL params (#11873) 2023-05-23 10:46:19 -05:00
Roberto Dip d0658868dc adjust the UI to show a refetching spinner on critical queries (#11868)
A possible solution for https://github.com/fleetdm/fleet/issues/11860,
this updates the "My Device" page to show the spinner rotating if
critical queries are being fetched:



https://github.com/fleetdm/fleet/assets/4419992/0a84508a-4df2-4da3-b9bc-b3e75bab5e82


# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Manual QA for all new/changed functionality
2023-05-23 12:43:50 -03:00
Mike Thomas 30354137b6 Website update wallpapers (#11759)
I updated the wallpapers on https://fleetdm.com/logos. Closes
https://github.com/fleetdm/fleet/issues/11667
.
2023-05-23 14:08:35 +09:00
Mike McNeil 301840a334 Update website-request.md (#11881) 2023-05-22 22:47:49 -05:00
Jarod ReyesandMike McNeil 4fd3f45fb3 Update README.md (#11863)
Add private channel DRI for marketing.

---------

Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
2023-05-22 22:42:05 -05:00
Mike Thomas 6e6699afd5 Update "Ask us anything" on pricing.ejs to link to /contact (#11877)
As the Hubspot chat flow is not configured to accept free-form
questions, I updated the "Ask us anything" button at the bottom of
/pricing to link to https://fleetdm.com/company/contact

- [x] Manual QA for all new/changed functionality
2023-05-23 11:52:52 +09:00
Noah Talerman da3a42d053 Contributing docs: Remove outdated command (#11788)
- Remove the `fleetctl apple-mdm apple-mdm enrollment-profiles
create-automatic` command because it no longer exists in Fleet
2023-05-22 17:08:29 -07:00
Eric b11f302981 Website: update earliest supported version of Edge (#11872)
Changes:
- Changed the earliest version of Edge that the Fleet website supports
from `v17` » `v80` (Global usage 0.02% for edge <`v18` (from
caniuse.com) )
2023-05-22 17:10:22 -05:00
JDandAndrew Baker 8c166b75fa Building an effective dashboard with Fleet's REST API, Flask, and Plotly (#11862)
@dherder's article, Building an Effective Dashboard with Fleet's REST
API, Flask, and Plotly: A Step-by-Step Guide.

---------

Co-authored-by: Andrew Baker <89049099+DrewBakerfdm@users.noreply.github.com>
2023-05-22 14:01:23 -07:00
Zachary Winnerman af8ea1eb2f Use tags for our code (#11854)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

See #11583
2023-05-22 14:07:09 -04:00
Mike McNeil 6cce5aab99 Remove bizops on-call for now (#11850)
.
2023-05-22 11:28:30 -05:00
Mike McNeil d4761b2bfa Delete application form because too many applicants for us to keep up… (#11849)
… with


.
2023-05-22 11:26:24 -05:00
Jacob Shandling d4b2d28021 Always update query param on policies page, even when empty (#11821) 2023-05-22 09:46:53 -04:00
Mike McNeil eb70a28adc Clarify you don’t have to use ALL of CIS (#11823)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-05-19 17:56:40 -05:00
Mike McNeil 00a1a0a947 apply Brock’s feedback (#11822) 2023-05-19 17:48:33 -05:00
Sabrina Coy 2e54cfc95c Update README.md to fix typo in "GitLab" name (#11819) 2023-05-19 15:54:23 -05:00
Zachary Winnerman 91e41ec670 Allow for parallel spinup of sandbox instances (#11779)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

Closes #7118
2023-05-19 16:00:51 -04:00
Mike McNeil 7c6b685eed Fix img (#11814)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-05-19 14:00:14 -05:00
Mike McNeil fcb5ad8740 fix link (#11813)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-05-19 13:58:50 -05:00
Mo Zhu d74a15642d convenience link to team schedule in REST API docs (#11782) 2023-05-19 11:44:27 -07:00
Mo Zhu 707a137d38 fix aggreement typo (#11808) 2023-05-19 11:43:29 -07:00
Mike McNeil 15ac8f7be7 Revamp readme (#11811) 2023-05-19 13:27:46 -05:00