#10292, #12554
When scanning tens of thousands of files for permissions, using the
`find` command exposed as a fleetd table is more performant than trying
to use the `file` table. This change caused the watchdog to *stop*
killing osquery because of exceeding memory or CPU limit.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
This PR requires the Windows MDM configuration changes - This will be
updated next week
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Documented any permissions changes
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
#10878
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#11531
How to use:
```
go run ./tools/mdm/apple/loadtest -api_token $API_TOKEN -fleet_url https://fleet.example.com -team_count 5 -team_extra_count 2 -loop_count 1
2023-05-15T16:05:06Z: 1. Creating 5 teams... (press enter to proceed)
2023-05-15T16:05:18Z: 1. Duration: 165.020742ms
2023-05-15T16:05:18Z: 2. Transfering one host to each team... (press enter to proceed)
2023-05-15T16:05:20Z: 2. Duration: 370.515741ms
2023-05-15T16:05:20Z: 3a. Add 10 profiles to all teams... (press enter to proceed)
2023-05-15T16:05:22Z: Applying profiles to team Team 0...
2023-05-15T16:05:22Z: Applying profiles to team Team 1...
2023-05-15T16:05:22Z: Applying profiles to team Team 2...
2023-05-15T16:05:22Z: Applying profiles to team Team 3...
2023-05-15T16:05:22Z: Applying profiles to team Team 4...
2023-05-15T16:05:22Z: 3a. Duration: 502.557741ms
2023-05-15T16:05:22Z: 3b. Waiting for all profiles to be applied on all teams...
2023-05-15T16:05:22Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:27Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:32Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:37Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:42Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:1 Pending:0 Failed:0} 43:{Verifying:1 Pending:0 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:48Z: 3b. Duration: 25.487993296s
2023-05-15T16:05:48Z: 4a. Modify a profile on all teams... (press enter to proceed)
2023-05-15T16:05:50Z: 4a. Duration: 654.712391ms
2023-05-15T16:05:50Z: 4b. Waiting for all profiles to be applied on all hosts of all teams...
2023-05-15T16:05:50Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:05:55Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:06:00Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:06:05Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:0 Pending:1 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:06:10Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:1 Pending:0 Failed:0} 43:{Verifying:0 Pending:1 Failed:0} 44:{Verifying:0 Pending:1 Failed:0}
45:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:06:15Z: Waiting for all profiles to be applied on hosts..., summary: map[41:{Verifying:0 Pending:1 Failed:0} 42:{Verifying:1 Pending:0 Failed:0} 43:{Verifying:1 Pending:0 Failed:0} 44:{Verifying:1 Pending:0 Failed:0}
45:{Verifying:1 Pending:0 Failed:0}]
2023-05-15T16:06:20Z: 4b. Duration: 30.48166655s
2023-05-15T16:06:20Z: 5. Creating extra 2 teams... (press enter to proceed)
2023-05-15T16:06:22Z: 5. Duration: 72.531251ms
2023-05-15T16:06:22Z: 6a. Moving one host to each new extra 2 teams... (press enter to proceed)
2023-05-15T16:07:23Z: 6a. Duration: 164.446788ms
2023-05-15T16:07:23Z: 6b. Waiting for all profiles to be applied on all hosts of the extra teams...
2023-05-15T16:07:23Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:07:29Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:07:34Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:07:39Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:07:44Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:1 Pending:0 Failed:0}]
2023-05-15T16:07:49Z: 6b. Duration: 25.210811603s
2023-05-15T16:07:49Z: 7a. Add 10 profiles to all extra 2 teams... (press enter to proceed)
2023-05-15T16:07:50Z: 7a. Duration: 175.694423ms
2023-05-15T16:07:50Z: 7b. Waiting for all profiles to be applied on all hosts of the extra teams...
2023-05-15T16:07:50Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:07:55Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:08:01Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:08:06Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:0 Pending:1 Failed:0}]
2023-05-15T16:08:11Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:1 Pending:0 Failed:0}]
2023-05-15T16:08:16Z: Waiting for all profiles to be applied on hosts..., summary: map[46:{Verifying:0 Pending:1 Failed:0} 47:{Verifying:1 Pending:0 Failed:0}]
2023-05-15T16:08:21Z: 7b. Duration: 30.219261468s
2023-05-15T16:08:21Z: 8. Destroy 2 extra teams... (press enter to proceed)
2023-05-15T16:08:31Z: 8. Duration: 134.372194ms
```
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
Related to #11534 this is an extract from the code I used to build a
prototype to see if `swiftDialog` would work for us.
This is very similar to the work we did for Nudge previously.
Script that takes either a text file or .csv containing a list of teams
as input and adds the teams to Fleet then generates installers.
---------
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
#7970
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Add the ability to have the `create_n_teams` script generate team names
of the form "Team {5-emoji}", where '5-emoji' is a sequence of 5
randomly generated emoji.
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
## Addresses #10756https://www.loom.com/share/b5ba605b892c4c789a73f05d160819cc
- Set the teams dropdown's max-height to be 66% of the viewport height
- Made a small script to automate creation of an arbitrary number of
teams
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#9609
This PR also fixes#10777.
The issue is: We were using `svc.AppConfig` instead of
`svc.ds.AppConfig` to retrieve the SMTP credentials.
`svc.AppConfig` obfuscates credentials, whereas `svc.ds.AppConfig` does
not.
To help prevent this from happening again I've renamed `svc.AppConfig`
to `svc.AppConfigObfuscated`.
I've also added a new test SMTP server
(https://github.com/axllent/mailpit) that supports Basic Authentication
and tests that make use of it to catch these kind of bugs (the tests are
executed when running `go test` with `MAIL_TEST=1`).
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#8411
PS: I've opened #10209 to solve the issue with Golang Code Coverage CI
checks.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
## Summary
This pull request is created by [Secure
Repo](https://app.stepsecurity.io/securerepo) at the request of @zwass.
Please merge the Pull Request to incorporate the requested changes.
Please tag @zwass on your message if you have any questions related to
the PR. You can also engage with the
[StepSecurity](https://github.com/step-security) team by tagging
@step-security-bot.
## Security Fixes
### Secure Dockerfiles
Pin image tags to digests in Dockerfiles. With the Docker v2 API
release, it became possible to use digests in place of tags when pulling
images or to use them in FROM lines in Dockerfiles.
- [The Open Source Security Foundation (OpenSSF) Security
Guide](https://github.com/ossf/scorecard/blob/main/docs/checks.md#pinned-dependencies)
## Feedback
For bug reports, feature requests, and general feedback; please create
an issue in
[step-security/secure-repo](https://github.com/step-security/secure-repo).
To create such PRs, please visit https://app.stepsecurity.io/securerepo.
Signed-off-by: StepSecurity Bot <bot@stepsecurity.io>
---------
Signed-off-by: StepSecurity Bot <bot@stepsecurity.io>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
#9260
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
#9253
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
---------
Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
Update the default file carver block size to be compatible with MySQL 8
& S3.
Update surrounding docs.
Various other updates to references of MySQL versions (all terraform
deploys are now defaulted MySQL 8 in AWS)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
Luke Heath
