For #26218
Refactoring android/mysql/testing_utils.go to remove duplication with
main mysql package.
No functional changes.
# Checklist for submitter
- [x] Manual QA for all new/changed functionality
For #24040
Add gitops option for the request to modify the app config.
There was an issue with the abm token teams getting reset to default
anytime the `PATCH /fleet/config` endpoint was called. @jahzielv and I
discussed various options on how to solve this and agreed that the
approach taken in this PR was the quickest but not the best. Ideally,
we'd like the gitops client to send back the data to the endpoint that
its going to update. This will allow the `PATCH /fleet/config` endpoint
to work like a standard `PATCH` request and only update the options
provided instead of updating the app config differently depending on the
client calling the endpoint.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
This feature was requested by a customer that has since decided not to
continue using the MSP dashboard. We had identified some edge cases with
the feature that we wanted to add patches for, so rather than leave it
in the current state (which isn't being used) we decided to back the
code out entirely.
This is a revert of commit
https://github.com/fleetdm/fleet/commit/8419b8e87a2c9bf98a2736726673f7893743f483.
For #26218
Basic Android MDM on/off backend functionality. Manually tested.
The following env vars must be set:
```
FLEET_DEV_ANDROID_ENABLED=1
FLEET_DEV_ANDROID_SERVICE_CREDENTIALS=$(cat credentials.json)
FLEET_DEV_ANDROID_PUBSUB_TOPIC=projects/your-project/topics/your-topic
```
I picked https://github.com/go-json-experiment/json as the JSON library,
which seems like the safest option.
- will become json/v2 at some point
- currently used in production by other companies, like Tailscale
- well-maintained
- Some context here: https://github.com/fleetdm/fleet/issues/25512
Plan for next work:
- refactoring from 1st PR
- add pubsub with device enroll -> spec proxy for fleetdm.com
- come back to this sub-task to add tests and finish TODOs
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> For #24222
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
> For #26178
# Checklist for submitter
## Data from testing
We tested the changes out by directly creating the index in our dogfood
env. We saw a decrease in resource usage, captured in these screenshots:
DB load (taller spike towards the left is without the index, subsequent
spikes are after the index was added)
DB load over a longer period (overnight 2025-02-12 -> 2025-02-13)
CPU utilization
Memory usage
---
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
Android scaffold code and refactorings
- Android packages intended to be decoupled from other Fleet code
Video explaining the PR: https://www.youtube.com/watch?v=cza-35Z9Wxk
# Checklist for submitter
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> For #26290
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> For #26275
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Manual QA for all new/changed functionality
- [x] Added/updated automated tests
For #26011, follow-up work for #26003.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
The software titles unique key was changed to include bundle identifier
in https://github.com/fleetdm/fleet/pull/25794. This caused an issue
when running gitops - installing a software with the same bundle
identifier but different names.
https://github.com/fleetdm/fleet/issues/26226
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
The host software version and VersionEndExcluding did not always get
parsed by semver properly. Switching to using `SmartVerCmp` from the
nvdtools code. This is much more relaxed when parsing versions.
https://github.com/fleetdm/fleet/issues/24810
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [ ] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #25648
Fixed issue where `fleetctl gitops` was NOT deleting macOS setup
experience bootstrap package and enrollment profile. GitOps should clear
all settings that are not explicitly set in YAML config files.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #17700
update error message for ddm profiles having the same name
- [ ] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> For #26086
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #25770
We already unmarshal macOS/Windows settings (added by Martin), so we
replace the path with an absolute file path and keep them unmarshalled
so they don't have to be re-unmarshalled later. Note: the custom
UnmarshalJSON method on these structs checks for (and handles) legacy
format (before labels were added).
Also some refactorings:
- extracted `extractControlsForNoTeam`
- reorganized `TestGitOpsBasicGlobalAndNoTeam` with subtests -- I did
not actually change functionality of this test
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
#22791
This will prevent the occasional redirect from breaking live queries.
Customers can still disable the redirects by setting
`redis.cluster_follow_redirections` to `false`.
Won't solve the false positive issues due to version number mismatches,
but will fix the false negative where CE wasn't matching at all, and
this is a full fix for IJ CE installed via JetBrains Toolbox.
For #25662.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #25812
I am adding some debug logging for DDM configuration profile status to
assist in future potential debug. This change should have no noticeable
functional changes.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- Add missing `status` for `installed_app_store_app`
- @noahtalerman: @iansltx and I tested the `installed_app_store_app`
activity and saw that the docs are missing `status`
- Clarify that `_software` activities are for Fleet-maintained apps and
custom packages
For #25235
This allows software with different names but the same bundle identifier
to be grouped under the same title. It also allows for software with the
same name but different bundle identifiers to be under two separate
titles.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
Also includes a CPE translation fix to avoid missing Python prereleases
on Windows, though we have further work to do to generate a CPE that
matches the one NVD uses.
For #24611.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
Refactoring some functionality out of the service package so it can be
reused by a different service package.
- auth middleware
- logging errors
No functional changes.
For #23465.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [X] Manual QA for all new/changed functionality
For #25901
Need to close `com.electron.dockerdesktop`
Check if destination for symlink exists prior to symlinking
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
For #25555
# Checklist for submitter
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
This PR updates the `NewLabel` service to use the
`UpdateLabelMembershipByHostIDs` method previously added by
@jacobshandling rather than using `ApplyLabels`. The latter method has
performance issues when adding large numbers of hosts at once to a
manual label (see #25555) because it does an expensive lookup of host
names before transforming those into Fleet host IDs. The new code skips
the middleman and transforms host identifiers directly to Fleet host
IDs, and does so using a batching strategy to ensure the queries don't
get too large.
This PR does update `UpdateLabelMembershipByHostIDs` slightly to return
an updated Label object and host IDs array, as this is the expected
return value for `NewLabel`. I update the method's tests accordingly. I
don't think any new tests for `NewLabel` are needed as it should have
the same functionality and return values.
## Manual Testing
On the main branch, I launched my local MySQL with the thread stack size
set to the minimal allowed, and used the API to try and create a new
label with 5,000 hosts attached, and received a 422 response from the
server. Server logs showed:
```
level=error ts=2025-01-28T15:08:20.465401Z component=http user=scott@fleetdm.com method=POST
uri=/api/latest/fleet/labels took=16.610292ms err="get hostnames by identifiers: Error 1436 (HY000): Thread stack
overrun: 111136 bytes used of a 131072 byte stack, and 20000 bytes needed. Use 'mysqld --thread_stack=#' to specify
a bigger stack."
```
On this branch, I kept the same MySQL settings and tried my API request
again and it was successful:
<img width="776" alt="image"
src="https://github.com/user-attachments/assets/c4f0f52b-4d09-457b-8096-4dd3a747b1f4"
/>
## QA
The script I used to create a new manual label with lots of hosts is at:
https://gist.github.com/sgress454/84f12064c437da456c456e25c26d9069
To run it, first grab a bearer token from any API request by opening the
network tab, clicking a Fleet API request, and in the headers tab
scrolling down to Authorization:
<img width="892" alt="image"
src="https://github.com/user-attachments/assets/5680f3bf-8db8-469a-9f03-000b86622c04"
/>
(only take the part _after_ "Bearer")
Then download the script from that gist and in its folder run:
```
NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js <the bearer token> "<a label name>"
```
e.g.
```
NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js U3HpbdtadmJXGKYSB0U/PbwfOpHbBt7FpkWmGKKYolOO1moLNZA6XxP+QO5LVukvAotZ7d+JbNUEEhYHZtxoqg== "some test label"
```
This will invoke the API on https://localhost:8080 and try to add 5000
hosts a new label "some test label".
If you need to change the # of hosts or the url of the server, there are
additional arguments:
```
NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js <the bearer token> "<a label name>" <number of hosts> <url>
```
e.g.
```
NODE_TLS_REJECT_UNAUTHORIZED=0 node ./add_hosts_to_label.js U3HpbdtadmJXGKYSB0U/PbwfOpHbBt7FpkWmGKKYolOO1moLNZA6XxP+QO5LVukvAotZ7d+JbNUEEhYHZtxoqg== "some test label" 10000 https://foo.bar
```
for #25427
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
This PR adds a new `ExcludeFleetMaintainedApps` option to the
ListSoftwareTitles datastore method, and the equivalent
`exclude_fleet_maintained_apps` to the `GET
/api/v1/fleet/software/titles` API.
The new functionality works by doing a left join from the
`software_titles` table to the `fleet_library_apps` table by bundle
identifier, and excluding any rows that are present in the
`fleet_library_apps` table.
New tests verify that the filtering works as expected and doesn't
interfere with other functions of the method.
for #24471
This PR adds missing support for updating a team's failing policy
webhook via GitOps:
* If `team_settings.webhook_settings.failing_policies_webhook` is not
set, the webhook will be disabled and all settings cleared.
* If `team_settings.webhook_settings.failing_policies_webhook` is set,
the webhook will be configured according to the specified settings.
A new test has been added to verify this functionality, and I tested it
locally using the `fleetctl gitops` command.
---------
Co-authored-by: Dante Catalfamo <43040593+dantecatalfamo@users.noreply.github.com>
For #17700
improve error message when a custom profile is edited or uploaded with
one that has the same name.
updates the error message for `POST /fleet/mdm/profiles` and in fleetctl
when using gitops
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
Sarah Gillespie