I tested the uninstall script by:
- Making a new agent package and installing it
- Checking with `dpkg --get-selections | grep 'fleet'` that
fleet-osquery is installed
- Checking with `sudo systemctl list-units | grep 'orbit'` that
orbit.service is running
- Uninstalling the package with uninstall-fleetd-linux.sh
- Checking the above commands again to see that fleet-osquery and
orbit.service are uninstalled.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- For Orbit and Fleet Desktop changes:
- [x] Manual QA done on one Linux machine (Ubuntu 24 on HP laptop).
## Issue
Closes#29893
## Description
- Update text for VPP command for pending_install and failed_install to
include verification nuance
- Add related tests
## Note
- Original PR pointing to `vpp-verify-followup` but should be repointed
to `main` once that branch is merged in
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [ ] Manual QA for all new/changed functionality
---------
Co-authored-by: Ian Littman <iansltx@gmail.com>
relates to [#28691](https://github.com/fleetdm/fleet/issues/28691)
This adds the ability to upload the EULA users see during the setup
experience via gitops. It follows patterns used for uploading the
bootstrap package via gitops.
I've also added a sha256 column to the `eulas` table in order to easily
compare the existing eula with a new one to see if we need to perform an
upload.
Finally I added the support to generate this new gitops setting with the
`generate-gitops` command
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [x] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [x] Verified that the setting is exported via `fleetctl
generate-gitops`
- [x] Added the setting to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [x] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [x] Verified that any relevant UI is disabled when GitOps mode is
enabled
- For database migrations:
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> Fixes#29851
> Fixes#29902
> Mainly followups from https://github.com/fleetdm/fleet/pull/30295,
plus improved integration testing
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
This PR specifies a binary identifier for `fleetctl` on macOS, which
resolves the codesignature testing issue from #30352.
# Tests
To test this, I unsigned an affected version of `fleetctl`:
```shell
codesign --remove-signature fleetctl
```
I then installed `rcodesign` 0.29.0, and signed the binary myself, with
the added `--binary-identifier` flag:
```shell
./rcodesign sign --p12-file Certificates.p12 --p12-password-file=.p12_password --for-notarization --binary-identifier com.fleetdm.fleetctl fleetctl
```
Then, I obtained the codesigning requirement from my newly signed
binary:
```shell
$ codesign -d -r- fleetctl
Executable=/Users/jacob.burley/Downloads/fleetctl_v4.67.3_macos/fleetctl
designated => identifier "com.fleetdm.fleetctl" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = "XXXXXXXXXX"
```
I then tested the code signature with the designated requirement given:
```shell
$ codesign --test-requirement='=identifier "com.fleetdm.fleetctl" and anchor apple generic and certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = "XXXXXXXXXX"' --verbose=2 --verify fleetctl
fleetctl: valid on disk
fleetctl: satisfies its Designated Requirement
fleetctl: explicit requirement satisfied
```
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
#29482
[Migrate to the AWS SDK for Go
v2](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html)
documents how to migrate codebases.
QA on features that use AWS SDK Go:
- Bootstrap package:
- upload: ✅
- download: ✅
- cleanup: ✅
- Software (upload, download, installation, etc.) ✅
- Cloudfront: Luckly, this feature was already using aws-sdk-go-v2.
- Carves ✅
- Logging:
- Firehose ✅
- Kinesis ✅
- Lambda ✅ (tested result logs to a lambda function on our AWS Dogfood
account)
- Email:
- Amazon SES TODO ⚠️ (this is what Dogfood uses and a few customers)
- We cannot easily test locally, we can use dogfood or load testing
(AWS) environments.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
# Details
This PR adds on to the https://github.com/fleetdm/fleet/pull/30278 which
added support for host vitals labels, by adding a cron job which updates
host vitals label membership every 5 minutes.
Unlike "dynamic" label types, where the hosts determine membership
themselves and report their decision to Fleet when they check in, "host
vitals" label membership is determine by Fleet. This means they can be
applied to hosts which don't check in at the `/distributed/write`
endpoint (like mobile devices).
The mechanism in the cron job is pretty naïve, it just lists all the
labels, post-filters for "host vitals" labels and updates membership for
each. Since the # of labels on an instance tends not to be excessive,
and since updating membership consists of one `DELETE` query and one
`INSERT...SELECT` query, this is not expected to contribute significant
load, but load testing should verify this.
# Checklist for submitter
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
# Testing
Tested by manually adding scim groups, users etc. in the db and adding a
couple of new host vitals labels using the API. I've uploaded a folder
containing a db snapshot and creds to [Google Drive](
https://drive.google.com/drive/folders/1pDlg2XtS139d3sxq9iFqFs6vez8LeUgg?usp=sharing).
To use it, create a new folder
`~/.fleet/snapshots/test_host_vitals_labels`, download the `db.sql.gz`
file into it, then do `fdm restore --prep` and select
"test_host_vitals_labels". After starting the server you can trigger the
new job using `fleetctl trigger --name host_vitals_label_membership` or
wait five minutes.
New automated tests were added for a small change to the `GetLabels()`
method, and for the new cron job. Tests for other functionality were
added in https://github.com/fleetdm/fleet/pull/30278.
For #29609.
From @marko-lisica:
> In #23900 we decided to error if $FLEET_VAR_HOST_END_USER_IDP_GROUPS
or $FLEET_VAR_HOST_END_USER_IDP_USERNAME is added to a profile and the
host doesn't have a username or groups assigned to it. We decided that
because we think that if user deploys profile with variables above then
user would expected that host have some value for those. I think we
should be consistent with departments.
Context:
https://fleetdm.slack.com/archives/C084F4MKYSJ/p1751281745736999?thread_ts=1751025745.430259&cid=C084F4MKYSJ.
# Details
This PR adds support for a new label membership type, `host_vitals`.
Membership for these labels is based on a database query created from
user-supplied criteria. In this first iteration, the allowed criteria
are very simple: a label can specify either an IdP group or IdP
department, and hosts with linked users with a matching group or
department.
Groundwork is laid here for more complex host vitals queries, including
`and` and `or` logic, different data types and different kinds of vitals
(rather than just the "foreign" vitals of which IdP is an example).
Note that this PR does _not_ include the cron job that will trigger
membership updating, and it doesn't include ; for sake of simplicity in
review that will be done in a follow-on PR.
## Basic flow
### Creating a host vitals label
1. A new label is created via the API / GitOps with membership type
`host_vitals` and a `criteria` property that's a JSON blob. Currently
the JSON can only contain `vital` and `value` keys (and must contain
those keys)
2. The server validates that the specified `vital` exists in our [set of
known host
vitals](https://github.com/fleetdm/fleet/pull/30278/files#diff-b6d4c48f2624b82c2567b2b88db1de51c6b152eeb261d40acfd5b63a890839b7R418-R436).
3. The server validates that the [criteria can be parsed into a
query](https://github.com/fleetdm/fleet/pull/30278/files?diff=unified&w=1#diff-4ac4cfba8bed490e8ef125a0556f5417156f805017bfe93c6e2c61aa94ba8a8cR81-R86).
This also happens during GitOps dry run.
4. The label is saved (criteria is saved as JSON in the db)
### Updating membership for a host vitals label
1. The label's criteria is used to generate a query to run on the
_Fleet_ db.
1. For each vital criteria, check the vital type. Currently only foreign
vitals are supported.
2. For foreign vitals, add its group to a set we keep track of.
3. Add a `WHERE` clause section for the vital and value, e.g.
`end_user_idp_groups = ?`
4. Once we have all the `WHERE` clauses, create the query as `SELECT %s
FROM %s` + any joins contributed by foreign vitals groups + `WHERE ` +
all the `WHERE` clauses we just calculated. The `%s` provide some
flexibility if we want to use these queries in other contexts.
2. Delete all existing label members
3. Do an `INSERT...SELECT` using the query we calculated from the label
criteria. The query will be `SELECT <label id> as label_id, hosts.id
FROM hosts JOIN ...`
## Future work
### Domestic vitals
These can be anything that we already store in the `hosts` table.
Domestic vitals won't add any `JOIN`s to the calculated label query, and
will simply be e.g. `hosts.hostname = ?`
### Custom vitals
We currently support an `additional_queries` config that will cause
other queries to run on hosts. The data returned from these queries is
stored in a `hosts_additional` table as a JSON blob. We can use MySQL
JSON functions to match values in this data, e.g.
`JSON_EXTRACT(host_additional, `$.some_custom_vital`) = ?`
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
> I'll add the changelog item when I add the cron job PR
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For new Fleet configuration settings
- [X] Verified that the setting can be managed via GitOps, or confirmed
that the setting is explicitly being excluded from GitOps. If managing
via Gitops:
- [X] Verified that the setting is exported via `fleetctl
generate-gitops`
- [X] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
Updated Kubernetes deploy guide
- Now includes new Kubernetes deployment instructions using Helm and
Terraform
- Added details for MySQL and Redis connection information when deployed
with Helm
- Added details for adding your Fleet Premium license
- Added details for enabling TLS
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
#29609
Verified the changes with [Entra ID's
validator](https://scimvalidator.microsoft.com/) and adding the
department attribute to the tester:
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 32 AM"
src="https://github.com/user-attachments/assets/45a5deb8-7c65-49df-b3e8-eb05bea11f6b"
/>
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 21 AM"
src="https://github.com/user-attachments/assets/91b554b5-b0b9-4bb6-a0cf-4e3b40e6ce21"
/>
- Tested with Okta
- TODO: Test with Entra ID and Google Workspace.
- I decided to not fail profile deployment if a user has no department
because it's not a required attribute, instead the
`FLEET_VAR_HOST_END_USER_IDP_DEPARTMENT` will be replaced with the empty
string.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
(https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
jkatz01