Scott Gress
c3b82539a5
<!-- Add the related story/sub-task/bug number, like Resolves #123, or remove if NA --> **Related issue:** For #44077 # Details * Adds `historical_data` key to app and team config (and gitops) with `uptime` and `vulnerabilities` subkeys. Keys default to `true`, meaning "collect this data" * Adds `enabled_historical_dataset` and `disabled_historical_dataset` activities when these values are flipped via GitOps or the config APIs The majority of the file changes in here are GitOps test files that need to be updated to have the new config in them. **This PR does _not_ implement using these configs to actually disable data collection or purge data; that will come in a follow-up PR (as well as the front-end)** # Checklist for submitter If some of the following don't apply, delete the relevant line. - [ ] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. n/a, unreleased ## Testing - [X] Added/updated automated tests - [ ] QA'd all new/changed functionality manually #### Defaults - [X] Fresh install: `GET /api/v1/fleet/config` returns `features.historical_data.uptime: true` and `features.historical_data.vulnerabilities: true` - [X] Created a new fleet via `POST /api/v1/fleet/teams`, then `GET /api/v1/fleet/fleets/{id}` returns `features.historical_data.uptime: true` and `features.historical_data.vulnerabilities: true` #### Global PATCH (`POST /api/v1/fleet/config`) - [X] PATCHed `{"features": {"historical_data": {"vulnerabilities": false}}}` — `vulnerabilities` flipped to `false`, `uptime` unchanged at `true` - [X] PATCHed `{"features": {"historical_data": {"uptime": false, "vulnerabilities": true}}}` — both values applied as sent - [X] PATCHed `{"features": {"historical_data": {"vulnerabilites": false}}}` (typo in sub-key) — request rejected with 4xx, stored config unchanged #### Fleet PATCH (`PATCH /api/v1/fleet/fleets/{id}`) - [X] PATCHed a fleet with `{"features": {"historical_data": {"uptime": false}}}` — fleet's `uptime` flipped to `false`, `vulnerabilities` unchanged - [X] Subsequent `GET /api/v1/fleet/fleets/{id}` returns the toggled values under `features.historical_data` (storage shape is symmetric with global) - [X] PATCHed a fleet with `{"features": {"enable_host_users": false}}` (a non-`historical_data` features sub-field) — request returned 200 but the fleet's `enable_host_users` is unchanged (silently ignored, per existing endpoint convention) #### GitOps — global (`fleetctl gitops -f global.yml`) - [X] Applied a YAML with `features.historical_data: {uptime: true, vulnerabilities: false}` — `vulnerabilities` is `false` after apply, `uptime` is `true` - [X] Applied a YAML whose `org_settings` omits `features` entirely — both sub-keys are `true` after apply (defaults injected even if previously disabled) - [X] Applied a YAML where `historical_data` only contains `uptime: false` — `uptime: false` is honored, `vulnerabilities` defaults to `true` - [X] Disabled `vulnerabilities` via the API, then ran `fleetctl gitops` with a YAML that doesn't pin it — `vulnerabilities` flips back to `true` (this is intentional; gitops is the source of truth) #### GitOps — fleet - [X] Applied a fleet YAML with `features.historical_data: {uptime: false}` — that fleet has `uptime: false`, `vulnerabilities: true` after apply - [X] Applied a fleet YAML whose `team_settings.features` omits `historical_data` — both sub-keys are `true` after apply - [X] Applied a fleet YAML that omits `features` entirely — both sub-keys are `true` after apply #### `fleetctl apply` (legacy, partial-merge) - [ ] Disabled `vulnerabilities` via the API, then ran `fleetctl apply` with a YAML that doesn't mention `historical_data` — `vulnerabilities` is still `false` (apply leaves omitted fields alone) #### Activities — global - [X] After PATCHing global to disable `vulnerabilities`, the latest activity is `disabled_historical_dataset` with payload `{"dataset": "vulnerabilities", "fleet_id": null, "fleet_name": null}` - [X] After PATCHing global with both sub-keys flipping in one request, two activities are emitted (one per sub-key) - [X] After PATCHing global with the same values that are already stored, zero new activities are emitted - [X] After re-enabling a previously disabled dataset, the activity type is `enabled_historical_dataset` #### Activities — per fleet - [X] After PATCHing fleet `workstations` to disable `uptime`, the activity is `disabled_historical_dataset` with payload `{"dataset": "uptime", "fleet_id": <workstations id>, "fleet_name": "workstations"}` - [X] Toggling the same dataset on two different fleets produces two distinct activities, one per fleet - [X] After a fleet PATCH with the same values already stored, zero new activities are emitted For unreleased bug fixes in a release candidate, one of: - [X] Confirmed that the fix is not expected to adversely impact load test results - [ ] Alerted the release DRI if additional load testing is needed ## New Fleet configuration settings - [ ] Setting(s) is/are explicitly excluded from GitOps If you didn't check the box above, follow this checklist for GitOps-enabled settings: - [X] Verified that the setting is exported via `fleetctl generate-gitops` - [X] Verified the setting is documented in a separate PR to [the GitOps documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485) - https://github.com/fleetdm/fleet/pull/44703 - [X] Verified that the setting is cleared on the server if it is not supplied in a YAML file (or that it is documented as being optional) - [X] Verified that any relevant UI is disabled when GitOps mode is enabled <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Historical-data controls: per-org and per-team toggles for uptime and vulnerability time‑series, with defaults applied when keys are omitted and enable/disable activities emitted on changes. * **Bug Fixes** * Partial updates and PATCH/GitOps flows preserve unspecified historical-data sub-keys instead of clearing them. * **Tests** * Expanded unit and integration tests covering defaults, partial PATCH/GitOps behavior, idempotency, and activity emission. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Github Actions
Fleet uses Github Actions for continuous integration (CI). This document describes best practices and at patterns for writing and maintaining Fleet's Github Actions workflows.
Bash
By default, Github Actions sets the shell to bash -e for linux and MacOS runners. To help write
safer bash scripts in run jobs and avoid common issues, override the default by adding the following
to the workflow file
defaults:
run:
# fail-fast using bash -eo pipefail. See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference
shell: bash
By specifying the default shell to bash, some extra flags are set. The option pipefail changes
the behaviour when using the pipe | operator such that if any command in a pipeline fails, that
commands return code will be used a the return code for the whole pipeline. Consider the following
example in test-go.yaml
- name: Run Go Tests
run: |
# omitted ...
make test-go 2>&1 | tee /tmp/gotest.log
If the pipefail option was not set, this job would always succeed because tee would always
return success. This is not the intended behavior. Instead, we want the job to fail if make test-go fails.
Concurrency
Github Action runners are limited. If a lot of workflows are queued, they will wait in pending until a runner becomes available. This has caused issue in the past where workflows take an excessively long time to start. To help with this issue, use the following in workflows
# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id}}
cancel-in-progress: true
When a workflow is triggered via a pull request, it will cancel previous running workflows for that
pull request. This is especially useful when changes are pushed to a pull request frequently.
Manually triggered workflows, workflows that run on a schedule, and workflows triggered by pushes to
main are unaffected.