diff --git a/.github/ISSUE_TEMPLATE/release-qa.md b/.github/ISSUE_TEMPLATE/release-qa.md index 1804d9881c..f89d079e4c 100644 --- a/.github/ISSUE_TEMPLATE/release-qa.md +++ b/.github/ISSUE_TEMPLATE/release-qa.md @@ -7,7 +7,9 @@ assignees: 'xpkoala,andreykizimenko,chrstphr84,Brajim20' --- -# Goal: easy-to-follow test steps for checking a release manually +# Goal + +Easy-to-follow test steps for checking a release manually. # Important reference data @@ -15,147 +17,227 @@ assignees: 'xpkoala,andreykizimenko,chrstphr84,Brajim20' 2. [Permissions documentation](https://fleetdm.com/docs/using-fleet/permissions) 3. [Fleet free vs premium documentation](https://fleetdm.com/pricing) -# Smoke Tests +# Smoke tests + Smoke tests are limited to core functionality and serve as a pre-release final review. If smoke tests are failing, a release cannot proceed. -## Fleet core: +## Fleet core **Fleet version** (Head to the "My account" page in the Fleet UI or run `fleetctl version`): -**Web browser** _(e.g. Chrome 88.0.4324)_: +**Web browser** _(e.g. Chrome 88.0.4324)_: ### Prerequisites -1. Local instance is running and up to date with the target release branch +1. Local instance is running and up to date with the target release branch. 2. In your browser, clear local storage using devtools. ### Orchestration +
| Test name | Step instructions | Expected result | pass/fail | |
|---|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail | |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail | |
| Update flow | -1. remove all fleet processes/agents/etc using `fleetctl preview reset` for a clean slate -2. run `fleetctl preview` with no tag for latest stable -3. create a host/report to later confirm upgrade with -4. STOP fleet-preview-server instances in containers/apps on Docker -5. run `fleetctl preview` with appropriate testing tag -6. Navigate through all new UI flows and confirm dashboard, hosts, controls, queries, policies, and settings pages are working as expected. | All previously created hosts/queries are verified to still exist | pass/fail | |
| Login flow | + | |||
| Update flow | +-1. navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results. -2. navigate to the login page and attempt to login with both valid and invalid sso credentials to verify expected results. - | +1. Remove all fleet processes/agents/etc using `fleetctl preview reset` for a clean slate. +2. Run `fleetctl preview` with no tag for latest stable. +3. Create a host/report to later confirm upgrade with. +4. STOP fleet-preview-server instances in containers/apps on Docker. +5. Run `fleetctl preview` with appropriate testing tag. +6. Navigate through all new UI flows and confirm dashboard, hosts, controls, queries, policies, and settings pages are working as expected. -1. text fields prompt when blank -2. correct error message is "authentication failed" -3. forget password link prompts for email -4. valid credentials result in a successful login. -5. valid sso credentials result in a successful login | pass/fail | |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). | + | +All previously created hosts/queries are verified to still exist. | +pass/fail | +pass/fail | +
| Login flow | +- | |||
| Log destination flow | Verify log destination for software, reports, policies, and packs. | +1. Navigate to the login page and attempt to login with both valid and invalid credentials to verify some combination of expected results. +2. Navigate to the login page and attempt to login with both valid and invalid SSO credentials to verify expected results. -1. Software, report, policy, and packs logs are successfully sent to external log destinations -2. Software, report, policy, and packs logs are successfully sent to Filesystem log destinations - - | pass/fail | - |
| IdP Provisioning (SCIM) | Verify host vitals sync | +1. Text fields prompt when blank. +2. Correct error message is "authentication failed". +3. Forget password link prompts for email. +4. Valid credentials result in a successful login. +5. Valid SSO credentials result in a successful login. + + | +pass/fail | +|
| Packs flow | +Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). | ++ +1. Packs successfully run on host machines after migrations. +2. New Packs can be created. +3. Packs can be edited and deleted. +4. Packs results information is logged. + + | +pass/fail | +|
| Log destination flow | +Verify log destination for software, reports, policies, and packs. | ++ +1. Software, report, policy, and packs logs are successfully sent to external log destinations. +2. Software, report, policy, and packs logs are successfully sent to Filesystem log destinations. + + | +pass/fail | +|
| IdP Provisioning (SCIM) | +Verify host vitals sync. | +1. Configure and verify provisioning with the following IdPs: 1. Okta - 3. Entra - 4. Hydrant/Google -2. Enroll hosts with EUA & IdP Provisioning enabled - 1. MacOS + 2. Entra + 3. Hydrant/Google +2. Enroll hosts with EUA & IdP Provisioning enabled: + 1. macOS 2. Windows 3. Ubuntu 4. iOS/iPadOS 5. Android - - | pass/fail | |
| GitOps and generate-gitops | Verify `fleetctl generate-gitops` and `GitOps` functionality | + | +pass/fail | +|
| GitOps and generate-gitops | +Verify fleetctl generate-gitops and GitOps functionality. |
+1. Generate-gitops from a version-matched fleetctl successfully outputs YAML from a brand new Fleet server (net of auto-populated fleets etc.). 2. Running GitOps either using the `gitops.sh` script directly (from the `fleet-gitops` repo) or by using the GitOps GitHub or GitLab workflow (attempting via one of these three is sufficient) succeeds. - | pass/fail | pass/fail | + -
| Fleet Free | Verify that product group features behave correctly on Fleet Free | + | ||
| Fleet Free | +Verify that product group features behave correctly on Fleet Free. | +Run basic checks for the product group area while using a Fleet Free license. - -- Features documented as Free work normally + +- Features documented as Free work normally: - Packs - - Gitops -- Premium features are correctly restricted or hidden + - GitOps +- Premium features are correctly restricted or hidden: - IdP information -- No UI, API, or workflow errors occur when using Free-only functionality +- No UI, API, or workflow errors occur when using Free-only functionality. Reference: https://fleetdm.com/pricing - | pass/fail | |
| UI / UX | Verify visual consistency and layout integrity across product group areas | - -Perform a quick visual scan of the UI and confirm: + | +pass/fail | +|
| UI / UX | +Verify visual consistency and layout integrity across product group areas. | +- | pass/fail | pass/fail | +
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| MDM enrollment flow | Verify MDM enrollments, run MDM commands | - + + | |
| MDM enrollment flow | +Verify MDM enrollments, run MDM commands. | ++ 1. Erase an ADE-eligible macOS host and verify able to complete automated enrollment flow. 2. With Windows MDM turned On, enroll a Windows host and verify MDM is turned On for the host. 3. Erase an Auto-Pilot enabled Windows host and complete automated enrollment flow. 4. Verify able to run MDM commands on both macOS and Windows hosts from the CLI. - | pass/fail |
| MDM migration flow | Verify MDM migration for ADE and non-ADE hosts | - + | +pass/fail | +
| MDM migration flow | +Verify MDM migration for ADE and non-ADE hosts. | ++ 1. Turn off MDM on an ADE-eligible macOS host and verify that the native, "Device Enrollment" macOS notification appears. 2. On the My device page, follow the "Turn on MDM" instructions and verify that MDM is turned on. 3. Turn off MDM on a non ADE-eligible macOS host. 4. Verify Windows host migrates from 3rd party MDM to Fleet when automatic migration is turned on. - | pass/fail |
| OS settings | Verify OS settings functionality | + | +pass/fail | +
| OS settings | +Verify OS settings functionality. | +1. Verify Profiles upload/download/delete (macOS & Windows). -2. Verify Profiles are delivered to host and applied. - | pass/fail |
| Setup experience | Verify macOS Setup experience | + | +pass/fail | +
| Setup experience | +Verify macOS Setup experience. | +1. Configure End user authentication. -3. Upload a Bootstrap package. -4. Add software (FMA, VPP, & Custom pkg) -5. Add a script -6. Enroll an ADE-eligible macOS host and verify successful authentication. -7. Verify Bootstrap package is delivered. -8. Verify SwiftDialogue window displays. -9. Verify software installs and script runs. - | pass/fail |
| iOS/iPadOS | Verify enrollment, profiles, & software installs | + | +pass/fail | +
| iOS/iPadOS | +Verify enrollment, profiles, & software installs. | +1. Verify ADE enrollment. 2. Verify BYOD OTA enrollment. @@ -163,220 +245,322 @@ Perform a quick visual scan of the UI and confirm: 4. Verify Profiles are delivered to host and applied. 5. Verify VPP apps install & display correctly in Activity feed. 6. Verify `Turn Off MDM` for BYOD & ADE hosts. - - | pass/fail |
| Android | Verify enrollment, profiles, & software installs | + | +pass/fail | +
| Android | +Verify enrollment, profiles, & software installs. | +1. Verify BYOD enrollment. 2. Verify Profiles are delivered to host and applied. 3. Verify apps install. -4. Verify certificate delivery +4. Verify certificate delivery. 5. Verify `Unenroll`. - - | pass/fail |
| Token & Certificate Renewals | APNs cert and ABM token renewal workflow | + | +pass/fail | +
| Token & Certificate Renewals | +APNs cert and ABM token renewal workflow. | +1. Renew APNs Certificate. 2. Renew ABM Token. 3. Ensure ADE hosts can enroll. - | pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free | + | +pass/fail | +
| Fleet Free | +Verify that product group features behave correctly on Fleet Free. | +Run basic checks for the product group area while using a Fleet Free license. - -- Features documented as Free work normally + +- Features documented as Free work normally: - Host enrollment - Apple, Windows, Android MDM - - Configuration profile delivery - - APNs Certificate renewal -- Premium features are correctly restricted or hidden + - Configuration profile delivery + - APNs Certificate renewal +- Premium features are correctly restricted or hidden: - Setup experience -- No UI, API, or workflow errors occur when using Free-only functionality +- No UI, API, or workflow errors occur when using Free-only functionality. Reference: https://fleetdm.com/pricing - | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas | - -Perform a quick visual scan of the UI and confirm: + | +pass/fail | +
| UI / UX | +Verify visual consistency and layout integrity across product group areas. | +- | pass/fail | pass/fail | +
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Report flow | Create, edit, run, and delete reports. | -1. permissions regarding creating/editing/deleting reports are up to date with documentation -2. syntax errors result in error messaging -3. queries can be run manually - | pass/fail |
| Host Flow | Verify a new host can be added and removed following modal instructions using your own device. | + | |
| Report flow | +Create, edit, run, and delete reports. | +-1. Host is added via command line -2. Host serial number and date added are accurate -3. Host is not visible after it is deleted -4. Warning and informational modals show when expected and make sense - | pass/fail |
| My device page | Verify the end user's my device page loads successfully. | +1. Permissions regarding creating/editing/deleting reports are up to date with documentation. +2. Syntax errors result in error messaging. +3. Queries can be run manually. -1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. -2. The "My device" page is populated correctly and as expected. + | +pass/fail | +
| Host flow | +Verify a new host can be added and removed following modal instructions using your own device. | ++ +1. Host is added via command line. +2. Host serial number and date added are accurate. +3. Host is not visible after it is deleted. +4. Warning and informational modals show when expected and make sense. + + | +pass/fail | +
| My device page | +Verify the end user's My device page loads successfully. | ++ +1. Clicking the Fleet desktop item, then "My device" successfully loads the My device page. +2. The "My device" page is populated correctly and as expected. 3. Styling and padding appears correct. - - | pass/fail |
| Scripts | Verify script library and execution | + + | +pass/fail | +
| Scripts | +Verify script library and execution. | +1. Verify able to run a script on all host types from CLI. 2. Verify scripts library upload/download/delete. 3. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. 4. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. 5. Verify UI loading state and statuses for scripts. -8. Disable scripts globally and verify unable to run. -9. Verify scripts display correctly in Activity feed. - | pass/fail |
| Software | Verify software library and install / download | + | +pass/fail | +
| Software | +Verify software library and install / download. | +1. Verify software library upload/download/delete. 2. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. 3. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. 4. Verify UI loading state and statuses for installing software. -7. Verify software installs display correctly in Activity feed. - | pass/fail |
| Fleet Free | Verify that product group features behave correctly on Fleet Free | + | +pass/fail | +
| Fleet Free | +Verify that product group features behave correctly on Fleet Free. | +Run basic checks for the product group area while using a Fleet Free license. - -- Features documented as Free work normally + +- Features documented as Free work normally: - Host details page - Reports (Add, edit, live report) - Software inventory - Scripts (Add, delete, run) - My device page (Mac, Windows, Linux) -- Premium features are correctly restricted or hidden +- Premium features are correctly restricted or hidden: - Add software -- No UI, API, or workflow errors occur when using Free-only functionality +- No UI, API, or workflow errors occur when using Free-only functionality. Reference: https://fleetdm.com/pricing - | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas | - -Perform a quick visual scan of the UI and confirm: + | +pass/fail | +
| UI / UX | +Verify visual consistency and layout integrity across product group areas. | +- | pass/fail | pass/fail | +
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Disk encryption | Verify disk encryption functionality | + | |
| Disk encryption | +Verify disk encryption functionality. | +1. Verify able to configure Disk encryption (macOS, Windows, & Linux). 2. Verify host enrolled with Disk encryption enforced successfully encrypts. - | pass/fail |
| Vulnerabilities | Verify that software vulnerabilities are correctly populated | + | +pass/fail | +pass/fail | +
| Vulnerabilities | +Verify that software vulnerabilities are correctly populated. | +- | |
| Certificate Authorities | Verify setup and certificate delivery | +1. Verify that known vulnerable software items display expected CVEs and severity information in the Software tab. +2. Verify that individual vulnerabilities can be previewed and open the correct NVD page when selected. +3. Verify that vulnerable software appears under "My device > Software" for affected hosts with expected CVEs. + + | +pass/fail | +
| Certificate Authorities | +Verify setup and certificate delivery. | +1. Configure and verify that certificates deploy to hosts with the following CAs: 1. DigiCert 2. NDES 3. SmallStep - - | pass/fail |
| OS updates | Verify OS updates flow | + | +pass/fail | +
| OS updates | +Verify OS updates flow. | +1. Configure OS updates (macOS & Windows). 2. Verify enforce minimumOS occurs during enrollment (macOS 14+). - | pass/fail |
| Lock & Wipe | Verify hosts can be locked & wiped | + | +pass/fail | +pass/fail | +
| Lock & Wipe | +Verify hosts can be locked & wiped. | +- | |
| Fleet Free | Verify that product group features behave correctly on Fleet Free | +1. Verify locking a host from the Fleet UI (macOS, Windows, & Linux). +2. Verify unlocking a host from the Fleet UI (macOS, Windows, & Linux). +3. Verify wiping a host from the Fleet UI (macOS, Windows, & Linux). +4. Verify wiping and locking hosts using `fleetctl` (macOS, Windows, & Linux). + + | +pass/fail | +
| Fleet Free | +Verify that product group features behave correctly on Fleet Free. | +Run basic checks for the product group area while using a Fleet Free license. - -- Features documented as Free work normally + +- Features documented as Free work normally: - Vulnerability detection - Individual CVE page -- Premium features are correctly restricted or hidden +- Premium features are correctly restricted or hidden: - Disk encryption - OS Updates - Lock / Wipe - Certificate authorities -- No UI, API, or workflow errors occur when using Free-only functionality +- No UI, API, or workflow errors occur when using Free-only functionality. Reference: https://fleetdm.com/pricing - | pass/fail |
| UI / UX | Verify visual consistency and layout integrity across product group areas | - -Perform a quick visual scan of the UI and confirm: + | +pass/fail | +
| UI / UX | +Verify visual consistency and layout integrity across product group areas. | +- | pass/fail | pass/fail | +
| Test name | Step instructions | Expected result | Pass/Fail | ||||||
|---|---|---|---|---|---|---|---|---|---|
| $Name | -{what a tester should do} | -{what a tester should see when they do that} | -pass/fail | -||||||
| Test name | Step instructions | Expected result | pass/fail | ||||||
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail | ||||||
| Release blockers | Verify there are no outstanding release blocking tickets. | -1. Check [this](https://github.com/fleetdm/fleet/labels/~release%20blocker) filter to view all open `~release blocker` tickets. +1. Check [this](https://github.com/fleetdm/fleet/labels/~release%20blocker) filter to view all open `~release blocker` tickets. 2. If any are found raise an alarm in the `#help-engineering` and `#g-mdm` (or `#g-endpoint-ops`) channels. | @@ -388,10 +572,10 @@ Perform a quick visual scan of the UI and confirm:Verify all load test metrics are within acceptable range on final build of RC. | -1. Check [this Google doc](https://docs.google.com/document/d/1V6QtFzcGDsLnn2PIvGin74DAxdAN_3likjxSssOMMQI/edit?tab=t.0#heading=h.15acjob4ji20) to review load test key metrics and checks. -2. After all expected changes have been merged to the RC branch, two load tests will need to be run - a new instance with no data, and a migrated instance. -3. For the new instance with no data, set up a load test environment using the RC branch and allow it at least 24hrs of run time. -4. For the migrated instance, set up a load test environment on the previous minor release branch. Once the environment has been set up and stabilized, follow the instructions in [Deploying code changes to fleet](https://github.com/fleetdm/fleet/blob/main/infrastructure/loadtesting/terraform/readme.md#deploying-code-changes-to-fleet) to migrate to the RC branch. Monitor the metrics post-migration to determine if any performance issues arise. +1. Check [this Google doc](https://docs.google.com/document/d/1V6QtFzcGDsLnn2PIvGin74DAxdAN_3likjxSssOMMQI/edit?tab=t.0#heading=h.15acjob4ji20) to review load test key metrics and checks. +2. After all expected changes have been merged to the RC branch, two load tests will need to be run - a new instance with no data, and a migrated instance. +3. For the new instance with no data, set up a load test environment using the RC branch and allow it at least 24hrs of run time. +4. For the migrated instance, set up a load test environment on the previous minor release branch. Once the environment has been set up and stabilized, follow the instructions in [Deploying code changes to fleet](https://github.com/fleetdm/fleet/blob/main/infrastructure/loadtesting/terraform/readme.md#deploying-code-changes-to-fleet) to migrate to the RC branch. Monitor the metrics post-migration to determine if any performance issues arise. 5. Record metrics in [this spreadsheet](https://docs.google.com/spreadsheets/d/1FOF0ykFVoZ7DJSTfrveip0olfyRQsY9oT1uXCCZmuKc/edit?usp=drive_link) for the two load test runs. | @@ -399,17 +583,17 @@ Perform a quick visual scan of the UI and confirm:|||||
| Migration Test | +Migration test | Verify Fleet can migrate to the next version with no issues. | -Using [this github action](https://github.com/fleetdm/fleet/actions/workflows/db-upgrade-test.yml) +Using [this GitHub action](https://github.com/fleetdm/fleet/actions/workflows/db-upgrade-test.yml): -1. Using the most recent stable version of Fleet and `main`, click `Run workflow` -2. Enter the Docker tag of Fleet starting version, e.g. `v4.64.2` -3. Enter the Docker tag of Fleet version to upgrade to, e.g. `rc-minor-fleet-v4.65.0` -4. Click `Run workflow` -5. Action should complete successfully +1. Using the most recent stable version of Fleet and `main`, click `Run workflow`. +2. Enter the Docker tag of Fleet starting version, e.g. `v4.64.2`. +3. Enter the Docker tag of Fleet version to upgrade to, e.g. `rc-minor-fleet-v4.65.0`. +4. Click `Run workflow`. +5. Action should complete successfully. | pass/fail | @@ -420,11 +604,11 @@ Using [this github action](https://github.com/fleetdm/fleet/actions/workflows/dbVerify Fleet can migrate when using real world data. |
-Using [this github action](https://github.com/fleetdm/confidential/actions/workflows/cloud-tests.yml)
+Using [this GitHub action](https://github.com/fleetdm/confidential/actions/workflows/cloud-tests.yml):
-1. Enter `fleetdm/fleet:rc-minor-fleet- |
pass/fail | @@ -435,7 +619,7 @@ Using [this github action](https://github.com/fleetdm/confidential/actions/workfVerify the latest Trivy scan of the RC image has no new high/critical vulnerabilities. | -1. Using [this GitHub Actions](https://github.com/fleetdm/fleet/actions/workflows/trivy-scan.yml). +1. Using [this GitHub action](https://github.com/fleetdm/fleet/actions/workflows/trivy-scan.yml). 2. Review the scan results for any new high or critical severity vulnerabilities introduced by this RC. 3. Attach a screenshot of the latest Trivy scan results to this issue as a comment. 4. If new high/critical vulnerabilities are found, raise an alarm in the appropriate channels. @@ -446,11 +630,11 @@ Using [this github action](https://github.com/fleetdm/confidential/actions/workf |
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| `fleetd` local testing | + +|||
fleetd local testing |
-1. Following [Testing TUF]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/README.md)) instructions create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install on macOS, Linux, and Windows hosts. IMPORTANT: Reminder to use an RC branch and not `main` + +1. Following [Testing TUF](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/README.md) instructions, create binaries for Mac, Windows, and Ubuntu using your local TUF repository and install on macOS, Linux, and Windows hosts. + +> IMPORTANT: Reminder to use an RC branch and not `main`. + |
-1. Confirm the hosts install with the updated version and are working correctly. -2. Confirm any new features and/or bug fixes associated with this release are working as intended. + +1. Confirm the hosts install with the updated version and are working correctly. +2. Confirm any new features and/or bug fixes associated with this release are working as intended. + |
-pass/fail | `fleetd` auto-update tests | +pass/fail | + + +
fleetd auto-update tests |
-1. Conduct the [`fleetd` auto-update n+1 test]([url](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md)) -2. QA certifies new release by commenting in issue. + +1. Conduct the [`fleetd` auto-update n+1 test](https://github.com/fleetdm/fleet/blob/main/tools/tuf/test/Fleetd-auto-update-test-guide.md). +2. QA certifies new release by commenting in issue. + |
-1. Agent successfully auto-updates. -2. Issue is certified by QA. + +1. Agent successfully auto-updates. +2. Issue is certified by QA. + |
-pass/fail | `fleetd` tests | +pass/fail | + + +
fleetd edge tests |
-1. Set up a host in your instance to receive updates from the `edge` channels. -2. Work with engineer leading the release to push changes to the `edge` channel. + +1. Set up a host in your instance to receive updates from the `edge` channels. +2. Work with engineer leading the release to push changes to the `edge` channel. + |
-1. Confirm the hosts running on the edge channel receive the update and are working correctly. -2. Confirm any new features and/or bug fixes associated with this release are working as intended. + +1. Confirm the hosts running on the edge channel receive the update and are working correctly. +2. Confirm any new features and/or bug fixes associated with this release are working as intended. + |
-pass/fail | pass/fail | +
| Test name | Step instructions | Expected result | pass/fail |
|---|---|---|---|
| Test name | Step instructions | Expected result | pass/fail |
| $Name | {what a tester should do} | {what a tester should see when they do that} | pass/fail |
| Report flow | Run reports. | -1. Queries can be run manually - | pass/fail |
| Report flow | +Run reports. | +- | |
| Host Flow | Verify a new host can be added using your own device. |
-1. Hosts can enroll and report correct version of `fleetd` (orbit, osquery, desktop). +1. Queries can be run manually. + + |
+pass/fail | +
| Host flow | +Verify a new host can be added using your own device. | ++ +1. Hosts can enroll and report correct version of `fleetd` (orbit, osquery, desktop). 2. Refetching host vitals completes and returns updated information. - | pass/fail |
| My device page | Verify the end user's my device page loads successfully. |
-1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. -2. The "My device" page is populated correctly and as expected. -3. Styling and padding appears correct. - | pass/fail | pass/fail | + -
| Scripts | Verify script execution |
-1. Verify able to run a script on all host types from CLI. -2. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. -3. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. + | |
| My device page | +Verify the end user's My device page loads successfully. | ++ +1. Clicking the Fleet desktop item, then "My device" successfully loads the My device page. +2. The "My device" page is populated correctly and as expected. +3. Styling and padding appears correct. + + | +pass/fail | +
| Scripts | +Verify script execution. | ++ +1. Verify able to run a script on all host types from CLI. +2. From Host details (macOS, Windows, & Linux) run a script that should PASS, verify. +3. From Host details (macOS, Windows, & Linux) run a script that should FAIL, verify. 4. Verify script results display correctly in Activity feed. - | pass/fail |
| Software | Verify software install / download |
-1. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. -2. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. + |
+pass/fail | +
| Software | +Verify software install / download. | ++ +1. From Host details (macOS, Windows, & Linux) run an install that should PASS, verify. +2. From My Device (macOS, Windows, & Linux) software tab should have self-service items available, verify. 3. Verify software installs display correctly in Activity feed. - | pass/fail |
| OS settings | Verify OS settings functionality |
-1. Verify able to configure Disk encryption (macOS, Windows, & Linux). + |
+pass/fail | +
| OS settings | +Verify OS settings functionality. | ++ +1. Verify able to configure Disk encryption (macOS, Windows, & Linux). 2. Verify host enrolled with Disk encryption enforced successfully encrypts. - | pass/fail |
| Packs flow | Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). |
-1. Packs successfully run on host machines after migrations -2. New Packs can be created. -3. Packs can be edited and deleted -4. Packs results information is logged - | pass/fail | pass/fail | + + +
| Packs flow | +Verify management, operation, and logging of ["2017 packs"](https://fleetdm.com/handbook/company/why-this-way#why-does-fleet-support-query-packs). | ++ +1. Packs successfully run on host machines after migrations. +2. New Packs can be created. +3. Packs can be edited and deleted. +4. Packs results information is logged. + + | +pass/fail | +
| Fleet Free | +Verify that Fleet Desktop works on Fleet Free. | +- | |
| Fleet Free | Verify that Fleet Desktop works on Fleet Free. |
After repointing a Fleet Desktop install at a server running Fleet Free:
-1. Clicking the Fleet desktop item, then "My device" successfully loads the my device page. -2. The "My device" page is populated correctly and as expected. -3. Styling and padding appears correct. - | pass/fail |
| Auto-updates disabled | Verify -that fleetd works on when the installer package is built with `--disable-updates`. | + | +pass/fail | +
| Auto-updates disabled | +Verify that fleetd works when the installer package is built with --disable-updates. |
++ +1. Generate package with `fleetctl package [...] --updates-disabled`. +2. Install packages on macOS, Windows, and Linux. 3. Smoke test orbit and Fleet Desktop functionality, and osquery tables. - | pass/fail | pass/fail | +