Updating actions/setup-go to v6.3.0 from a mix of different versions.
This gets us faster CI runs, with improvements such as:
- built in Go module cache AND Go build cache (separate cache no longer
needed)
- using go.mod resulting in fewer cache invalidations
- faster Node 24 runtime
- using go.dev download URL, which is more reliable
Fixes#40975.
8.0.32 (was running in Aurora managed cloud at the time) -> 8.0.39 (what
we're running now) 8.0.36 -> 8.0.44 (latest 8.0.x version supported by
Aurora; holding off on 8.0.45 until Aurora supports it) 8.4.7 -> 8.4.8
9.5.0 -> 9.6.0
Also bumped the supported Aurora version from 3.07.0 to 3.08.2 to match
what we're running in managed cloud right now
Fleet might work on older patch versions but we'll no longer dev/test on
them. MySQL 9.x not testing previous minor versions matches with our
previous approach for that version.
Since these are all patch/minor bumps (and the overnight build cases are
patch bumps/are covered by AWS envs) automated testing should be
sufficient here.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38538
This PR is just refactoring GitHub workflows. No significant functional
differences.
In this PR, we create a reusable workflow for running a single Go test
suite. This eliminates/reduces the complex and hard to maintain strategy
matrix from the original job.
This is pre-work before splitting off activity bounded context tests
into their own job.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Upgraded security protections across build and deployment workflows
for enhanced runner environment hardening.
* Strengthened CI/CD infrastructure security measures throughout
automated processes.
* No direct user-facing changes.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is to avoid a similar issue that we just faced in the future.
The issue was this PR https://github.com/fleetdm/fleet/pull/37118 was
merged, but all checks passed since it was a html file only PR, go tests
did not run. So we did not catch the integration tests asserting content
in these files was broken.
The following PR fixed it: https://github.com/fleetdm/fleet/pull/37196
This PR proposes to add those two files to GO test triggers in CI to
avoid breaking the tests accidentally in the future.
Resolves#36909.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
for #31321
# Details
Small updates from [community
PR](https://github.com/fleetdm/fleet/pull/31134):
* Updated config vars to match
[docs](https://github.com/fleetdm/fleet/blob/docs-v4.75.0/docs/Configuration/fleet-server-configuration.md#server_private_key_region)
* Added support for specifying region in config (already documented)
* Removed parsing of ARN for region
* Made retry backoff intervals a bit longer
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
(already added in the community PR
[here](https://github.com/fleetdm/fleet/blob/sgress454/updates-for-private-key-in-aws-sm/changes/private-key-secrets-manager#L0-L1)
## Testing
- [X] Added/updated automated tests
- [X] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Added support for specifying the AWS region for server private key
retrieval from AWS Secrets Manager via server.private_key_region.
- Chores
- Renamed configuration keys:
- server.private_key_secret_arn → server.private_key_arn
- server.private_key_secret_sts_assume_role_arn →
server.private_key_sts_assume_role_arn
- server.private_key_secret_sts_external_id →
server.private_key_sts_external_id
- Update your configuration to use the new keys.
- Adjusted retry backoff for Secrets Manager retrieval to improve
resilience.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#31693
Manually forced a run for MySQL 8.4.6 to validate.
# Checklist for submitter
- Changes not needed since this is not a product change.
## Testing
- [x] Added/updated automated tests
Also ensures we run integration tests when docker-compose files used by
`fleetctl preview` are changed, so we don't merge any more test failures
due to those.
For #28902
Modified the GitHub Actions workflow to prevent it from hanging at the
"waiting for mysql..." step. The updated workflow now:
- Times out after 5 minutes of unsuccessful MySQL connection attempts
- Logs all connection attempts with timestamps and error messages when a
timeout occurs
- Dumps MySQL container logs when a timeout occurs, providing valuable
diagnostic information
- Automatically stops and restarts all Docker containers using the same
command as the original "Start Infra Dependencies" step
- Retries this process up to 5 times before failing the job
Replacing MySQL 8.4 with MySQL 9.X in our regular (every commit) tests
to speed up dev feedback. The MySQL 8.4 tests will still run nightly.
The vulnerability feed download CI fails below are not related to this
change.
This PR makes several improvements to test-go CI job
- remove ZSH dependency from all test suites except for a new `scripts`
suite
- add a `fast` suite that does not have out-of-process dependencies,
which completes in ~1.5 minutes
- contributors can add their fast tests to this suite so they can see
the results in CI faster
- Rename `core` to `main` test suite to be consistent with Makefile. It
is the default bucket for tests.
- Cleaned up Makefile so that it is more straightforward to add new test
suites or move Go packages between suites
- Do not stop the test suites on a fail.
- We do not want to be blocked by a test fail that another product team
introduced
- Sometimes, we want to see all test failures so we can fix them all at
once.
- Removed `test-schema` and `mock` prerequisites for `test-go` since
they are not needed and just take up time.
- But also added `test-schema` run to one of the test suites just in
case.
Unfortunately, `fleetctl` is still the bottleneck and needs to be
refactored. New issue filed:
https://github.com/fleetdm/fleet/issues/27927
## Before
<img width="248" alt="image"
src="https://github.com/user-attachments/assets/110ffc1d-f090-4d3e-be77-0419b9577d20"
/>
## After
<img width="320" alt="image"
src="https://github.com/user-attachments/assets/8d01ea11-408f-4eb6-81d8-9c25410b8830"
/>
Release notes: https://tip.golang.org/doc/go1.24
> Go modules can now track executable dependencies using tool directives
in go.mod. This removes the need for the previous workaround of adding
tools as blank imports to a file conventionally named “tools.go”. The go
tool command can now run these tools in addition to tools shipped with
the Go distribution. For more information see [the
documentation](https://tip.golang.org/doc/modules/managing-dependencies#tools).
The new -tool flag for go get causes a tool directive to be added to the
current module for named packages in addition to adding require
directives.
I ran:
```
go get -tool github.com/fleetdm/fleet/v4/server/goose
go get -tool github.com/kevinburke/go-bindata
go get -tool github.com/quasilyte/go-ruleguard/dsl
go rm tools.go
go mod tidy
```
`make deps-go` was failing in CI because of the removal of `tools.go`
(my guess is that `go get .` was a nop because there was nothing in `.`
to download).
So, taking the chance of removing `deps-go` because `go` will download
packages during the build process. AFAICS there's no need to download
everything beforehand.
From what I can tell, continue-on-error has been false for the
integration suites since the suites were renamed to `integration-*`, so
this fixes that issue in addition to continuing to run test suites when
the vulns suite fails (which may be due to vulns feed updates).
This also makes the vulns test more resilient to new CVEs being reported
on Python 3.12.0, which is rather likely to collect new CVEs.
# Checklist for submitter
- [x] Added/updated automated tests
Also mention that we test with 8.4.2 in a few more places.
Note that while I'm editing release articles, this isn't retconning
minimum requirements; we mention in 4.55.0 release notes further down
that we expect 8.0.36.
Done as part of oncall improvements.
`vars.GO_VERSION` can only be changed by admins and it's not public
(Fleet devs don't know the current value of the variable), this approach
uses the version specified in our `go.mod` file.
This ensures `#help-engineering` is notified when a dogfood deploy is in
progress. It helps set people's expectations about what's going on while
the server is temporarily down.
for #18297
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Manual QA for all new/changed functionality
Juan Fernandez