Resolves#36909.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For #32571.
Original PR from the community:
https://github.com/fleetdm/fleet/pull/32573.
Changes on this PR:
- Only setting the checksum algorithm when using GCS as backend (to not
break other S3 backends).
- Changes for carves, bootstrap packages, and software icons which also
use S3.
## Testing
- [X] QA'd all new/changed functionality manually
```sh
FLEET_S3_SOFTWARE_INSTALLERS_BUCKET=some-software-installers-bucket \
FLEET_S3_SOFTWARE_INSTALLERS_ACCESS_KEY_ID=... \
FLEET_S3_SOFTWARE_INSTALLERS_SECRET_ACCESS_KEY=... \
FLEET_S3_SOFTWARE_INSTALLERS_ENDPOINT_URL=https://storage.googleapis.com \
FLEET_S3_SOFTWARE_INSTALLERS_REGION=us \
FLEET_S3_SOFTWARE_INSTALLERS_FORCE_S3_PATH_STYLE=true \
FLEET_S3_CARVES_BUCKET=some-carves-bucket \
FLEET_S3_CARVES_ACCESS_KEY_ID=... \
FLEET_S3_CARVES_SECRET_ACCESS_KEY=... \
FLEET_S3_CARVES_ENDPOINT_URL=https://storage.googleapis.com \
FLEET_S3_CARVES_REGION=us \
FLEET_S3_CARVES_FORCE_S3_PATH_STYLE=true \
./build/fleet serve --dev --dev_license --logging_debug 2>&1 | tee ~/fleet.txt
```
# Overview
This PR implements the S3 upload manager under-the-hood of our
`datastore/s3` client's `Put()` method.
# Description
As surfaced by #31667, the current S3 implementation utilizes the `PUT`
operation which means services, such as MinIO, take issue with attempted
uploads that are [too
large](https://github.com/minio/minio/blob/master/cmd/streaming-signature-v4.go#L260).
The `PUT` operation can also present challenges in memory-constrained
environments as the entire upload target is read into memory before it's
shipped.
# Notes
- See the `TODO` comment section, there's more cool stuff we can and
should do with this in the future!
# Standard Pull Request Details
## Testing
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
---------
Signed-off-by: Illbjorn <am@hades.so>
Fix unreleased bug #30693.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated testing documentation to include a missing command for
creating the Firehose delivery stream for "status" logs.
* **Refactor**
* Centralized AWS STS Assume Role credential configuration across
multiple AWS integrations (S3, Firehose, Kinesis, Lambda, SES) to use a
shared helper, improving maintainability and consistency.
* Removed deprecated inline credential configuration logic in favor of
the new centralized approach.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#29482
[Migrate to the AWS SDK for Go
v2](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html)
documents how to migrate codebases.
QA on features that use AWS SDK Go:
- Bootstrap package:
- upload: ✅
- download: ✅
- cleanup: ✅
- Software (upload, download, installation, etc.) ✅
- Cloudfront: Luckly, this feature was already using aws-sdk-go-v2.
- Carves ✅
- Logging:
- Firehose ✅
- Kinesis ✅
- Lambda ✅ (tested result logs to a lambda function on our AWS Dogfood
account)
- Email:
- Amazon SES TODO ⚠️ (this is what Dogfood uses and a few customers)
- We cannot easily test locally, we can use dogfood or load testing
(AWS) environments.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Manual QA for all new/changed functionality
For #24869
This subtask contains code to sign the CloudFront software installer and
bootstrap package URL using AWS SDK URL signer.
It works with the current bootstrap package delivery. For software
installers, fleetd will need to be modified to take advantage of this
URL in a future subtask (which will also include updated API contributor
docs).
My article on signed URLs, for context:
https://victoronsoftware.com/posts/cloudfront-signed-urls/
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
> Related issue; #19526
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
This improves the installerstore CLI tool with:
- The ability to create tests buckets for local development (otherwise you have to interact with another CLI or the MinIO UI)
- Improved error handling and messaging.
Related to #6365 this adds a new tool to upload pre-built Orbit installers to a storage blob. It uses the same file conventions that the Fleet server expects, making it useful for local testing and infra envs alike.\
Usage and details in the README
Related to #6365, this extends the datastore/s3 package to retrieve installers from S3 according to the conventions listed in the parent issue. This also includes:
- A minor refactor to decouple Carves-related functionality from the core S3 functionality
- Set-up to run tests using minio (only enabled via the FILE_STORAGE_TEST env flag)
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
Jonathan Katz