Update documentation for file carving (#3213)

This commit is contained in:
Zach Wasserman
2021-12-09 09:59:23 -08:00
committed by GitHub
parent 32571c0990
commit 2f4ea1da85
2 changed files with 29 additions and 8 deletions
+27 -8
View File
@@ -194,22 +194,29 @@ Given a working flagfile for connecting osquery agents to Fleet, add the followi
```
--disable_carver=false
--carver_disable_function=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=2000000
--carver_block_size=2097152
```
The default flagfile provided in the "Add New Host" dialog also includes this configuration.
#### Carver block size
The `carver_block_size` flag should be configured in osquery. 2MB (`2000000`) is a good starting value.
The `carver_block_size` flag should be configured in osquery.
The configured value must be less than the value of `max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for MySQL 5.7 is 4MB and for MySQL 8 it is 64MB.
For the (default) MySQL Backend, the configured value must be less than the value of
`max_allowed_packet` in the MySQL connection, allowing for some overhead. The default for MySQL 5.7
is 4MB and for MySQL 8 it is 64MB. 2MiB (`2097152`) is a good starting value.
In case S3 is used as the storage backend, this value must be instead set to be at least 5MB due to the [constraints of S3's multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html).
For the S3/Minio backend, this value must be set to at least 5MiB (`5242880`) due to the
[constraints of S3's multipart
uploads](https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html).
Using a smaller value for `carver_block_size` will lead to more HTTP requests during the carving process, resulting in longer carve times and higher load on the Fleet server. If the value is too high, HTTP requests may run long enough to cause server timeouts.
Using a smaller value for `carver_block_size` will lead to more HTTP requests during the carving
process, resulting in longer carve times and higher load on the Fleet server. If the value is too
high, HTTP requests may run long enough to cause server timeouts.
#### Compression
@@ -262,7 +269,8 @@ The same is not true if S3 is used as the storage backend. In that scenario, it
#### Minio
Configure the following:
- `FLEET_S3_ENDPOINT_URL=minio_host`
- `FLEET_S3_ENDPOINT_URL=minio_host:port`
- `FLEET_S3_BUCKET=minio_bucket_name`
- `FLEET_S3_SECRET_ACCESS_KEY=your_secret_access_key`
- `FLEET_S3_ACCESS_KEY_ID=acces_key_id`
- `FLEET_S3_FORCE_S3_PATH_STYLE=true`
@@ -282,8 +290,19 @@ fleetctl query --labels 'All Hosts' --query 'SELECT * FROM carves'
can be helpful to debug carving problems.
#### Ensure carver_block_size is set appropriately
#### Ensure `carver_block_size` is set appropriately
This value must be less than the `max_allowed_packet` setting in MySQL. If it is too large, MySQL will reject the writes.
`carver_block_size` is an osquery flag that sets the size of each part of a file carve that osquery
sends to the Fleet server.
When using the MySQL backend (default), this value must be less than the `max_allowed_packet`
setting in MySQL. If it is too large, MySQL will reject the writes.
When using S3, the value must be at least 5MiB (5242880 bytes), as smaller multipart upload
sizes are rejected. Additionally [S3
limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html) the maximum number of
parts to 10,000.
The value must be small enough that HTTP requests do not time out.
Start with a default of 2MiB for MySQL (2097152 bytes), and 5MiB for S3/Minio (5242880 bytes).
+2
View File
@@ -1769,6 +1769,8 @@ See [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) f
AWS S3 Region. Leave blank to enable region discovery.
Minio users must set this to any nonempty value (eg. `minio`), as Minio does not support region discovery.
- Default value:
- Environment variable: `FLEET_S3_REGION`
- Config file format: