Instead of synchronously updating the seen_time column for a host on an update, batch these updates to be written together every 1 second.
This results in a ~33% reduction in MySQL CPU usage in a local test with 4,000 simulated hosts and MySQL running in Docker.
Prevent abuse of these endpoints with rate limiting backed by Redis. The
limits assigned should be appropriate for almost any Fleet deployment.
Closes#530
This PR contains the initial implementation of the fleetctl updates commands, along with documentation on using this to self-host an agent update server.
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
- Return same error in all cases for login endpoint.
- Log error details in server logs.
- Make most login errors take ~1s to prevent timing attacks.
- Don't return forgot password errors.
- Log password errors in server logs.
- Make most forgot password requests take ~1s to prevent timing attacks.
Fixes#531
Make fleetctl preview available over HTTP
Run a second copy of the Fleet server listening over HTTP on
localhost:1337 so that the UI can be used without the errors displayed
with a self-signed TLS certificate. Osquery clients and fleetctl
continue to communicate with the existing Fleet server on
https://localhost:8412.
Closes#504
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
- Download configuration on every run. This ensures that users always
get the latest version.
- Pull latest Docker containers on every run.
- Add stop and reset commands (for cleanup).
Fixes#378
Osquery now exposes more information during host enrollment than Fleet
previously handled. We can use this to provide more options to users in
problematic enrollment scenarios.
Users can configure --osquery_host_identifier in Fleet to set which
identifier is used to determine uniqueness of hosts. The
default (provided) replicates existing behavior in Fleet. For many
users, setting this to instance will provide better enrollment
stability.
Closes#373
- Better documentation and error for missing docker-compose.
- Handle case of no existing config file.
- Make logs directory world-writable (to allow writes from inside container).
Fixes#286
- Add --dev flag that will set default flag values. This simplifies the
invocation of Fleet in a development environment.
- Change defaults in docker-compose to use `fleet` in place of `kolide`.
- Skip prompt in `prepare db` when `--dev` specified.
- Update developer documentation.
Updates to MySQL configuration in docker-compose.yml may require
existing development containers and volumes to be deleted (this will
delete data in MySQL):
```shell
docker-compose rm -sf
docker volume rm fleet_mysql-persistent-volume
```
Closes#170
- Make the preview directory in the default .fleet directory.
- Check for Docker daemon installed but not running.
- Add message for Chrome users on self-signed certs.
- Display login information on later invocations of command.
- Remove "Kolide" from error messages.
Closes#190
Part of #197
Mozilla's recommended settings have changed since this was last updated.
We now link directly to the relevant revision in the Mozilla wiki to
avoid confusion if this is updated.
- Run the `fleetctl setup` command to set up the Fleet server with
default username ('admin') and password ('admin123#').
- Configures fleetctl if it has not yet been configured.
Closes#152
The current implementation of FleetDM doesn't support Docker secrets for supplying the MySQL password and JWT key. This PR provides the ability for a file path to read in secrets. The goal of this PR is to avoid storing secrets in a static config or in an environment variable.
Example config for Docker:
```yaml
mysql:
address: mysql:3306
database: fleet
username: fleet
password_path: /run/secrets/mysql-fleetdm-password
redis:
address: redis:6379
server:
address: 0.0.0.0:8080
cert: /run/secrets/fleetdm-tls-cert
key: /run/secrets/fleetdm-tls-key
auth:
jwt_key_path: /run/secrets/fleetdm-jwt-key
filesystem:
status_log_file: /var/log/osquery/status.log
result_log_file: /var/log/osquery/result.log
enable_log_rotation: true
logging:
json: true
```
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
This adds a `fleetctl preview` command to fleetctl that will download
and start https://github.com/fleetdm/osquery-in-a-box. This is
potentially the easiest way for a user to try out Fleet.
Adds endpoints and fleetctl commands to retrieve various debug profiles
from the Fleet server.
The best summary is from the help text:
```
fleetctl debug
NAME:
fleetctl debug - Tools for debugging Fleet
USAGE:
fleetctl debug command [command options] [arguments...]
COMMANDS:
profile Record a CPU profile from the Fleet server.
cmdline Get the command line used to invoke the Fleet server.
heap Report the allocated memory in the Fleet server.
goroutine Get stack traces of all goroutines (threads) in the Fleet server.
trace Record an execution trace on the Fleet server.
archive Create an archive with the entire suite of debug profiles.
OPTIONS:
--config value Path to the Fleet config file (default: "/Users/zwass/.fleet/config") [$CONFIG]
--context value Name of Fleet config context to use (default: "default") [$CONTEXT]
--help, -h show help
```
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
Fixes issue where `fleetctl get options` returns `option` (singular)
as `kind` instead of `options` (plural). This would cause `fleetctl
apply -f options.yml` to fail on options yaml generated by `fleetctl get
options` with this error: `unknown kind "option"`.
Getting a single host with `fleetctl get host foobar` will look up the
host with the matching hostname, uuid, osquery identifier, or node key,
and provide the full host details along with the labels the host is a
member of.
This commit takes advantage of the existing pagination APIs in the Fleet
server, and provides additional APIs to support pagination in the web
UI. Doing this dramatically reduces the response sizes for requests from
the UI, and limits the performance impact of UI clients on the Fleet and
MySQL servers.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Zach Wasserman