Commit Graph
367 Commits
Author SHA1 Message Date
Gabriel HernandezandLucas Rodriguez f61a17bd9b add google analytics to sandbox instances (#6941)
* add google analytics to sandbox instances

* Add serverType variable to frontend handler

* update version of html-webpack-plugin

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2022-08-01 11:27:12 +01:00
Roberto Dip 780db937de pass a pointer to config instead of the whole config to goroutines (#6823) 2022-07-26 12:27:30 -03:00
Roberto Dip d63f56f8c0 add support for notarization in fleetdm/fleetctl images (#6818)
#6674
2022-07-25 20:06:10 -03:00
gillespi314 cc34585a47 Add statistics for number of hosts that have not submitted results for distributed queries (#6495) 2022-07-20 20:54:10 -05:00
Michal Nicpon d4be5ad2a1 add upgrade tests (#6596)
* add upgrade tests

* fix lint issues

go.mod

* remove req.cnf

* revert unrelated changes

* make version configurable in test

* fix golangci-lint ruleguard issue

Related to https://github.com/go-critic/go-critic/issues/1152
Need to have github.com/quasilyte/go-ruleguard/dsl

* fix lint issues

* fix

* clean up docker-compose.yml

* fix http request

* add readme

* fix lint issues

* address feedback

* fix

* add platform

* address feedback

* run go fmt
2022-07-19 15:11:51 -06:00
Eng Zer Jun 1ab171faf3 test: use T.Setenv to set env vars in tests (#6714)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-07-18 14:22:28 -03:00
Roberto Dip 69f8f2a73b add API endpoints to retrieve pre-built installers (#6672)
Rel: #6365, this adds a new endpoint to check and download pre-built installers.
2022-07-18 13:44:30 -03:00
Roberto Dip 86366b2ed8 add a new S3 datastore to retrieve pre-built packages (#6631)
Related to #6365, this extends the datastore/s3 package to retrieve installers from S3 according to the conventions listed in the parent issue. This also includes:

- A minor refactor to decouple Carves-related functionality from the core S3 functionality
- Set-up to run tests using minio (only enabled via the FILE_STORAGE_TEST env flag)
2022-07-14 14:14:24 -03:00
Roberto Dip 08cd206de7 only send vuln configs to goroutine (#6650)
In https://github.com/fleetdm/fleet/pull/6630 I added a new config for packaging, but when I started the server I got:

```
~/fleet $ ./build/fleet serve --dev --dev_license
level=info ts=2022-07-13T21:36:06.055998Z component=redis mode=standalone
fatal error: newproc: function arguments too large for new goroutine

runtime stack:
runtime.throw({0x103d85259, 0x37})
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/panic.go:1198 +0x54
runtime.newproc1(0x104569a30, 0x14000ffda28, 0x7f8, 0x140000001a0, 0x103bc85a4)
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/proc.go:4299 +0x4d0
runtime.newproc.func1()
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/proc.go:4255 +0x4c
runtime.systemstack()
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/asm_arm64.s:230 +0x6c

goroutine 1 [running]:
runtime.systemstack_switch()
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/asm_arm64.s:187 +0x8 fp=0x14000ffd9c0 sp=0x14000ffd9b0 pc=0x102b60958
runtime.newproc(0x7f8, 0x104569a30)
	/Users/roperzh/.gvm/gos/go1.17/src/runtime/proc.go:4254 +0x54 fp=0x14000ffda10 sp=0x14000ffd9c0 pc=0x102b38034
main.runCrons({0x1045cd500, 0x14000262990}, {0x104611a38, 0x14000262930}, 0x14000298190, {0x1045a0480, 0x140009c9f20}, {{{0x103d1adc4, 0x3}, {0x103d2b188, ...}, ...}, ...}, ...)
	/Users/roperzh/fleet/cmd/fleet/serve.go:694 +0x2c4 fp=0x14000ffe260 sp=0x14000ffda10 pc=0x103bc85a4
main.createServeCmd.func1(0x1400027ca00, {0x140000bcb40, 0x0, 0x2})
```

With my local changes, `serve.go:694` is this line:

https://github.com/fleetdm/fleet/blob/7559988000efc3d7408d9914edd5a3f2f1052138/cmd/fleet/serve.go#L685-L686

After passing only a subset of the config the issue was solved.
2022-07-14 08:04:29 -03:00
Roberto Dip 6faee84f57 allow to configure a default global enroll secret (#6609)
Related to #6365 this adds a new config to set a global enroll token that will be used by the server.
2022-07-12 19:12:10 -03:00
Roberto Dip f7dd8c86cd implement a docker image to package orbit natively in Linux (#6504)
Related to #6364 and #6363, this:

- Adds a new Docker image, `fleetdm/fleetctl` equipped with all necessary dependencies to build Fleet-osquery binaries for all platforms
- Modifies the package generation logic to special case this scenario via an environment variable `FLEETCTL_NATIVE_TOOLING`
- Adds a new GitHub workflow to test this

There are more details in the README, but part of the special-casing logic is in place to output the binaries to a folder named `build` when they are run with `FLEETCTL_NATIVE_TOOLING`, this is so we can persist the binary generated by the docker container via a bind mount:

```bash
docker run -v "$(pwd):/build" fleetdm/fleetctl package --type=msi
```

To test this changeset, I have generated packages for all platforms, both via the new Docker image and via the classic `fleetctl package`.
2022-07-11 09:49:13 -03:00
Lucas Manuel Rodriguez a336ed61e5 Add gotestfmt to improve test output and fix flaky tests (#6528) 2022-07-11 08:12:33 -03:00
Luke Heath 7408dc0aae Update dev license (#6476) 2022-07-01 10:17:19 -07:00
Zach Wasserman db22f68c88 Separate health checks for MySQL and Redis (#6468)
This required a bit of refactoring of some mocking due to how the code
generation does not handle having the same function in different types.
2022-07-01 08:08:03 -03:00
Zach Wasserman 974c8e008d Improve logging for fleetctl with permissions errors (#6460) 2022-06-30 13:24:19 -07:00
Roberto Dip b2d07e56b5 report error analytics (#6341)
Related to #5898, this reports an anonymized summary of errors stored in Redis into the analytics payload.

For each error stored, this includes:

- A `count` attribute with the number of occurrences of the error
- A `loc` attribute with the 3 topmost lines in the stack trace. Note that stack traces only contain package name + line number (example: github.com/fleetdm/fleet/server.go:12

This also includes a minor refactor around error types.
2022-06-28 16:31:14 -03:00
Martin Angers 539be8ee09 Add battery info in host details response (#6394) 2022-06-28 14:11:49 -04:00
Juan Fernandez 79bf51b03c Feature 6242: Use oval to detect vulnerabilities on Fedora hosts (#6330)
* Feature 6242: Scan Fedora hosts using OVAL definitions
2022-06-24 11:02:51 -04:00
Juan Fernandez 9d01ba33c6 Feature 6096: Scan RHEL/CentOS hosts using OVAL definitions (#6241)
Extended the OVAL parser/analyzer so that we can scan RHEL based systems.
2022-06-23 16:44:45 -04:00
Martin Angers 8f66a50f2b Add hosts enrolled by operating system to anonymous usage stats (#6329) 2022-06-22 16:38:11 -04:00
Roberto Dip 2948e112f6 ensure software_host_counts is cleaned when software is deleted (#6270)
Related to #5982, this ensures we clean up software_host_counts rows referencing software that is not longer present in the software table.
2022-06-22 17:35:53 -03:00
Martin Angers dc9e158153 Add weekly active users count in anonymous usage stats (#6317) 2022-06-22 15:20:57 -04:00
Ahmed Elshaer a9f48ff561 Create Bulk Users from CSV (#3372)
* Create Bulk Users

* WIP: Adding a test for bulk user import

* adding a user bulk create test

* Fixing description, removing password required, and adding more test cases

* Fixing description, removing password required, and adding more test cases

* Fixed all comments and added Random Password Generator

* returning an error in generateRandomPassword

* Using 2 loops to create user list and then create the actual users

* Adding a bulk user delete

* fixing a mistake in temp csv

* fixed lints and removed yamlFlag
2022-06-22 13:34:58 -03:00
Roberto Dip 4a867d53dc use a single context for background jobs and HTTP handlers (#6313) 2022-06-21 15:09:00 -03:00
Martin Angers 81f0e0ccfa Track active hosts count and enforce limit (#6099) 2022-06-13 16:29:32 -04:00
Lucas Manuel Rodriguez 11af33e9a1 Allow troubleshooting of mocked live query store (#6197) 2022-06-13 10:18:03 -03:00
Roberto Dip 18de43f35b fix fleetctl debug commands on Windows (#6186)
As reported in #6127, the `fleetctl debug` `archive` and `errors` commands were failing on Windows because filenames are not allowed to contain colons `:`.

This changeset removes colina from the filename of the archives generated by both commands.
2022-06-10 21:59:44 -03:00
Lucas Manuel Rodriguez 4cfeaa1580 Do not use golangci action for better reproducibility (use make lint-go) (#6175)
* Do not use golangci action for better reproducibility

* Add fix to trigger build

* Fix all reported issues

* fix more lint errors

* Add missing import

* Remove unused method

* Remove change not necessary
2022-06-10 18:52:24 -03:00
gillespi314 a3ab5646f5 Add new fleet_desktop property to config object (#6151) 2022-06-10 10:39:02 -05:00
Lucas Manuel Rodriguez 3dd89bc0a1 Fix flaky test TestVulnerabilityDataStream (#6162) 2022-06-09 10:36:37 -03:00
Michal Nicpon bac168e3fd fix mem usage cve sync (#6131) 2022-06-08 13:15:44 -06:00
Juan Fernandez ef73039559 Improve vulnerability detection for Ubuntu (#6102)
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts

To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
2022-06-07 21:09:47 -04:00
Martin Angers 0b5fcd8e27 Add support for custom headers in fleetctl (#6118) 2022-06-07 16:00:09 -04:00
Martin Angers a23e0c41ff Support failing policies integrations (#5973) 2022-06-06 10:41:51 -04:00
Lucas Manuel Rodriguez e0d7318236 Fix osquery-perf bug introduced in orbit_info handling (#5986) 2022-06-01 13:57:44 -03:00
Michal Nicpon 30a864c8d2 Sync CVE scores periodically (#5838) 2022-06-01 10:06:57 -06:00
Michal Nicpon 1da9425869 make softare actually unique for osquery perf (#5956) 2022-05-31 10:37:57 -06:00
Lucas Manuel Rodriguez 0b511ccd1e Add orbit_info support to osquery-perf (#5922)
* Add support for orbit_info to osquery-perf

* Add orbit_info support to osquery-perf

* Use 0.5 as default for orbit_info

* Fix CodeQL security warning on osquery-perf
2022-05-31 10:15:58 -03:00
Lucas Manuel Rodriguez 9080563b7d Fix some go lint warnings (#5888)
* Fix go lint warnings

* Add test for SMTPConfigured update
2022-05-31 07:19:57 -03:00
gillespi314 baeff6e893 Add validation for pack scheduled query interval (#5918) 2022-05-26 16:54:21 -05:00
Desmi-Dizney 0ed6069a26 Editor pass - Add Zendesk external service integration for vulnerability automations (#5781)
Editor pass for:
-  https://github.com/fleetdm/fleet/pull/5372
2022-05-25 09:33:00 -05:00
gillespi314 2db2c16511 Include MIA hosts under total count for Offline hosts (#5854) 2022-05-23 14:11:02 -05:00
Michal Nicpon 983f9e59b3 Include CVE scores when listing software (#5673) 2022-05-20 10:58:40 -06:00
gillespi314 4a4e832d3a Increase minimum password length to 12 characters (#5712) 2022-05-18 12:03:00 -05:00
Roberto Dip a6a97f31cf add documentation for debug tools (#5737)
This adds documentation about our debugging endpoints and a brief excerpt about the fleetctl debug command with instructions to generate the archive so we have a place with instructions to which we can direct people.

It also adds the .prof file extension to profiling files, which hopefully clarifies that they are meant to be used by go tool pprof.
2022-05-17 10:00:47 -03:00
Roberto Dip 32cc4c4641 allow fleetctl preview to work with docker compose v1 and v2 (#5755)
This adds compatibility in `fleetctl preview` to work with docker compose (version 2). Since this version was released this April, we are still keeping backwards compatibility and using docker-compose as a fallback.

As v2 is now the recommended version and v1 is deprecated, this also rewords all prompts and help messages to say "docker compose".

Rel: #5746
2022-05-16 18:06:29 -03:00
Martin Angers e6b90ca8b9 Support per-task configuration for async host processing configuration (#5700) 2022-05-16 09:44:50 -04:00
Martin Angers 1fa7bb7a19 Support async saving of hosts' last seen time (#5640) 2022-05-10 11:29:17 -04:00
Lucas Manuel Rodriguez fda79a8770 Run network test serially to prevent timeouts on Github CI (#5557)
* Run network test serially to prevent timeouts on Github CI

* Revert lint changes

* Add simple file lock

* Revert test change

* Clarify error check
2022-05-10 11:52:33 -03:00
Roberto Dip cba78bdcd1 don't clear errors automatically when reading them from Redis (#5623)
this change prevents errors from being automatically cleared once they are read. A new flag `-flush` is introduced to flush errors on read if necessary.
2022-05-10 11:44:43 -03:00