Commit Graph
558 Commits
Author SHA1 Message Date
gillespi314 d172128183 Add label information to host summary response (#5573) 2022-05-10 10:32:55 -05:00
Martin Angers 1fa7bb7a19 Support async saving of hosts' last seen time (#5640) 2022-05-10 11:29:17 -04:00
Michal Nicpon a005b7e7b7 fix schema dump (#5562) 2022-05-10 09:03:04 -06:00
gillespi314 998471f716 Omit empty device mapping from host responses where optional device_mapping param is missing (#5541) 2022-05-05 13:13:53 -05:00
gillespi314 991c7ccadc Add device_mapping to GET /hosts response (#5383) 2022-05-02 16:34:14 -05:00
gillespi314 1dabf52834 Add Zendesk external service integration for vulnerability automations (#5372) 2022-05-02 15:58:34 -05:00
Roberto Dip a4725518ac filter out Google Chrome profiles without an associated email before ingesting (#5440)
To add support for #400, we're using the macadmins/osquery-extension to gather Google Chrome profiles from hosts.

Under the hood, the extension looks and parses a json file in which Chrome stores a bunch of data. Given that emails are not required to create Chrome profiles, some of the profiles stored in this file and returned by the query we're using contain empty emails.

The idea after this change is to prevent empty emails from being ingested in the first place instead of filtering them after the fact. I have also included a migration to clean the rows with empty email columns.

Fixes #4780
2022-05-02 09:55:40 -03:00
Roberto Dip 2c23cb5f24 fix mysql tests by with a regexp assertion of possible error messages (#5444)
this fixes #5377 by asserting that the error message starts with "tsl" or "x509" as either of the strings represent the same error depending on the mysql version.

we should refactor the code to avoid checking on the error messages and probably use different error types instead.
2022-04-29 09:31:58 -03:00
Martin Angers 0d0c17e0c6 Add osquery.min_software_last_opened_at_diff configuration option (#5394) 2022-04-27 09:47:09 -04:00
Lucas Manuel Rodriguez 407d8b527e Remove unused Datastore.SaveHost and unused Modified fields (#5245) 2022-04-26 18:33:29 -03:00
Martin Angers 253baee657 Collect last_opened_at for macOS software, and return it in host details payload (#5376) 2022-04-26 14:16:59 -04:00
Michal Nicpon a64982f96c Remove use of JSON_ARRAYAGG (#5350)
Not supported in MySQL < 5.7.22
2022-04-25 14:43:08 -06:00
Lucas Manuel Rodriguez 42f53b63b2 Remove recursion and return doRetryErr on getOrGenerateSoftwareIdDB (#5239)
* Remove recursion and return doRetryErr on getOrGenerateSoftwareIdDB

* Use GetContext and LIMIT 1
2022-04-20 11:02:35 -03:00
Lucas Manuel Rodriguez 29a902960e Fix typos and amend PR template (#5227) 2022-04-19 18:09:55 -03:00
Juan Fernandez 2479d58262 Using '@' in target search causes 422 error response (#5148)
* Bug 4852:

When searching for Labels, breakdown query strings containing '@' in multiple search terms to
avoid issues with MySQL FTS.
2022-04-19 09:28:49 -04:00
gillespi314 70a61b86f1 Update os version aggregated stats for all teams (#5083) 2022-04-18 16:19:58 -05:00
Michal Nicpon 2dbf1d7975 fix update updated_at for aggregated_stats (#5112)
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date

* basic sql formatting in code ie whitespace around operators
2022-04-15 14:09:47 -06:00
Martin Angers 7187f1adac Queue jobs for Jira integration when enabled and new vulnerabilities are found. (#4975) 2022-04-11 16:42:16 -04:00
Martin Angers 5483adc26b Detect the NOPERM error to mean redis cluster is disabled (#5058) 2022-04-11 16:17:30 -04:00
Ricky GrassmuckandRicky Grassmuck eddc2e2bda Replace uses of ansi quotes (") in SQL statements (#4726)
To ensure the product works properly when using MySQL with `ANSI_QUOTES` mode enabled,
replace all uses of `""` for values inside SQL statements with `''`

Co-authored-by: Ricky Grassmuck <r.grassmuck@cpanel.net>
2022-04-11 11:47:50 -07:00
Martin Angers f28dc10a51 Merge jira-integration branch to main (#4949) 2022-04-05 12:56:15 -04:00
Martin Angers 90b15071a4 Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
Lucas Manuel Rodriguez 57816592ba Add read replica testing helpers and fix non-sso login bug (#4908)
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)

New version:

```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
	sqlStatement := `
		INSERT INTO sessions (
			user_id,
			` + "`key`" + `
		)
		VALUES(?,?)
	`
	result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
	if err != nil {
		return nil, ctxerr.Wrap(ctx, err, "inserting session")
	}

	id, _ := result.LastInsertId() // cannot fail with the mysql driver
	return ds.sessionByID(ctx, ds.writer, uint(id))
}
```

- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).

PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
2022-04-04 16:52:05 -07:00
Martin Angers 71f2690ada Add Jira integrations config support (#4863) 2022-03-30 09:10:02 -04:00
Michal Nicpon a6902cc083 Add os versions endpoint (#4749) 2022-03-28 09:15:45 -06:00
Martin Angers c4946335ff Add migration to cleanup host-related tables (#4778)
#4448
2022-03-23 15:22:10 -07:00
Lucas Manuel Rodriguez 8f9ed3ef7b Fix Datastore.DeleteHosts (#4745) 2022-03-23 12:15:05 -03:00
Michal Nicpon 7b671ac2a3 Add team failing policies webhook (#4633)
* add config to teams
* update api docs
* update tests
2022-03-21 13:16:47 -06:00
Lucas Manuel RodriguezandZach Wasserman ecdfd627b6 Fleet Desktop MVP (#4530)
* WIP

* WIP2

* Fix orbit and fleetctl tests

* Amend macos-app default

* Add some fixes

* Use fleetctl updates roots command

* Add more fixes to Updater

* Fixes to app publishing and downloading

* Add more changes to support fleetctl cross generation

* Amend comment

* Add pkg generation to ease testing

* Make more fixes

* Add changes entry

* Add legacy targets (until our TUF system exposes the new app)

* Fix fleetctl preview

* Fix bool flag

* Fix orbit logic for disabled-updates and dev-mode

* Fix TestPreview

* Remove constant and fix zip-slip attack (codeql)

* Return unknown error

* Fix updater's checkExec

* Add support for executable signing in init_tuf.sh

* Try only signing orbit

* Fix init_tuf.sh targets, macos-app only for osqueryd

* Specify GOARCH to support M1s

* Add workflow to generate osqueryd.app.tar.gz

* Use 5.2.2 on init_tuf.sh

* Add unit test for tar.gz target

* Use artifacts instead of releases

* Remove copy paste residue

* Fleet Desktop Packaging WIP

* Ignore gosec warning

* Trigger on PR too

* Install Go in workflow

* Pass url parameter to desktop app

* Fix fleetctl package

* Final set of changes for v1 of Fleet Desktop

* Add changes

* PR fixes

* Fix CI build

* add larger menu bar icon

* Add transparency item

* Delete host_device_auth entry on host deletion

* Add SetTargetChannel

* Update white logo and add desktop to update runner

* Add fleet-desktop monitoring to orbit

* Define fleet-desktop app exec name

* Fix update runner creation

* Add API test before enabling the My device menu item

Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2022-03-21 14:53:53 -03:00
Benjamin Edwards 74bb559645 Add public ip to hosts & derive geolocation when rendering host (#4652)
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
2022-03-21 12:29:52 -04:00
Zach Wasserman efbc2b92bb Fix race condition in updates test (#4661)
Copy the DefaultOptions in order to prevent a data race on the Targets
map. This race should only have effected testing.

Race detector output:

```
WARNING: DATA RACE
Read at 0x00c0000908d0 by goroutine 15:
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.18.0/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath.func1()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:58 +0xb6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Previous write at 0x00c0000908d0 by goroutine 12:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.18.0/x64/src/runtime/map_faststr.go:203 +0x0
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath.func1()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:62 +0x1cb
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Goroutine 15 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x724
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:53 +0x1a4
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x47

Goroutine 12 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486 +0x724
  github.com/fleetdm/fleet/v4/orbit/pkg/update.TestMakeRepoPath()
      /home/runner/work/fleet/fleet/orbit/pkg/update/update_test.go:53 +0x1a4
  testing.tRunner()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1439 +0x213
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.18.0/x64/src/testing/testing.go:1486
      +0x47
 ```
2022-03-18 09:30:45 -07:00
Martin Angers c0591dac5d Ignore pagination params for the download csv endpoint (#4608) 2022-03-16 09:01:52 -04:00
Lucas Manuel Rodriguez d2ba34c8fc Add distributed discovery query support for detail queries, add orbit… (#4597)
* Add distributed discovery query support for detail queries, add orbit_info ingestion

* Amend changes file
2022-03-15 16:51:00 -03:00
Martin Angers a1c67547b3 Add new endpoints to retrieve device information by orbit identifier (#4531) 2022-03-09 16:13:56 -05:00
Martin Angers c8bc026d6f Migrate special-case endpoints to new pattern (#4511) 2022-03-08 11:27:38 -05:00
Tomas Touceda dd62695791 Add host device auth table (#4494) 2022-03-08 09:05:53 -03:00
Martin Angers 4930ca2d0e Support listing software hosts count filtered by team (#4388) 2022-02-28 13:55:14 -05:00
Martin Angers 18e7c8e236 Implement cron cleanup job of policy membership when policy platform is updated (#4331) 2022-02-23 16:10:37 -05:00
Martin Angers ad5146c491 Allow updating the policy platform (part 1 of the ticket) (#4311) 2022-02-22 13:42:03 -05:00
Tomas Touceda 4034a7ab59 Make SearchHosts behave the same as the filtering in ListHosts (#4295) 2022-02-22 10:19:51 -03:00
Martin Angers 2ab1b9ec85 Remove expiration of API-only user tokens (#4314) 2022-02-22 08:12:03 -05:00
Michal Nicpon 4ce23c7d1b Add team targets to pack spec (#4272)
* skip flaky tests
2022-02-21 09:18:58 -07:00
Tomas Touceda 6d582cffa0 Don't depend on last inserted id to get the id for the new software (#4298)
* Don't depend on last inserted id to get the id for the new software

* Correct typo
2022-02-18 16:30:24 -03:00
Tomas Touceda 8e68ec3b96 Insert one CVE per CPE when there are multiple matches (#4297)
* Insert one CVE per CPE when there are multiple matches

* Remove comment

* No need to do sqlx.In
2022-02-18 15:25:26 -03:00
gillespi314 089f49cc5d Add new activity types for policy-related activities (#4043)
* Add new activities for policy create, delete, and modify, and apply policy spec
* Add PoliciesByID ds method; refactor delete svcs
2022-02-16 09:33:56 -06:00
Tomas Touceda a49f2a06ed Allow removal of munki versions (#4189)
* Skip any versions that are empty in munki

* Return not found if munki is uninstalled

* Fix lint

* wip

* Add deleted at for host_munki_version
2022-02-15 16:29:14 -03:00
Michal Nicpon 0be26613b3 Cache team agent options (#4193)
* use raw literal for json
* wrap cache to clone all values
2022-02-15 12:07:51 -07:00
Zachary WinnermanandTomas Touceda c5c72ed713 Add apm for testing apm (#4053)
* Add apm for testing apm

* Testing opentracing

* testing

* Testing

* go fmt

* Add config switch for tracing.

* fixup

* Update cmd/fleet/serve.go

Co-authored-by: Tomas Touceda <chiiph@gmail.com>

* Add support for both elasticapm and opentelemetry

* Fix driver stuff and config options

* Fixup

* fixup

* Add changes file

* Add config for sql driver

* fixup

* Add doc to exported field

* testing

* fixup

* fixup

* Testing again

* fixup

* testing

* Undo

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-15 12:42:22 -05:00
Lucas Manuel Rodriguez be72dc356c Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Lucas Manuel Rodriguez 9c25ea1641 Prepare LoadHostByNodeKey query once (#4128)
* Prepare LoadHostByNodeKey query once

* Use a protected map for storing statements

* Add proposed test
2022-02-14 12:13:38 -03:00