Commit Graph
210 Commits
Author SHA1 Message Date
Juan Fernandez 7f83135aa1 Feature: Store installed file path when ingesting software (#11214)
Store software installed paths into the host_software_installed_paths table when ingesting osquery software data.
2023-05-17 14:49:09 -04:00
Juan Fernandez d353a3deb4 Handle flaky vulnerability tests (#11262)
- Refactored some of the vulnerabilities tests to help with flakiness.
- Don't load NVD assets if local assets have a timestamp of today.
2023-04-21 19:37:29 -04:00
Juan Fernandez 4c2ddba2e4 Clean out-of-date NVD results. (#10514)
Keep the vulnerabilities detected via NVD and stored in the DB in sync. with the results from the NVD vulnerability process.
2023-04-03 13:45:18 -04:00
Juan Fernandez 56ed2727b5 Updated translation rules so that Docker Desktop can be mapped to the proper CPE (#10326)
Updated translation rules so that Docker Desktop can be mapped to the proper CPE.
2023-03-09 17:46:57 -04:00
Juan Fernandez 6ff2c449cf Improve test coverage and bug fix (#10301)
Addresses https://github.com/fleetdm/fleet/issues/10112

Improved test coverage, also fixed a bug related to vulnerability processing for mac office apps.
2023-03-06 15:07:27 -04:00
Juan Fernandez 7e366272c0 Feature 9386: Parse the Mac Office release notes for vulnerability processing (#9993)
This PR adds the capability of parsing the release notes posted in https://learn.microsoft.com/en-us/officeupdates/release-notes-office-for-mac into a JSON metadata file (to be released in the NVD repo) and use it for detecting vulnerabilities on Mac Office apps.
2023-02-24 14:18:25 -04:00
Benjamin Edwards 0db432f7e3 vuln processing distinct command (#9813)
closes https://github.com/fleetdm/fleet/issues/3723

Add new vuln processing command, configs, and documentation on how to utilize.
2023-02-17 10:00:57 -05:00
Juan Fernandez e376d53294 Bug 9386: Docs chrome extension has the wrong CPE. (#9524)
Related to #9386 - this should fix one of the three reported problems.

* Add the ability to add exclusion rules to cpe_translations.
* Added exclusion rule for Docs chrome extension.
2023-01-27 15:11:20 -04:00
Lucas Manuel Rodriguez f64ba83b2e Make test more stable (#9162) 2023-01-03 16:39:28 -03:00
Lucas Manuel Rodriguez ac22aadc13 Fleet server and tooling to use NETWORK_TEST_GITHUB_TOKEN when environment variable is set. (#9143)
* WIP

* Add more logging

* Check rate limit at end of action

* Add github client in more places

* Add new published firefox 93 vulnerabilities to tests

* Remove fmt printfs

* Restore CI check settings

* Readd newline
2023-01-03 14:56:11 -03:00
gillespi314 94dd1c3745 Ingest pending MDM hosts (#9065)
Co-authored-by @roperzh
2022-12-26 15:32:39 -06:00
gillespi314 6fb3a87ae9 Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
Roberto Dip 837fef4bc4 ensure we provide a future date in MSRC test (#8897)
This test was failing in Dec 2022, because in this line:

```go
_, err := sut.GetFeed((now.AddDate(0, 1, 0)).Month(), now.Year())
```

`(now.AddDate(0, 1, 0)).Month()` returns `"January"` , and `now.Year()` returns `2022` , so we were sending a date in the past.
2022-12-01 11:29:07 -03:00
Michal Nicpon 9ad1721efd fix issue with duplicate vulns detected using nvd (#8613)
The OVAL analyzer falsely assumes that any vulnerabilities detected on a
host only come from OVAL. However, it is possible that NVD detects
vulnerabilities on these hosts even though it excludes software from
deb_packages and rpm_packages. For example, a python package twisted
v22.20 has a vulnerability CVE-2022-39348 detected by NVD. The OVAL
analyzer would delete this vulnerability, and it would be re-inserted by
the NVD scanner on the next run. This creates a loop.

The fix is to only delete vulnerabilities that are actually detected
using OVAL. We already store this in the source column in the
software_cve table.
2022-11-10 10:28:00 -07:00
Juan Fernandez 53e112d264 Feature 7494: Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities (#7889)
Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities
2022-10-28 11:12:21 -04:00
Juan Fernandez 9444016571 Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE (#8149)
Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE.
2022-10-24 12:46:18 +00:00
Juan Fernandez 582772fbee Fixed bug with duplicated vulnerabilities (#8089)
Previous code assumed CVEs were unique per NVD file, which was   incorrect.
2022-10-05 19:48:37 -04:00
Martin Angers ec75fb10b2 Fix CI tests for Go (#8079) 2022-10-04 15:48:14 -04:00
Juan Fernandez 812d3c85de Fixes various bugs with NVD vulnerability detection (#7963)
- Improved NVD CPE matching process.
- Fixed bug with the 'software/<id>' endpoint not showing the generated_cpe value.
2022-10-04 07:04:48 -04:00
Tomas Touceda 8457e55b53 Bump go to 1.19.1 (#7690)
* Bump go to 1.19.1

* Bump remaining go-version to the 1.19.1

* Add extra paths for test-go

* Oops, putting the right path in the right place

* gofmt file

* gofmt ALL THE THINGS

* Moar changes

* Actually, go.mod doesn't like minor versions
2022-09-12 20:32:43 -03:00
Juan Fernandez 98e4d6e0bc Feature 7394: Use MSRC parser to generate security bulletin artifacts (#7491)
Generate security artifacts using the MSRC parser.
2022-09-12 15:17:13 -04:00
Michal Nicpon 0709d1bc5c improve vuln cpe matching on macos (#6985)
* add cpe translations
* fix matching on target_sw
2022-09-01 10:02:07 -06:00
Juan Fernandez 2699c22143 Feature 7077: Add MSRC feed parser (#7424)
Added parser for MSRC
2022-08-30 16:39:50 -04:00
Juan Fernandez 3048a07fd1 Feature 7076: Ingest installed windows updates (#7138)
* Ingest installed Windows updates and store them in the windows_updates table. 
* Added config option for enabling/disabling Windows update ingestion and Windows OS vuln. detection.
2022-08-26 14:55:03 -04:00
Juan Fernandez 0fab2fe0eb Feature 6538: Remove cpe_id from the software_cve table (#6587)
* Don't insert dummy values for non-match CPEs, exclude OVAL platforms when getting software iterator.
* Removed CPEID from SoftwareVulnerability type and software_cve.cpe_id column.
* Removed GeneratedCPEID property
2022-08-24 13:10:58 -04:00
Juan Fernandez 966bfbf85e Feature 6487: Deprecate cpe_id from software_cve table (#6562)
Part 2/3 of the removal of the cpe_id column from the software_cve table in favor of using the newly added software_id coumn.
2022-08-04 09:24:44 -04:00
Juan Fernandez af80406b59 Bug 6777: Make the regexp used for detecting the oval platform major, min versions more permissive (#6800)
When getting the major/minor version for OVAL processing, handle the LTS post-fix for Ubuntu platforms
2022-07-21 12:55:07 -04:00
Juan Fernandez 5b90d11c85 Bug 6508: Don't show non-CVE entries (#6686)
Only include vulnerabilities (CVEs) in Fleet UI and API
2022-07-18 09:23:32 -04:00
Juan Fernandez 79bf51b03c Feature 6242: Use oval to detect vulnerabilities on Fedora hosts (#6330)
* Feature 6242: Scan Fedora hosts using OVAL definitions
2022-06-24 11:02:51 -04:00
Juan Fernandez 9d01ba33c6 Feature 6096: Scan RHEL/CentOS hosts using OVAL definitions (#6241)
Extended the OVAL parser/analyzer so that we can scan RHEL based systems.
2022-06-23 16:44:45 -04:00
Eng Zer Jun abe33f1d8d test: use T.TempDir to create temporary test directory (#6080)
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-06-13 10:20:38 -03:00
Lucas Manuel Rodriguez 4cfeaa1580 Do not use golangci action for better reproducibility (use make lint-go) (#6175)
* Do not use golangci action for better reproducibility

* Add fix to trigger build

* Fix all reported issues

* fix more lint errors

* Add missing import

* Remove unused method

* Remove change not necessary
2022-06-10 18:52:24 -03:00
Lucas Manuel Rodriguez 3dd89bc0a1 Fix flaky test TestVulnerabilityDataStream (#6162) 2022-06-09 10:36:37 -03:00
Michal Nicpon bac168e3fd fix mem usage cve sync (#6131) 2022-06-08 13:15:44 -06:00
Juan Fernandez 337734fce1 Bug: Issue while parsing dpkg objects with inline names (#6146)
* Bug: Issue while parsing dpkg objects with inline names

Dpkg objects can also define their names inline, not only using variable references.
2022-06-08 14:29:58 -04:00
Juan Fernandez ef73039559 Improve vulnerability detection for Ubuntu (#6102)
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts

To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
2022-06-07 21:09:47 -04:00
Michal Nicpon 1044182b7b fix epss probability (#6083) 2022-06-03 11:37:47 -06:00
Michal Nicpon 30a864c8d2 Sync CVE scores periodically (#5838) 2022-06-01 10:06:57 -06:00
Michal Nicpon 983f9e59b3 Include CVE scores when listing software (#5673) 2022-05-20 10:58:40 -06:00
Lucas Manuel Rodriguez 114b678e26 test: Add retry to the correct part that has sync enabled (#5688) 2022-05-11 10:11:51 -03:00
Lucas Manuel Rodriguez fda79a8770 Run network test serially to prevent timeouts on Github CI (#5557)
* Run network test serially to prevent timeouts on Github CI

* Revert lint changes

* Add simple file lock

* Revert test change

* Clarify error check
2022-05-10 11:52:33 -03:00
Martin Angers 36702ede8d Make recent vulnerabilities max age configurable. (#5081) 2022-04-12 14:48:15 -04:00
Tomas Touceda f2aba83a73 Extend vulnerability age to 30 (#4901) 2022-04-06 11:42:02 -03:00
Michal Nicpon 779623934c Fix update invite email conflict (#4751) 2022-03-28 09:38:57 -06:00
Martin Angers 80079e4fd7 Fix race in mock datastore for recent_vulns test (#4230) 2022-02-16 09:23:27 -05:00
Lucas Manuel Rodriguez be72dc356c Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Tomas Touceda c45115a915 Properly handle path in CVE URL prefix (#4174) 2022-02-11 14:10:13 -03:00
Martin Angers 6e2ba62744 Trigger webhooks for recently published vulnerabilities (#3941) 2022-02-02 16:34:37 -05:00
Tomas Touceda b90e2e2e3d Issue 3901 match target sw (#3982)
* Allow to search with target_sw with a period

* Remove tests for a different thign

* Add a test for programs as a source as well

* Use MATCH again and add proper tests
2022-02-02 17:17:41 -03:00
Martin Angers c997f853e5 Make creation of http.Client uniform across the codebase (#3097) 2021-11-24 15:56:54 -05:00