Commit Graph
550 Commits
Author SHA1 Message Date
Luke Heath 69e128c902 Update freeze instructions in engineering handbook (#20524) 2024-07-18 12:38:54 -07:00
Luke Heath 1032841800 Adding changes for Fleet v4.54.0 (#20280) (#20561) 2024-07-17 16:50:26 -07:00
Zach Wasserman 1de5599e8e Improvements to logging in mdmproxy (#20201)
- Add --debug flag to log Fleet requests and responses
- Add --log-skipped flag to disable logging most garbage requests by
default
2024-07-17 13:35:45 -07:00
ecf2346ace Add support for Linux ARM64 (#19931)
#1845

Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Roberto Dip <rroperzh@gmail.com>
2024-07-17 16:07:59 -04:00
Lucas Manuel Rodriguez 8cbcf3f9f7 Add fake certificates for testing TLS issues (#20390)
Changes to help QA #6085.
2024-07-16 13:21:39 -03:00
Victor Lyuboslavsky 7bcd61a8bd Distributed lock and store calendar_events UUID as binary in MySQL (#20277)
#19352

Fix for code review comment:
https://github.com/fleetdm/fleet/pull/20156#discussion_r1668421504

Also includes changes from https://github.com/fleetdm/fleet/pull/20252

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
2024-07-10 08:49:05 -05:00
Luke Heath 12c3d76b24 Update go.mod (#20273)
Fixes code scanning warning: 

As of Go 1.21, toolchain versions [must use the 1.N.P
syntax](https://go.dev/doc/toolchain#version).

1.22 in tools/terraform/go.mod does not match this syntax and there is
no additional toolchain directive, which may cause some go commands to
fail.
2024-07-09 11:44:35 -07:00
Tim Lee 383c59bbd6 Add retries to VM Auto Enroll Script (#20153) 2024-07-09 11:51:31 -06:00
Lucas Manuel Rodriguez 28ca463d13 iOS/iPadOS as platforms/labels (#20126)
#19963 

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Manual QA for all new/changed functionality

---

# API changes for dashboard UI changes

## Main dashboard page

`GET /api/latest/fleet/host_summary?low_disk_space=32` (see
`ios`/`ipados` platforms and `iOS`/`iPadOS` labels)
```json
{
  "totals_hosts_count": 9,
  "online_count": 0,
  "offline_count": 9,
  "mia_count": 0,
  "missing_30_days_count": 0,
  "new_count": 0,
  "all_linux_count": 2,
  "low_disk_space_count": 3,
  "builtin_labels": [
    {
      "id": 1,
      "name": "macOS 14+ (Sonoma+)",
      "description": "macOS hosts with version 14 and above",
      "label_type": "builtin"
    },
    {
      "id": 7,
      "name": "All Hosts",
      "description": "All hosts which have enrolled in Fleet",
      "label_type": "builtin"
    },
    {
      "id": 8,
      "name": "macOS",
      "description": "All macOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 9,
      "name": "Ubuntu Linux",
      "description": "All Ubuntu hosts",
      "label_type": "builtin"
    },
    {
      "id": 10,
      "name": "CentOS Linux",
      "description": "All CentOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 11,
      "name": "MS Windows",
      "description": "All Windows hosts",
      "label_type": "builtin"
    },
    {
      "id": 12,
      "name": "Red Hat Linux",
      "description": "All Red Hat Enterprise Linux hosts",
      "label_type": "builtin"
    },
    {
      "id": 13,
      "name": "All Linux",
      "description": "All Linux distributions",
      "label_type": "builtin"
    },
    {
      "id": 14,
      "name": "chrome",
      "description": "All Chrome hosts",
      "label_type": "builtin"
    },
    {
      "id": 15,
      "name": "iOS",
      "description": "All iOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 16,
      "name": "iPadOS",
      "description": "All iPadOS hosts",
      "label_type": "builtin"
    }
  ],
  "platforms": [
    {
      "platform": "darwin",
      "hosts_count": 3
    },
    {
      "platform": "ios",
      "hosts_count": 1
    },
    {
      "platform": "ipados",
      "hosts_count": 1
    },
    {
      "platform": "rhel",
      "hosts_count": 1
    },
    {
      "platform": "ubuntu",
      "hosts_count": 1
    },
    {
      "platform": "windows",
      "hosts_count": 2
    }
  ]
}
```

## After selecting a platform

`GET /api/latest/fleet/host_summary?platform=ios&low_disk_space=100`
(similar with `ipados`)
```json
{
  "totals_hosts_count": 1,
  "online_count": 0,
  "offline_count": 1,
  "mia_count": 0,
  "missing_30_days_count": 0,
  "new_count": 0,
  "all_linux_count": 0,
  "low_disk_space_count": 1,
  "builtin_labels": [
    {
      "id": 1,
      "name": "macOS 14+ (Sonoma+)",
      "description": "macOS hosts with version 14 and above",
      "label_type": "builtin"
    },
    {
      "id": 7,
      "name": "All Hosts",
      "description": "All hosts which have enrolled in Fleet",
      "label_type": "builtin"
    },
    {
      "id": 8,
      "name": "macOS",
      "description": "All macOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 9,
      "name": "Ubuntu Linux",
      "description": "All Ubuntu hosts",
      "label_type": "builtin"
    },
    {
      "id": 10,
      "name": "CentOS Linux",
      "description": "All CentOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 11,
      "name": "MS Windows",
      "description": "All Windows hosts",
      "label_type": "builtin"
    },
    {
      "id": 12,
      "name": "Red Hat Linux",
      "description": "All Red Hat Enterprise Linux hosts",
      "label_type": "builtin"
    },
    {
      "id": 13,
      "name": "All Linux",
      "description": "All Linux distributions",
      "label_type": "builtin"
    },
    {
      "id": 14,
      "name": "chrome",
      "description": "All Chrome hosts",
      "label_type": "builtin"
    },
    {
      "id": 15,
      "name": "iOS",
      "description": "All iOS hosts",
      "label_type": "builtin"
    },
    {
      "id": 16,
      "name": "iPadOS",
      "description": "All iPadOS hosts",
      "label_type": "builtin"
    }
  ],
  "platforms": [
    {
      "platform": "ios",
      "hosts_count": 1
    }
  ]
}
```

### To populate list of MDM solutions of a selected platform

`GET /api/latest/fleet/hosts/summary/mdm\?platform=ios` (similar with
`ipados`)

```json
{
  "counts_updated_at": "2024-06-27T21:56:45Z",
  "mobile_device_management_enrollment_status": {
    "enrolled_manual_hosts_count": 0,
    "enrolled_automated_hosts_count": 1,
    "pending_hosts_count": 0,
    "unenrolled_hosts_count": 0,
    "hosts_count": 1
  },
  "mobile_device_management_solution": [
    {
      "id": 1,
      "name": "Fleet",
      "server_url": "https://lucas-fleet.ngrok.app/mdm/apple/mdm",
      "hosts_count": 1
    }
  ]
}
```

### To populate OS versions of a selected platform

`GET /api/latest/fleet/os_versions?platform=ipados` (similar with `ios`)
```json
{
  "meta": {
    "has_next_results": false,
    "has_previous_results": false
  },
  "count": 1,
  "counts_updated_at": "2024-06-27T21:36:12Z",
  "os_versions": [
    {
      "os_version_id": 7,
      "hosts_count": 1,
      "name": "iPadOS 17.5.1",
      "name_only": "iPadOS",
      "version": "17.5.1",
      "platform": "ipados",
      "vulnerabilities": []
    }
  ]
}
```

## Filtering hosts by the two new `iOS`/`iPadOS` labels

Works the same as with other labels.
2024-07-08 18:05:29 -03:00
Victor Lyuboslavsky df141cdfa4 Check for calendar updates after callbacks from Google (#20156)
#19352 

Video explaining code changes:
https://www.loom.com/share/370200a276b84aa388effd6ebd762e01?sid=038508c4-f3c2-40c0-baf6-6b6df682d1f0

In maintenance windows using Google Calendar, calendar event is now
recreated within 30 seconds if deleted or moved to the past.
- Added new endpoint for Google Calendar:
`/api/_version_/fleet/calendar/webhook/{event_uuid}`
- Added UUID to `calendar_events` table to make webhook lookup more
efficient
- webhook endpoint will only recreate event if needed -- it will not
fire webhook. Webhook is still done by the cron job.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
2024-07-08 10:20:03 -05:00
Martin Angers b077d7e669 Merge main, fix conflicts and reorder migrations 2024-07-03 15:51:49 -04:00
Zach Wasserman b81ef1617b Update Go to 1.22.5 for mdmproxy (#20200)
Fixes a [DoS
vulnerability](https://groups.google.com/g/golang-announce/c/gyb7aM1C9H4)
that affects the http.ReverseProxy used in the MDM proxy.
2024-07-03 09:54:13 -07:00
Martin Angers 2fd25cae48 Merge main, fix conflicts 2024-07-02 08:44:34 -04:00
Robert Fairburn bb1b90a1d8 Update/ensure pinned Dockerfile images (#20141)
Note: This ensures that we are using the latest stable base images
possible and that they are tagged/pinned to images we have confidence
in. Note: Not pinning in `tools/fleet-docker/Dockerfile` was a large
oversight as it was the Dockerfile used in releases.
2024-07-01 19:36:42 -05:00
Luke Heath 93925ecded Release script updates for minor release branching (#19968) 2024-07-01 14:57:11 -07:00
George KarrandRobert Fairburn 415b0a7710 Adding changes for Fleet v4.53.1 (#20130)
Co-authored-by: Robert Fairburn <8029478+rfairburn@users.noreply.github.com>
2024-07-01 14:02:22 -05:00
Luke Heath 972988e39b Adding changes for Fleet v4.53.0 (#19965) 2024-06-28 14:21:58 -07:00
Gabe Lopez 2ba1f87dca Update publish_release.sh (#19994)
replaced auto assignment from George to pezhub (Gabe Lopez)
2024-06-28 10:02:49 -07:00
Martin Angers 81a3569300 Merge main and fix conflicts 2024-06-26 10:43:05 -04:00
Martin Angers c10a4d3f7b CP Exclude Labels: API changes (#19846) 2024-06-25 15:26:28 -04:00
Zach WassermanandRobert Fairburn e4c9712b61 MDM proxy for seamless migrations (#19779)
Implementation for the proxy described in #19387.

---------

Co-authored-by: Robert Fairburn <8029478+rfairburn@users.noreply.github.com>
2024-06-25 09:31:48 -07:00
Roberto Dip 2a9be32ffa update Go to go1.22.4 (#19958)
for #19844

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
2024-06-25 10:55:36 -03:00
George Karr 2106767860 Adding changes for Fleet v4.52.0 (#19916) 2024-06-24 15:58:58 -05:00
dependabot[bot] 971b25cfaf Bump golang.org/x/net from 0.21.0 to 0.23.0 in /tools/terraform (#19911) 2024-06-21 12:27:48 -07:00
Mike YoderandBrock Walters c7ea0125d6 Support for Terraforming Fleet Teams (#18750)
This project adds support for terraforming teams in Fleet. If you have
100+ teams, managing them is is prone to error, mistakes, lost
configuration, and general pain. An industry standard tool like
terraform can unify this configuration as code.

In order to do this, I wrote a terraform provider that on one end talks
to the Fleet api, and on the other end implements an interface for
terraform. More information is in the README.

A small sample `main.tf` file is supplied.

---------

Co-authored-by: Brock Walters <153771548+nonpunctual@users.noreply.github.com>
2024-06-20 12:47:35 -07:00
Roberto Dip 1cc13a09fb 🧹 friday cleanup party: substitute deprecated import of go-kit (#19774)
`go-kit/kit/log` was deprecated and generating warnings

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Manual QA for all new/changed functionality
2024-06-17 10:27:31 -03:00
Lucas Rodriguez 63a6bb8bf3 Merge branch 'main' into feature_19010-ipad-ios-lock-wipe 2024-06-14 12:45:47 -03:00
Lucas Manuel Rodriguez 904e8a6825 Added server_settings.query_report_cap (#19692)
#19600

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-06-14 12:24:01 -03:00
George Karr 4e2f7e53c3 Merge branch 'main' into feature_19010-ipad-ios-lock-wipe 2024-06-13 14:01:02 -05:00
Lucas Manuel Rodriguez af525223f2 Implement support for Wipe for iOS/iPadOS devices (#19704)
Backend changes for #19010.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Manual QA for all new/changed functionality
2024-06-13 12:26:02 -03:00
Tim Lee 6d2471d5e5 vscode false positivies (#19508)
#18733 

Added a rule for the affected CVEs and made a change to the `IfIgnore`
func. Manually tested CVE removals with `/tools/nvd/nvdvuln`. Existing
test coverage ensures the rule doesn't cause false negatives on vscode
extensions.

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [] Added/updated tests
- [X] Manual QA for all new/changed functionality
2024-06-13 11:03:53 -04:00
George KarrandLuke Heath 5af2f9751a Adding changes for Fleet v4.51.1 (#19649)
Co-authored-by: Luke Heath <luke@fleetdm.com>
2024-06-12 16:27:18 -05:00
Lucas Manuel RodriguezandVictor Lyuboslavsky 01dd0c7c4d Update osquery flags to 5.12.2 (#19338)
#17375

Updating the osquery flags for 5.12.2
And making the changes so that we can just run `cd server/fleet/ && go
generate` on a macOS host every time we need to do this.

Manual tested by setting `logger_tls_backoff_max` in Fleet agent
settings (which is a 5.12.0 flag).

- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
2024-06-10 17:02:49 -03:00
Luke Heath fbe9c1b498 Adding changes for Fleet v4.51.0 (#19601) 2024-06-10 09:47:23 -07:00
Luke Heath 225fe666d2 Updating release minor version steps (#19248) 2024-06-07 11:04:45 -07:00
0ea339c7e6 Add macOS tcc_access table to fleetd (#19355)
## Addresses #18222 

Table results:
<img width="2991" alt="Screenshot 2024-05-29 at 6 15 21 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/eb87e744-658a-4937-92a4-30b6038a4625">

Optimized querying of host `TCC.db`s as constrained by query `WHERE`
clauses on `uid`:
<img width="1419" alt="Screenshot 2024-06-03 at 6 20 50 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/62475537-61c5-4d75-8b8e-10fe7d21462d">
<img width="1419" alt="Screenshot 2024-06-03 at 6 19 31 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/9901095f-5a61-4671-b45e-5935837f2f0c">
<img width="1419" alt="Screenshot 2024-06-03 at 6 15 01 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/6c6891cc-6baf-4b00-b446-a967d80cacfd">
<img width="1419" alt="Screenshot 2024-06-03 at 6 17 54 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/cadbb76d-abab-405f-8b65-683885e9e164">



- [x] Changes file added for user-visible changes in `orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality on macOS (only
supported OS)

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
2024-06-06 13:52:06 -07:00
Jahziel Villasana-Espinoza a344155cb2 fix: add missing variables 2024-06-03 11:31:01 -04:00
Jahziel Villasana-Espinoza a343eda885 Feat save certs (#19390)
for #10383

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
2024-05-31 16:58:34 -04:00
Sharon Katz 0d0d37d30d Adding changes for Fleet v4.50.2 (#19410) 2024-05-31 16:51:21 -04:00
Roberto Dip 178c0d565f Merge remote-tracking branch 'origin/main' into feat-save-certs 2024-05-30 18:34:40 -03:00
Roberto Dip 545e56d288 19016 ingest certs on start (#19360)
For #19016

This changes all the places where we previously assumed that certs were
hardcoded when the Fleet server started to query the database instead.

The plan is to loadtest afterwards, but as a first preemptive measure,
this adds a caching layer on top the mysql datastore.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality
2024-05-30 18:18:42 -03:00
Sharon Katz 1047ed2622 Adding changes for Fleet v4.50.1 (#19334) 2024-05-30 12:23:59 -04:00
Grant Bilstad 821a2e0e45 added mention of new vuln endpoint (#17912) 2024-05-29 17:08:05 -06:00
Lucas Manuel Rodriguez 984711060f Some docs added while learning DDM for iOS/iPadOS support research (#18561)
Some docs added while working on #18423.
2024-05-28 15:03:39 -03:00
Victor Lyuboslavsky addbceece0 Activity feed webhooks backend (#19261)
This is a redo of the already-approved PR
https://github.com/fleetdm/fleet/pull/19125

The difference is that this merge is into main as opposed to the feature
branch.
2024-05-24 11:25:27 -05:00
Lucas Manuel Rodriguez b7aac2cfab Fix bundle_identifier for common software in osquery-perf (#19023)
bundle_identifier should be unique for the common software (should not
have hostname name in it).
2024-05-24 09:26:42 -03:00
Roberto Dip 987b64a0c7 update to go1.22.3 + dependencies (#19142)
For #16795, this:

- Updates Go to go1.22.3
- Per
https://github.com/fleetdm/fleet/issues/16795#issuecomment-2100450618, I
also ran the following to update the versions requested by @getvictor

```
go get github.com/kataras/golog@v0.1.12
go get github.com/kataras/iris/v12@v12.2.11
go get github.com/sethvargo/go-password@v0.3.0
```

**Notes**

After this is merged people will need to update their Go version. I use
gvm and I did it like:

```
$ gvm install go1.22.3
$ gvm use go1.22.3 --default
```

**Relevant changes**

The release notes mention:

> Previously, the variables declared by a “for” loop were created once
> and updated by each iteration. In Go 1.22, each iteration of the loop
> creates new variables, to avoid accidental sharing bugs.

However, we already have a lint rule (see
https://github.com/fleetdm/fleet/pull/13877) for this scenario, so it
shouldn't affect us.
2024-05-23 16:23:38 -03:00
Luke Heath a540e0e38b Adding changes for Fleet v4.50.0 (#19191) 2024-05-23 11:11:28 -05:00
Luke Heath 2a70605976 Tweaks to Fleet releaser script (#19208) 2024-05-23 11:07:08 -05:00
Lucas Manuel Rodriguez a10befe657 Release fleetd 1.25.0 (#19203) 2024-05-22 14:38:24 -03:00