Fixes to fleetctl debug connection and TLS certs documentation (#20166)
#6085 - [X] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://fleetdm.com/docs/contributing/committing-changes#changes-files) for more information. - [X] Added/updated tests - [x] Manual QA for all new/changed functionality
This commit is contained in:
Lucas Manuel Rodriguez
@@ -389,7 +389,7 @@ jobs:
|
||||
|
||||
- name: Uninstall pkg
|
||||
run: |
|
||||
./orbit/tools/cleanup/cleanup_macos.sh
|
||||
sudo ./orbit/tools/cleanup/cleanup_macos.sh
|
||||
|
||||
orbit-ubuntu:
|
||||
timeout-minutes: 60
|
||||
|
||||
@@ -238,7 +238,7 @@ jobs:
|
||||
|
||||
- name: Uninstall Orbit
|
||||
run: |
|
||||
./orbit/tools/cleanup/cleanup_macos.sh
|
||||
sudo ./orbit/tools/cleanup/cleanup_macos.sh
|
||||
|
||||
orbit-ubuntu:
|
||||
timeout-minutes: 10
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Certificates in fleetd
|
||||
|
||||
There are three components in fleetd connecting to the Fleet server using TLS: `orbit`, `Fleet Desktop` and `osqueryd`.
|
||||
This article aims to describe how TLS CA root certificates are configured in fleetd to connect to a Fleet server securely.
|
||||
|
||||
## Default
|
||||
|
||||
The default behavior is using the `fleetctl package` command without the `--fleet-certificate` flag.
|
||||
|
||||
- By default, `orbit` and `Fleet Desktop` will use the system's CA root store to connect to Fleet.
|
||||
- `osqueryd` doesn't support using the system's CA root store, it requires passing in a certificate file with the root CA store (via the `--tls_server_certs` flag). The `fleetctl` executable contains an embedded `certs.pem` file generated from https://curl.se/docs/caextract.html [0]. When generating a fleetd package with `fleetctl package` such embedded `certs.pem` file is added to the package [1]. Fleetd configures `osqueryd` to use the `certs.pem` file as CA root store by setting the `--tls_server_certs` argument to such path.
|
||||
|
||||
## Using `--fleet-certificate` in `fleetctl package`
|
||||
|
||||
When using `--fleet-certificate` in `fleetctl package`, such certificate file is used as a CA root store by `orbit`, `Fleet Desktop` and `osqueryd` (the system's CA store is not used when generating the fleetd package this way).
|
||||
|
||||
## Issues with internal and/or intermediates certificates
|
||||
|
||||
TLS clients require the CA root and all intermediate certificates that signed the leaf server certificate to be verified.
|
||||
This means that if the bundled certificate in fleetd [1] doesn't have intermediate certificates that signed the leaf certificate, then the Fleet server will have to be configured to serve the "fullchain".
|
||||
Here's a list of some scenarios assuming your Fleet server certificate has an intermediate signing certificate:
|
||||
- ✅ Using fullchain in the Fleet server and root CA only client side.
|
||||
- ✅ Using fullchain in the Fleet server and root+intermediate bundle client side.
|
||||
- ✅ Using the leaf certificate in the Fleet server and root+intermediate bundle client side.
|
||||
- ✅ Using the leaf certificate + intermediate bundle in the Fleet server and root CA only client side.
|
||||
- ❌ Using the leaf certificate in the Fleet server and root CA only client side. In this scenario the client side (fleetd) doesn't know of the intermediate certificate and thus cannot verify it.
|
||||
|
||||
We've seen TLS certificate issues in the following configurations: (for more information see https://github.com/fleetdm/fleet/issues/6085):
|
||||
- Certificates signed by internal CA/intermediates.
|
||||
- Certificates issued by Let's Encrypt (that do not serve the fullchain certificate).
|
||||
|
||||
When there are certificate issues you will see the following kind of errors in server logs:
|
||||
```
|
||||
2024/07/05 15:03:52 http: TLS handshake error from <remote_ip>:<remote_port>: remote error: tls: bad certificate
|
||||
2024/07/05 15:03:53 http: TLS handshake error from <remote_ip>:<remote_port>: local error: tls: bad record MAC
|
||||
```
|
||||
and the following kind of errors on the client side (fleetd):
|
||||
```
|
||||
2024-07-05T15:04:52-03:00 DBG get config error="POST /api/fleet/orbit/config: Post \"https://fleet.example.com/api/fleet/orbit/config\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
|
||||
```
|
||||
```
|
||||
W0705 15:16:44.739495 1251102656 init.cpp:760] Error reading config: Request error: certificate verify failed
|
||||
```
|
||||
|
||||
To troubleshoot issues with certificates you can use `fleetctl debug connection` command, e.g.:
|
||||
```sh
|
||||
fleetctl debug connection \
|
||||
--fleet-certificate ./your-ca-root.pem \
|
||||
https://fleet.example.com
|
||||
```
|
||||
|
||||
[0]: We have a Github CI action that runs daily that updates the [certs.pem on the repository](https://github.com/fleetdm/fleet/blob/main/orbit/pkg/packaging/certs.pem) whenever there's a new version of `cacert.pem` in https://curl.se/docs/caextract.html. Such file is embedded into the `fleetctl` executable and used when generating fleetd packages.
|
||||
[1]: The bundled certificate in fleetd is installed in `/opt/orbit` in macOS/Linux and `C:\Program Files\Orbit` on Windows. By default its name is `certs.pem`, but it will have a different name if the `--fleet-certificate` flag was used when generating the package (`fleetctl package`).
|
||||
|
||||
|
||||
<meta name="articleTitle" value="Certificates in fleetd">
|
||||
<meta name="authorFullName" value="Lucas Manuel Rodriguez">
|
||||
<meta name="authorGitHubUsername" value="lucasmrod">
|
||||
<meta name="category" value="guides">
|
||||
<meta name="publishedOn" value="2024-08-09">
|
||||
<meta name="articleImageUrl" value="../website/assets/images/articles/apple-developer-certificates-on-linux-for-configuration-profile-signing-1600x900@2x.png">
|
||||
<meta name="description" value="TLS certificates in fleetd">
|
||||
@@ -0,0 +1,2 @@
|
||||
* Fixed `fleetctl debug connection` to support server TLS certificates with intermediates.
|
||||
* Added support to `fleetctl debug connection` to test TLS connection with the embedded certs.pem in the fleetctl executable (default root CA used to generate fleetd packages). This can help find issues during package generation instead of during package installation.
|
||||
+57
-17
@@ -12,9 +12,11 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/fleetdm/fleet/v4/orbit/pkg/packaging"
|
||||
"github.com/fleetdm/fleet/v4/pkg/certificate"
|
||||
"github.com/fleetdm/fleet/v4/pkg/secure"
|
||||
"github.com/fleetdm/fleet/v4/server/fleet"
|
||||
@@ -447,10 +449,39 @@ or provide an <address> argument to debug: fleetctl debug connection localhost:8
|
||||
cc.Address = "https://" + cc.Address
|
||||
}
|
||||
|
||||
if certPath := getFleetCertificate(c); certPath != "" {
|
||||
// if a certificate is provided, use it as root CA
|
||||
cc.RootCA = certPath
|
||||
cc.TLSSkipVerify = false
|
||||
usingHTTPS := strings.HasPrefix(cc.Address, "https://")
|
||||
|
||||
//
|
||||
// Scenarios:
|
||||
// - If a --fleet-certificate is provided, use it as root CA.
|
||||
// - If a --fleet-certificate is not provided, but a cc.RootCA is set in the configuration, use it as root CA.
|
||||
// - If a --fleet-certificate is not provided and there isn't a cc.RootCA set in the configuration, use the embedded certs as root CA.
|
||||
//
|
||||
usingEmbeddedCA := false
|
||||
if usingHTTPS {
|
||||
certPath := getFleetCertificate(c)
|
||||
if certPath != "" {
|
||||
// if a certificate is provided, use it as root CA
|
||||
cc.RootCA = certPath
|
||||
cc.TLSSkipVerify = false
|
||||
} else { // --fleet-certificate is not set
|
||||
if cc.RootCA == "" {
|
||||
// If a certificate is not provided and a cc.RootCA is not set in the configuration,
|
||||
// then use the embedded root CA which is used by osquery to connect to Fleet.
|
||||
usingEmbeddedCA = true
|
||||
tmpDir, err := os.MkdirTemp("", "")
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create temporary directory: %w", err)
|
||||
}
|
||||
certPath := filepath.Join(tmpDir, "certs.pem")
|
||||
if err := os.WriteFile(certPath, packaging.OsqueryCerts, 0o600); err != nil {
|
||||
return fmt.Errorf("failed to create temporary certs.pem file: %s", err)
|
||||
}
|
||||
defer os.RemoveAll(certPath)
|
||||
cc.RootCA = certPath
|
||||
cc.TLSSkipVerify = false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
cli, baseURL, err := rawHTTPClientFromConfig(cc)
|
||||
@@ -460,16 +491,20 @@ or provide an <address> argument to debug: fleetctl debug connection localhost:8
|
||||
|
||||
// print a summary of the address and TLS context that is investigated
|
||||
fmt.Fprintf(c.App.Writer, "Debugging connection to %s; Configuration context: %s; ", baseURL.Hostname(), configContext)
|
||||
rootCA := "(system)"
|
||||
if cc.RootCA != "" {
|
||||
rootCA = cc.RootCA
|
||||
|
||||
if usingHTTPS {
|
||||
rootCA := cc.RootCA
|
||||
if usingEmbeddedCA {
|
||||
rootCA += " (embedded certs used by default to generate fleetd packages)"
|
||||
}
|
||||
fmt.Fprintf(c.App.Writer, "Root CA: %s; ", rootCA)
|
||||
|
||||
tlsMode := "secure"
|
||||
if cc.TLSSkipVerify {
|
||||
tlsMode = "insecure"
|
||||
}
|
||||
fmt.Fprintf(c.App.Writer, "TLS: %s.\n", tlsMode)
|
||||
}
|
||||
fmt.Fprintf(c.App.Writer, "Root CA: %s; ", rootCA)
|
||||
tlsMode := "secure"
|
||||
if cc.TLSSkipVerify {
|
||||
tlsMode = "insecure"
|
||||
}
|
||||
fmt.Fprintf(c.App.Writer, "TLS: %s.\n", tlsMode)
|
||||
|
||||
// Check that the url's host resolves to an IP address or is otherwise
|
||||
// a valid IP address directly.
|
||||
@@ -479,14 +514,19 @@ or provide an <address> argument to debug: fleetctl debug connection localhost:8
|
||||
fmt.Fprintf(c.App.Writer, "Success: can resolve host %s.\n", baseURL.Hostname())
|
||||
|
||||
// Attempt a raw TCP connection to host:port.
|
||||
if err := dialHostPort(c.Context, timeoutPerCheck, baseURL.Host); err != nil {
|
||||
dialURL := baseURL.Host
|
||||
if baseURL.Port() == "" {
|
||||
fmt.Fprintf(c.App.Writer, "Assumming port 443.\n")
|
||||
dialURL += ":443"
|
||||
}
|
||||
if err := dialHostPort(c.Context, timeoutPerCheck, dialURL); err != nil {
|
||||
return fmt.Errorf("Fail: dial server: %w", err)
|
||||
}
|
||||
fmt.Fprintf(c.App.Writer, "Success: can dial server at %s.\n", baseURL.Host)
|
||||
|
||||
if cert := getFleetCertificate(c); cert != "" {
|
||||
// Run some validations on the TLS certificate.
|
||||
if err := checkFleetCert(c.Context, timeoutPerCheck, cert, baseURL.Host); err != nil {
|
||||
// Run some validations on the TLS certificate.
|
||||
if usingHTTPS {
|
||||
if err := checkFleetCert(c.Context, timeoutPerCheck, cc.RootCA, baseURL.Host); err != nil {
|
||||
return fmt.Errorf("Fail: certificate: %w", err)
|
||||
}
|
||||
fmt.Fprintln(c.App.Writer, "Success: TLS certificate seems valid.")
|
||||
|
||||
@@ -44,7 +44,8 @@ oug6edBNpdhp8r2/4t6n3AouK0/zG2naAlmXV0JoFuEvy2bX0BbbbPg+v4WNZIsC
|
||||
)
|
||||
|
||||
func TestDebugConnectionCommand(t *testing.T) {
|
||||
t.Run("without certificate", func(t *testing.T) {
|
||||
t.Run("without certificate, plain http server", func(t *testing.T) {
|
||||
// Plain HTTP server
|
||||
_, ds := runServerWithMockedDS(t)
|
||||
|
||||
ds.VerifyEnrollSecretFunc = func(ctx context.Context, secret string) (*fleet.EnrollSecret, error) {
|
||||
|
||||
@@ -58,6 +58,20 @@ You can use your software management tool of choice to distribute Fleet's agent
|
||||
|
||||
You can include Fleet Desktop in Fleet's agent (fleetd) by including `--fleet-desktop` in the `fleetctl package` command.
|
||||
|
||||
### Debug TLS certificates and connection to Fleet
|
||||
|
||||
You can use `fleetctl debug connection` to troubleshoot issues with server/client TLS certificates, e.g.:
|
||||
```sh
|
||||
# Test TLS connection using the CA root file that will be embedded on fleetd packages:
|
||||
fleetctl debug connection \
|
||||
https://fleet.example.com
|
||||
|
||||
# Test TLS connection using a custom CA root file:
|
||||
fleetctl debug connection \
|
||||
--fleet-certificate ./your-ca-root.pem \
|
||||
https://fleet.example.com
|
||||
```
|
||||
|
||||
## Enroll Chromebooks
|
||||
|
||||
> The fleetd Chrome browser extension is supported on ChromeOS operating systems that are managed using [Google Admin](https://admin.google.com). It is not intended for non-ChromeOS hosts with the Chrome browser installed.
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
# TUF Notes
|
||||
|
||||
Orbit uses https://theupdateframework.io/ for automatic updates.
|
||||
This guide has some notes on how Orbit uses such system.
|
||||
|
||||
## How fleetctl and orbit use TUF
|
||||
|
||||
A TUF client needs trusted signing keys and a URL to fetch new updates.
|
||||
Both fleetctl (when using the `package` command) and orbit are TUF clients:
|
||||
- `fleetctl package` uses a TUF server to fetch targets and assemble a installer Orbit package.
|
||||
- `orbit` uses a TUF server to keep its components up-to-date.
|
||||
|
||||
### fleetctl package
|
||||
|
||||
To generate installer packages, the `fleetctl package` command needs:
|
||||
1. Root TUF keys
|
||||
2. Update URL
|
||||
|
||||
By default, fleetctl uses a hardcoded TUF root key and Fleet DM's TUF URL, see [update.go#L32-L33](https://github.com/fleetdm/fleet/blob/6a437aaa358350d88b13905c2dc357c26c065fd5/orbit/pkg/update/update.go#L32-L33).
|
||||
you can set alternative root TUF keys and update URL via the `--update-roots` and `--update-url` options.
|
||||
|
||||
Sample command using alternative root keys and update URL:
|
||||
```sh
|
||||
fleetctl package \
|
||||
--type=pkg \
|
||||
--fleet-url=https://example.com:8080 \
|
||||
--enroll-secret=foobar \
|
||||
'--update-roots={"signed":{"_type":"root","spec_version":"1.0","version":1,"expires":"2032-10-16T08:09:53-03:00","keys":{"2b757c4827a3bafafff84baee96671d0101d91a71305e897887a7bc23135863d":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"37368304a31a89f84b6c60cf4baeb312036b516cd44584cabd28c748ec7d1acc"}},"4d05ec4fad838337a596ca9488f673828ab4a6f598f960e6bfefa652a94d5e5e":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"ef54804d10c3e76e03289f81897f25495766046badaed98ab74844efb85450e9"}},"603d02b3f0a4b540ad8cfb0650ec2f9818eac55a01faa74fdcb2f7fcee2e99f3":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"6509f680ed6ea7a9196cee411213daede1a94e950ea700c200d6b1de2085e178"}},"81dd8f7c50b98fe1c01c4b77452c459228d064560692d33084cb0b04ea74d5ae":{"keytype":"ed25519","scheme":"ed25519","keyid_hash_algorithms":["sha256","sha512"],"keyval":{"public":"2765dcf1630f93fd78a7eb9552ccd2f8a5f6d5697ed74aff8b9dc2ec0e5b476b"}}},"roles":{"root":{"keyids":["603d02b3f0a4b540ad8cfb0650ec2f9818eac55a01faa74fdcb2f7fcee2e99f3"],"threshold":1},"snapshot":{"keyids":["2b757c4827a3bafafff84baee96671d0101d91a71305e897887a7bc23135863d"],"threshold":1},"targets":{"keyids":["4d05ec4fad838337a596ca9488f673828ab4a6f598f960e6bfefa652a94d5e5e"],"threshold":1},"timestamp":{"keyids":["81dd8f7c50b98fe1c01c4b77452c459228d064560692d33084cb0b04ea74d5ae"],"threshold":1}},"consistent_snapshot":false},"signatures":[{"keyid":"603d02b3f0a4b540ad8cfb0650ec2f9818eac55a01faa74fdcb2f7fcee2e99f3","sig":"05292c2c39d5073673a97f2f3b54988e64b9dc8d60eecaf4f2cf575888bd0083b50259df4fa0c33efa8ec528fb4af15ec0c6cd98e4b4b6959b73783bc3a22c06"}]}' \
|
||||
--update-url=http://mytuf-server:8081
|
||||
```
|
||||
|
||||
The `fleetctl package` command will trust the provided (or hardcoded) root key and download (+verify) the latest version of the root metadata file from the TUF server. Such file is signed by the root key and specifies the other top-level roles.
|
||||
|
||||
The `tuf-metadata.json` file is placed in the generated installer package (stored in the Orbit root path) and will be used by Orbit at runtime (see below).
|
||||
|
||||
### Orbit
|
||||
|
||||
Orbit trusts such the packaged `tuf-metadata.json` and uses it as "root of trust" to bootstrap the TUF system. You can also specify an alternative TUF URL via the `--update-url` argument (this is needed in case of domain change of the TUF file server).
|
||||
|
||||
#### Edge case when tuf-metadata.json is missing
|
||||
|
||||
If the `tuf-metadata.json` file is not in the expected location (e.g. was moved or deleted for some reason), then Orbit will attempt to use the [hard-coded Fleet DM's root key](https://github.com/fleetdm/fleet/blob/6a437aaa358350d88b13905c2dc357c26c065fd5/orbit/pkg/update/update.go#L33) to bootstrap the TUF system. This is handy for systems that use our (Fleet DM) TUF server in case the `tuf-metadata.json` is gone for some reason.
|
||||
@@ -279,10 +279,10 @@ func writeOsqueryFlagfile(opt Options, orbitRoot string) error {
|
||||
}
|
||||
|
||||
// Embed the certs file that osquery uses so that we can drop it into our installation packages.
|
||||
// This file copied from https://raw.githubusercontent.com/osquery/osquery/master/tools/deployment/certs.pem
|
||||
// This file is generated and updated by .github/workflows/update-certs.yml.
|
||||
//
|
||||
//go:embed certs.pem
|
||||
var osqueryCerts []byte
|
||||
var OsqueryCerts []byte
|
||||
|
||||
func writeOsqueryCertPEM(opt Options, orbitRoot string) error {
|
||||
path := filepath.Join(orbitRoot, "certs.pem")
|
||||
@@ -290,7 +290,7 @@ func writeOsqueryCertPEM(opt Options, orbitRoot string) error {
|
||||
return fmt.Errorf("mkdir: %w", err)
|
||||
}
|
||||
|
||||
if err := os.WriteFile(path, osqueryCerts, 0o644); err != nil {
|
||||
if err := os.WriteFile(path, OsqueryCerts, 0o644); err != nil {
|
||||
return fmt.Errorf("write file: %w", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -57,9 +57,15 @@ func ValidateConnectionContext(ctx context.Context, pool *x509.CertPool, targetU
|
||||
}
|
||||
|
||||
cert := state.PeerCertificates[0]
|
||||
intermediates := x509.NewCertPool()
|
||||
for _, intermediate := range state.PeerCertificates[1:] {
|
||||
intermediates.AddCert(intermediate)
|
||||
}
|
||||
|
||||
if _, err := cert.Verify(x509.VerifyOptions{
|
||||
DNSName: parsed.Hostname(),
|
||||
Roots: pool,
|
||||
DNSName: parsed.Hostname(),
|
||||
Roots: pool,
|
||||
Intermediates: intermediates,
|
||||
}); err != nil {
|
||||
return ctxerr.Wrap(ctx, err, "verify certificate")
|
||||
}
|
||||
|
||||
@@ -172,7 +172,6 @@ func newBaseClient(
|
||||
// Ignoring "G402: TLS InsecureSkipVerify set true", needed for development/testing.
|
||||
tlsConfig.InsecureSkipVerify = true //nolint:gosec
|
||||
default:
|
||||
// Use only the system certs (doesn't work on Windows)
|
||||
rootCAPool, err = x509.SystemCertPool()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("loading system cert pool: %w", err)
|
||||
|
||||
Reference in New Issue
Block a user