Discharge7443/fleet-terraform
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
2f09a68085bbd19c413444f7d4128dabf56584dd
fleet-terraform/k8s/README.md
T

25 KiB
Raw Blame History

Fleet deployment with Terraform

This deployment guide has been tested with

  • k3s
    • istio ingress
    • nginx ingress

Usage

1. Create namespace

This terraform will not auto-provision a namespace. You can add one with kubectl create namespace <name> or by creating a YAML file containing a service and applying it to your cluster.

2. Create Secrets

If you have a requirement to pull container images from a Private registry via image_pull_secrets, you can configure them using the instructions below. Additionally, you can instruct the terraform desployment to add the image_pull_secret via module.fleet.image_pull_secrets.

kubectl create secret docker-registry <secret_name> \
  --docker-server=<your_private_registry_url> \
  --docker-username=<your_user_name> \
  --docker-password=<your_password> \
  --docker-email=<your_email> \
  --dry-run=client -o yaml

The output that is generated by the above command will generate a file that looks like the below format, you can copy and paste to save to a file or redirect the output directly into a file.

apiVersion: v1
kind: Secret
metadata:
  name: <secret_name>
  namespace: <namespace_name>
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64_encoded_config>

In order for the deployment to go through successfully, you'll need to create some secrets so Fleet knows how to authenticate against things like MySQL and Redis.

---
apiVersion: v1
kind: Secret
metadata:
  name: redis
  namespace: <namespace>
type: kubernetes.io/basic-auth
stringData:
  password: <redis-password-here>
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql
  namespace: <namespace>
type: kubernetes.io/basic-auth
stringData:
  password: <mysql-password-here>

If you use Fleet's TLS capabilities, TLS connections to the MySQL server, or AWS access secret keys, additional secrets and keys are needed. The name of each Secret must match the value of secret_name for each section in module.fleet located in main.tf. The key of each secret must match the related key value from the values file. For example, to configure Fleet's TLS, you would use a secret like the one below.

---
apiVersion: v1
kind: Secret
metadata:
  name: fleet
  namespace: <namespace_name>
type: kubernetes.io/tls
data:
  tls.crt: |
    <base64-encoded-tls-cert-here>
  tls.key: |
    <base64-encoded-tls-key-here>

If you have a Fleet premium license you would create a secret like the one below.

---
apiVersion: v1
kind: Secret
metadata:
  name: license
  namespace: <namespace>
type: Opaque
stringData:
  license-key: <fleet-license-here>

Once all of your secrets are configured, use kubectl apply -f <secret_file_name.yaml> --namespace <your_namespace> to configure them in the cluster.

3. Further Configuration

To configure how Fleet runs, such as specifying the number of Fleet instances to deploy or changing the logger plugin for Fleet, edit the module.fleet located in main.tf file to your desired settings.

nginx ingress

Assuming no other ingress controllers are deployed to your environment, you can deploy nginx ingress components by executing kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.7.1/deploy/static/provider/baremetal/deploy.yaml

You will need an nginx ingress Service, similar to the one below. To deploy run kubectl apply -f <path_to_service_file>.

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller-loadbalancer
  namespace: ingress-nginx
spec:
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
    - name: https
      port: 443
      protocol: TCP
      targetPort: 443
  type: LoadBalancer

The below will be in preparation for deployment through the terraform apply in step 5.

In main.tf make sure the following variable fleet.tls.enabled = false, otherwise the Fleet terraform deployment will fail. In main.tf make sure the following map is configured with the correct values.

  • ingress.enabled must be true, if you'd like Fleet to deploy the nginx ingress for you.
  • ingress.class_name needs to be set to nginx, but can be changed to values like traefik, if you have another compatible ingress.
  • ingress.hosts.name must have a matching entry in ingress.tls.hosts
    • example: ingress.hosts.name = fleet.example.com and ingress.tls.hosts = fleet.example.com
  • Last, ingress.tls.secret_name must be a valid secret name in your current namespace.*
    • Note: The TLS must contain a valid certificate that matches the hostnames provided for ingress.hosts.name and ingress.tls.hosts.
...
    ingress = {
        enabled = true
        class_name = "nginx"
        annotations = {}
        labels = {}
        hosts = [{
            name = "fleet.localhost.local"
            paths = [{
                path = "/"
                path_type = "ImplementationSpecific"
            }]
        }]
        tls = {
            secret_name = "chart-example-tls"
            hosts = [
                "fleet.localhost.local"
            ]
        }
    }
...
istio ingress

There are different ways to deploy istio to your cluster. We will cover the helm deployment in the official istio documentation.

Assuming no other ingress controllers are deployed to your environment, you can deploy istio components by executing the following commands.

helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update
kubectl create namespace istio-system
helm install istio-base istio/base -n istio-system
helm install istiod istio/istiod -n istio-system
kubectl create namespace istio-ingress
helm install istio-ingress istio/gateway -n istio-ingress --wait

In main.tf make sure the following variable fleet.tls.enabled = false, otherwise the Fleet terraform deployment will fail.

You will need to create a TLS secret specifically for use by the istio ingress gateway like the example below or re-use Fleet secret created in step 2. You can apply the secret with the following command kubectl apply -f <path_to_secret_yaml_file>.

---
apiVersion: v1
kind: Secret
metadata:
  name: fleet
  namespace: <namespace_name>
type: kubernetes.io/tls
data:
  tls.crt: |
    <base64-encoded-tls-cert-here>
  tls.key: |
    <base64-encoded-tls-cert-here>

After the secret has been created, you can create your istio ingress Gateway and istio Virtual Service. In the examble below you should make reference, in the Gateway and VirtualService to hostname (example: fleet.example.com) covered by your TLS certificate stored in the TLS secret created above (example: fleet).

---
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: istio-gateway-fleet
  namespace: <namespace_name>
spec:
  selector:
    istio: ingress # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: fleet # must be the same as secret
    hosts:
    - 'fleet.example.com'
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: fleet-vs
  namespace: <namespace_name>
spec:
  hosts:
  - "fleet.example.com"
  gateways:
  - istio-gateway-fleet
  http:
  - match:
    - uri:
        prefix: "/"
    route:
    - destination:
        port:
          number: 8080
        host: fleet

4. Setup provider.tf

Setup your provider.tf with the correct credentials, whether it's for a self-hosted or managed service k8s deployment. The following link to the kubernetes provider terraform docs has examples documented for AWS EKS, GCP GKE, and Azure.

provider "kubernetes" {
  # config_path = "/path/to/kubeconfig"
  config_path = ""
}

5. Deploy Fleet

From the location where your main.tf resides, execute the following commands.

terraform init
terraform plan
terraform apply

Upgrade Fleet

Fleet should not be running when an upgrade is initiated because database migrations need to take place first. After main.tf has been updated to increment the version of the Fleet image_tag, the following commands can be executed to upgrade Fleet while bringing Fleet down so migrations can run.

terraform init
terraform apply -replace=module.fleet.kubernetes_deployment.fleet

Remove Fleet

1. Tear down Fleet

From the location where your main.tf resides, execute the following commands.

terraform init
terraform destroy

2. Remove all secrets

Using the file configured for your secrets, use kubectl delete -f <secret_file_name.yaml> --namespace <your_namespace> to remove the secrets.

Requirements

No requirements.

Providers

Name Version
kubernetes 2.38.0

Modules

No modules.

Resources

Name Type
kubernetes_cron_job_v1.fleet_vuln_processing_cron_job resource
kubernetes_deployment.fleet resource
kubernetes_ingress_v1.fleet-ingress resource
kubernetes_job.migration resource
kubernetes_role.fleet-role resource
kubernetes_role_binding.fleet-role-binding resource
kubernetes_service.fleet-service resource
kubernetes_service_account.fleet-sa resource
kubernetes_namespace.fleet data source

Inputs

Name Description Type Default Required
affinity_rules Used to configure affinity rules for the fleet deployment, migration job, and vuln-processing cron job. 
object({
    required_during_scheduling_ignored_during_execution  = optional(list(any), [])
    preferred_during_scheduling_ignored_during_execution = optional(list(any), [])
  })
 n/a yes
anti_affinity_rules Used to configure anti-affinity rules for the fleet deployment, migration job, and vuln-processing cron job. 
object({
    required_during_scheduling_ignored_during_execution = optional(list(any), [])
    preferred_during_scheduling_ignored_during_execution = optional(list(any),
      [
        {
          weight = 100
          label_selector = {
            match_expressions = [
              {
                key      = "app"
                operator = "In"
                values   = ["fleet"]
              }
            ]
          }
          topology_key = "kubernetes.io/hostname"
        }
    ])
  })
 n/a yes
cache Used to configure redis specific values for use in the Fleet deployment, migration job, and vuln-processing cron job. 
object({
    enabled      = optional(bool, false)
    address      = optional(string, "redis:6379")
    database     = optional(number, 0)
    use_password = optional(bool, false)
    secret_name  = optional(string, "redis")
    password_key = optional(string, "password")
  })
 n/a yes
database Used to configure database specific values for use in the Fleet deployment, migration job, and vuln-processing cron job. 
object({
    enabled           = optional(bool, false)
    secret_name       = optional(string, "mysql")
    address           = optional(string, "mysql:3306")
    database          = optional(string, "fleet")
    username          = optional(string, "fleet")
    password_key      = optional(string, "password")
    max_open_conns    = optional(number, 50)
    max_idle_conns    = optional(number, 50)
    conn_max_lifetime = optional(number, 0)

    tls = object({
      enabled     = optional(bool, false)
      config      = optional(string, "")
      server_name = optional(string, "")
      ca_cert_key = optional(string, "")
      cert_key    = optional(string, "")
      key_key     = optional(string, "")
    })
  })
 n/a yes
database_read_replica Used to configure database_read_replica specific values for use in the Fleet deployment and vuln-processing cron job. 
object({
    enabled           = optional(bool, false)
    secret_name       = optional(string, "mysql")
    address           = optional(string, "mysql-ro:3306")
    database          = optional(string, "fleet")
    username          = optional(string, "fleet-ro")
    password_key      = optional(string, "ro-password")
    password_path     = optional(string, "")
    max_open_conns    = optional(number, 50)
    max_idle_conns    = optional(number, 50)
    conn_max_lifetime = optional(number, 0)

    tls = optional(object({
      enabled     = optional(bool, false)
      config      = optional(string, "")
      server_name = optional(string, "")
      ca_cert_key = optional(string, "")
      cert_key    = optional(string, "")
      key_key     = optional(string, "")
    }))
  })
 n/a yes
environment_from_config_maps Used to configure additional environment variables from a config map for the fleet deployment and vuln-processing cron job. list(map(string)) [] no
environment_from_secrets Used to configure additional environment variables from a secret for the fleet deployment and vuln-processing cron job. list(map(string)) [] no
environment_variables Used to configure additional environment variables for the fleet deployment and vuln-processing cron job. list(map(string)) [] no
fleet Used to configure Fleet specific values for use in the Fleet deployment, migration job, and vuln-processing cron job. 
object({
    listen_port = optional(number, 8080)
    secret_name = optional(string, "fleet")
    migrations = object({
      auto_apply_sql_migrations = optional(bool, true)
      migration_job_annotations = optional(map(string), {})
      parallelism               = optional(number, 1)
      completions               = optional(number, 1)
      active_deadline_seconds   = optional(number, 900)
      backoff_limit             = optional(number, 6)
      manual_selector           = optional(bool, false)
      restart_policy            = optional(string, "Never")
    })
    tls = object({
      enabled           = optional(bool, false)
      unique_tls_secret = optional(bool, false)
      secret_name       = optional(string, "fleet-tls")
      compatibility     = optional(string, "modern")
      cert_secret_key   = optional(string, "server.cert")
      key_secret_key    = optional(string, "server.key")
    })
    auth = object({
      b_crypto_cost = optional(number, 12)
      salt_key_size = optional(number, 24)
    })
    app = object({
      token_key_size               = optional(number, 24)
      invite_token_validity_period = optional(string, "120h")
    })
    session = object({
      key_size = optional(number, 64)
      duration = optional(string, "2160h")
    })
    logging = object({
      debug          = optional(bool, false)
      json           = optional(bool, false)
      disable_banner = optional(bool, false)
    })
    software_installers = object({
      s3 = object({
        bucket_name         = optional(string, "")
        prefix              = optional(string, "")
        endpoint_url        = optional(string, "")
        force_s3_path_style = optional(bool, false)
        region              = optional(string, "")
        access_key_id       = optional(string, "")
        secret_key          = optional(string, "s3-software-installers")
        sts_assume_role_arn = optional(string, "")
      })
    })
    carving = object({
      s3 = object({
        bucket_name         = optional(string, "")
        prefix              = optional(string, "")
        endpoint_url        = optional(string, "")
        force_s3_path_style = optional(bool, false)
        region              = optional(string, "")
        access_key_id       = optional(string, "")
        secret_key          = optional(string, "s3-bucket")
        sts_assume_role_arn = optional(string, "")
      })
    })
    license = object({
      secret_name = optional(string, "")
      license_key = optional(string, "license-key")
    })
    extra_volumes       = optional(list(any), [])
    extra_volume_mounts = optional(list(any), [])
    security_context = object({
      run_as_user     = optional(number, null)
      run_as_group    = optional(number, null)
      run_as_non_root = optional(bool, true)
    })
  })
 n/a yes
gke Used to configure gke specific values for use in the Fleet deployment, migration job, and vuln-processing cron job. 
object({
    workload_identity_email = optional(string, "")
    cloud_sql = object({
      enable_proxy     = optional(bool, false)
      image_repository = optional(string, "gcr.io/cloudsql-docker/gce-proxy")
      image_tag        = optional(string, "1.17-alpine")
      verbose          = optional(bool, true)
      instance_name    = optional(string, "")
    })
    ingress = object({
      use_managed_certificate = optional(bool, false)
      use_gke_ingress         = optional(bool, false)
      node_port               = optional(number, 0)
      hostnames               = optional(list(string), [""])
    })
  })
 n/a yes
hostname Used as the hostname that you will access fleet on. string "fleet.localhost" no
image_pull_secrets Used to inject image pull secrets for access to a private container registry. 
list(object({
    name = string
  }))
 [] no
image_repository Used to populate the image repository for fleet. string "fleetdm/fleet" no
image_tag Used to populate the fleet version that will be deployed. string "v4.79.0" no
ingress Used to configure values for ingress. 
object({
    enabled     = optional(bool, false)
    class_name  = optional(string, "nginx")
    labels      = optional(map(string), {})
    annotations = optional(map(string), {})
    hosts       = optional(list(any), [])
    tls = object({
      secret_name = optional(string, "")
      hosts       = optional(list(string), [])
    })
  })
 n/a yes
namespace The value for this variable will be used as the name of the namespace that fleet will be deployed to. string "fleet" no
node_selector Used to populate node selector values. map(any) {} no
osquery Used to configure osquery specific values for use in the Fleet deployment, migration job, and vuln-processing cron job. 
object({
    secret_name            = optional(string, "osquery")
    node_key_size          = optional(number, 24)
    label_update_interval  = optional(string, "30m")
    detail_update_interval = optional(string, "30m")
    logging = object({
      status_plugin = optional(string, "filesystem")
      result_plugin = optional(string, "filesystem")
      filesystem = object({
        status_log_file    = optional(string, "osquery_status")
        result_log_file    = optional(string, "osquery_result")
        enable_rotation    = optional(bool, false)
        enable_compression = optional(bool, false)
        volume_size        = optional(string, "20Gi")
      })
      firehose = object({
        region              = optional(string, "")
        access_key_id       = optional(string, "")
        secret_key          = optional(string, "firehose")
        sts_assume_role_arn = optional(string, "")
        status_stream       = optional(string, "")
        result_stream       = optional(string, "")
      })
      kinesis = object({
        region              = optional(string, "")
        access_key_id       = optional(string, "")
        secret_key          = optional(string, "kinesis")
        sts_assume_role_arn = optional(string, "")
        status_stream       = optional(string, "")
        result_stream       = optional(string, "")
      })
      lambda = object({
        region              = optional(string, "")
        access_key_id       = optional(string, "")
        secret_key          = optional(string, "lambda")
        sts_assume_role_arn = optional(string, "")
        status_stream       = optional(string, "")
        result_stream       = optional(string, "")
      })
      pubsub = object({
        project      = optional(string, "")
        status_topic = optional(string, "")
        result_topic = optional(string, "")
      })
    })
  })
 n/a yes
pod_annotations Used to populate the annotations for pods. map(any) {} no
replicas Used to drive the number of fleet deployment replicas. number 3 no
resources Used to populate resource values for the fleet deployment and migration job. 
object({
    limits = optional(object({
      cpu    = optional(string, "1")
      memory = optional(string, "4Gi")
      }), {
      cpu    = "1"
      memory = "4Gi"
    })
    requests = optional(object({
      cpu    = optional(string, "0.1")
      memory = optional(string, "50Mi")
      }), {
      cpu    = "0.1"
      memory = "50Mi"
    })
  })
 n/a yes
service_account_annotations Used to populate the annotations for the fleet service account. map(any) {} no
service_annotations Used to populate the annotations for the fleet service. map(any) {} no
tolerations Used to configure tolerations. list(any) [] no
vuln_processing Used to configure the values for the vuln-processing cron job. 
object({
    ttl_seconds_after_finished = optional(number, 100)
    restart_policy             = optional(string, "Never")
    dedicated                  = optional(bool, false)
    schedule                   = optional(string, "0 * * * *")
    resources = object({
      limits = optional(object({
        cpu    = optional(string, "1")
        memory = optional(string, "4Gi")
        }), {
        cpu    = "1"
        memory = "4Gi"
      })
      requests = optional(object({
        cpu    = optional(string, "0.1")
        memory = optional(string, "50Mi")
        }), {
        cpu    = "0.1"
        memory = "50Mi"
      })
    })
  })
 n/a yes

Outputs

No outputs.

Reference in New Issue View Git Blame Copy Permalink