Files

Terraform Google Cloud Fleet Deployment

This Terraform project automates the deployment of Fleet Device Management (Fleet) on the Google Cloud Platform (GCP). It provisions a new GCP project (or uses an existing one if configured) and deploys all necessary infrastructure components, including:

  • A new Google Cloud Project (via terraform-google-modules/project-factory)
  • VPC Network and Subnets
  • Cloud NAT for outbound internet access
  • Cloud SQL for MySQL (database for Fleet)
  • Memorystore for Redis (caching for Fleet)
  • Google Cloud Storage (GCS) bucket (for Fleet software installers, S3-compatible access configured)
  • Google Secret Manager (for storing sensitive data like database passwords and Fleet server private key)
  • Cloud Run v2 Service (for the main Fleet application)
  • Cloud Run v2 Job (for database migrations)
  • External HTTP(S) Load Balancer with managed SSL certificate
  • Cloud DNS for managing the Fleet endpoint
  • Appropriate IAM service accounts and permissions

Prerequisites

  1. Terraform: Version ~> 1.11 (as specified in main.tf). Install from terraform.io.
  2. Google Cloud SDK (gcloud): Installed and configured. Install from cloud.google.com/sdk.
  3. GCP Account & Permissions:
    • A Google Cloud Organization ID.
    • A Billing Account ID.
    • Credentials configured for Terraform to interact with GCP. This typically means running gcloud auth application-default login with a user account that has:
      • Organization Administrator or Folder Creator & Project Creator roles at the Organization/Folder level (for the project-factory module).
      • Billing Account User role on the Billing Account.
      • Once the project is created, subsequent applies for resources within the project will generally require Owner or Editor on the provisioned project.
    • Alternatively, use a Service Account with appropriate permissions.
  4. Registered Domain Name: You will need a domain name that you can manage DNS records for. This project will create a Cloud DNS Managed Zone for this domain (or a subdomain).
  5. (Optional) Fleet License Key: If you have a premium Fleet license, you can provide it.

Configuration

  1. Clone the repository (if applicable) or ensure you have all the files.

  2. Create a terraform.tfvars file: Copy terraform.tfvars.example (you'll need to create this, see example below) to terraform.tfvars and populate it with your specific values.

    Example terraform.tfvars.example:

    # Required: GCP Organization and Billing
    org_id               = "YOUR_ORGANIZATION_ID" # e.g., "123456789012"
    billing_account_id   = "YOUR_BILLING_ACCOUNT_ID" # e.g., "012345-6789AB-CDEF01"
    
    # Required: DNS Configuration
    # The public DNS zone name (e.g., "example.com." - note the trailing dot)
    # This project will create/manage this zone in Cloud DNS.
    dns_zone_name        = "your-domain.com."
    # The fully qualified domain name for your Fleet instance (e.g., "fleet.your-domain.com.")
    dns_record_name      = "fleet.your-domain.com."
    
    # Optional: Project Configuration
    project_name         = "my-fleet-project" # Name for the new GCP project
    random_project_id    = true               # Set to false if you want to use project_name as project_id (must be globally unique)
    
    # Optional: Labels
    labels = {
      application = "fleet"
      environment = "production"
      owner       = "devops-team"
    }
    
    # --- byo-project module specific variables (passed through from root) ---
    # You can override defaults from byo-project/variables.tf here if needed,
    # or set values for variables that don't have defaults.
    
    # Example: Fleet Configuration including migration execution
    fleet_config = {
      # image_tag                = "fleetdm/fleet:v4.x.y" # Specify your desired Fleet version
      # license_key              = "YOUR_FLEET_LICENSE_KEY"
      # installers_bucket_name   = "my-unique-fleet-installers" # Must be globally unique
      # fleet_cpu                = "1000m"
      # fleet_memory             = "4096Mi"
      # debug_logging            = false
      # min_instance_count       = 1
      # max_instance_count       = 3
      # exec_migration           = true # Set to true to run migrations on `terraform apply`
                                       # (e.g., when changing image_tag)
      # use_h2c                  = false # Set to true to use h2c
    }
    
    

Key Variables to Set:

  • org_id: Your GCP Organization ID.
  • billing_account_id: Your GCP Billing Account ID.
  • dns_zone_name: The DNS zone that will be created/managed in Cloud DNS (e.g., mydomain.com.). Must end with a dot.
  • dns_record_name: The specific DNS record for Fleet (e.g., fleet.mydomain.com.). Must end with a dot.
  • project_name: A descriptive name for the project to be created.
  • fleet_config.image_tag: The Docker image tag for the Fleet version you want to deploy (e.g., fleetdm/fleet:v4.90.0).
  • fleet_config.exec_migration: Set to true when you are upgrading the fleet_config.image_tag to automatically run database migrations. Set to false if you want to manage migrations manually or if it's not an image upgrade.
  • fleet_config.license_key (Optional, inside the fleet_config object): Your Fleet license key if you have one.

Review variables.tf and byo-project/variables.tf for all available options and their defaults.

Known Limitations

  1. To handle large uploads, we need to bypass GCP cloud run's 32MB HTTP/1 body limit, by setting fleet_config.use_h2c = true which forces Fleet to use HTTP/2. However, as Cloud Run documentation notes, HTTP/2 end-to-end breaks connection upgrades, affecting WebSocket support and Fleet functionality for features such as Live Queries.

    Workaround: Use Fleet Gitops Flow

    • Set fleet_config.use_h2c = false to force HTTP/1
    • If you have large uploads, use fleetctl apply with a url field in your software YAML. The data flow avoids the ingress limit:
      • Apply the YAML (small request).
      • The Fleet Server makes an outbound GET request to download the file from the URL (S3, GCS, etc).
      • Cloud Run doesn't apply the 32MB limit to outbound requests.

Deployment Steps

  1. Authenticate with GCP: Ensure your gcloud CLI is authenticated with the account having the necessary permissions.

    gcloud auth application-default login
    
  2. Initialize Terraform: Navigate to the root directory of the project (where main.tf is located) and run:

    terraform init
    
  3. Plan the Deployment: Review the changes Terraform will make:

    terraform plan -out=tfplan
    
  4. Apply the Configuration: Provision the resources:

    terraform apply tfplan
    

    This process can take a significant amount of time (1020 minutes or more), especially for the initial project creation, API enablement, and Cloud SQL instance provisioning.

  5. Update DNS (if not using Cloud DNS for the parent zone): If dns_zone_name (e.g., fleet.your-domain.com.) is a new zone created by this Terraform, and your parent domain (e.g., your-domain.com.) is managed elsewhere (e.g., GoDaddy, Cloudflare), you need to delegate this new zone.

    • Get the Name Servers (NS) records for the newly created google_dns_managed_zone (${var.prefix}-zone):
      # First, get the project ID and zone name if you don't know them
      PROJECT_ID=$(terraform output -raw project_id 2>/dev/null || gcloud projects list --filter="name~^my-fleet-project" --format="value(projectId)" | head -n 1)
      DNS_ZONE_INTERNAL_NAME="fleet-zone" # Or as configured by 'prefix' variable
      
      gcloud dns managed-zones describe ${DNS_ZONE_INTERNAL_NAME} \
          --project=${PROJECT_ID} \
          --format="value(nameServers)"
      
    • Add these NS records to your parent domain's DNS settings at your domain registrar or DNS provider.

    If dns_zone_name (e.g., your-domain.com.) was an existing zone you own and manage via Cloud DNS in this project, Terraform would add records to it directly. The current setup creates the zone, so delegation is key if the parent is external.

Accessing Fleet

Once the deployment is complete, database migrations have run successfully, and DNS has propagated (which can take a few minutes):

  • Open your web browser and navigate to the dns_record_name you configured (e.g., https://fleet.your-domain.com).
  • You should be greeted by the Fleet setup page or login screen.

Key Resources Created (within the byo-project module)

  • VPC Network (fleet-network): Custom network for Fleet resources.
  • Subnet (fleet-subnet): Subnetwork within the VPC.
  • Cloud Router & NAT (fleet-cloud-router, fleet-vpc-nat): For outbound internet access from Cloud Run and other private resources.
  • Cloud SQL for MySQL (fleet-mysql-xxxx): Managed MySQL database.
  • Memorystore for Redis (fleet-cache): Managed Redis instance.
  • GCS Bucket (fleet-installers-xxxx): For S3-compatible storage.
  • Secret Manager Secrets:
    • fleet-db-password-xxxx: Stores the generated MySQL user password.
    • fleet-private-key-xxxx: Stores a generated private key for Fleet server.
  • Service Account (fleet-run-sa): Used by Cloud Run services/jobs with necessary permissions.
  • Cloud Run Service (fleet-service): Hosts the main Fleet application.
  • Cloud Run Job (fleet-migration-job): For database migrations.
  • Network Endpoint Group (fleet-neg): Connects the Load Balancer to the Cloud Run service.
  • Cloud DNS Managed Zone (fleet-zone): Manages DNS records for dns_zone_name.
  • External HTTP(S) Load Balancer (fleet-lb): Provides public access to Fleet via HTTPS.
  • DNS A Record: Points dns_record_name to the Load Balancer's IP.

(Note: xxxx indicates a random suffix added by some modules/resources for uniqueness.)

Cleaning Up

To destroy all resources created by this Terraform project:

  1. Navigate to the root directory of the project.
  2. Run the destroy command:
    terraform destroy
    
  3. Confirm by typing yes when prompted.

Important Considerations

  • Permissions: The user or service account running Terraform needs extensive permissions, especially for project creation. For ongoing management within the project, a role like Project Editor or more granular roles might suffice.
  • Fleet Configuration: This Terraform setup provisions the infrastructure. Further Fleet application configuration (e.g., SSO, SMTP, agent options) will need to be done through the Fleet UI or API after deployment.
  • Security:
    • Review IAM permissions granted.
    • The Load Balancer is configured for HTTPS and redirects HTTP.
    • Cloud SQL and Memorystore are not publicly accessible and are connected via Private Service Access.
  • Scalability: Cloud Run scaling parameters (min_instance_count, max_instance_count) and database/cache tier can be adjusted in variables.tf (via fleet_config, database_config, cache_config) to suit your load.

Networking Diagram

graph TD
    subgraph External
        Internet[(Internet)]
        Users[Web Console / fleetd agent]
        GitHub[(GitHub - Vulnerability Resources)]
    end

    subgraph "Google Cloud Platform (GCP)"
        subgraph VPC [VPC]
            direction LR
            subgraph PublicFacing [Public Zone]
                LB[Global Load Balancer]
            end
            subgraph PrivateZone [Private Zone]
                CloudRun[Cloud Run: Fleet App]
                CloudSQL[Cloud SQL for MySQL]
                Redis[Memorystore for Redis]
                NAT[Cloud NAT]
            end

            CloudRun --> Redis
            CloudRun --> CloudSQL
            CloudRun --> NAT
        end
    end

    Users -- "fleet.yourdomain.com" --> Internet
    Internet -- "fleet.yourdomain.com" --> LB
    LB --> CloudRun
    NAT --> GitHub

Requirements

Name Version
terraform ~> 1.11
google 6.35.0

Providers

No providers.

Modules

Name Source Version
fleet ./byo-project n/a
project_factory terraform-google-modules/project-factory/google ~> 18.0.0

Resources

No resources.

Inputs

Name Description Type Default Required
billing_account_id billing account id any n/a yes
cache_config Configuration for the Memorystore (Redis) instance.
object({
name = string
tier = string
engine_version = string
connect_mode = string
memory_size = number
})
{
"connect_mode": "PRIVATE_SERVICE_ACCESS",
"engine_version": null,
"memory_size": 1,
"name": "fleet-cache",
"tier": "STANDARD_HA"
}
no
database_config Configuration for the Cloud SQL (MySQL) instance.
object({
name = string
database_name = string
database_user = string
collation = string
charset = string
deletion_protection = bool
database_version = string
tier = string
})
{
"charset": "utf8mb4",
"collation": "utf8mb4_unicode_ci",
"database_name": "fleet",
"database_user": "fleet",
"database_version": "MYSQL_8_0",
"deletion_protection": false,
"name": "fleet-mysql",
"tier": "db-n1-standard-1"
}
no
dns_record_name The DNS record for Fleet (e.g., 'fleet.my-fleet-infra.com.') string n/a yes
dns_zone_name The DNS name of the managed zone (e.g., 'my-fleet-infra.com.') string n/a yes
fleet_config Configuration for the Fleet application deployment.
object({
installers_bucket_name = string
image_tag = string
fleet_cpu = string
fleet_memory = string
debug_logging = bool
license_key = optional(string)
min_instance_count = number
max_instance_count = number
exec_migration = bool
use_h2c = bool
extra_env_vars = optional(map(string))
extra_secret_env_vars = optional(map(object({
secret = string
version = string
})))
})
{
"debug_logging": false,
"exec_migration": true,
"extra_env_vars": {},
"extra_secret_env_vars": {},
"fleet_cpu": "1000m",
"fleet_memory": "4096Mi",
"image_tag": "fleetdm/fleet:v4.90.0",
"installers_bucket_name": "",
"max_instance_count": 5,
"min_instance_count": 1,
"use_h2c": false
}
no
fleet_image n/a string "v4.67.3" no
labels resource labels map(string)
{
"application": "fleet"
}
no
location The general location for resources, e.g., 'us' for GCS buckets. string "us" no
org_id organization id any n/a yes
prefix A prefix used for naming resources within the byo-project module. string "fleet" no
project_name n/a string "fleet" no
random_project_id n/a bool true no
region The GCP region for regional resources. string "us-central1" no
vpc_config Configuration for the VPC network and subnets.
object({
network_name = string
subnets = list(object({
subnet_name = string
subnet_ip = string
subnet_region = string
subnet_private_access = bool
}))
})
{
"network_name": "fleet-network",
"subnets": [
{
"subnet_ip": "10.10.10.0/24",
"subnet_name": "fleet-subnet",
"subnet_private_access": true,
"subnet_region": "us-central1"
}
]
}
no

Outputs

Name Description
cloud_router_name The name of the Cloud Router created for NAT.
cloud_run_service_location The location of the deployed Fleet Cloud Run service.
cloud_run_service_name The name of the deployed Fleet Cloud Run service.
dns_managed_zone_name The name of the Cloud DNS managed zone created for Fleet.
dns_managed_zone_name_servers The authoritative name servers for the created Cloud DNS managed zone. Delegate your domain to these.
fleet_application_url The primary URL to access the Fleet application (via the Load Balancer).
fleet_service_account_email The email address of the service account used by the Fleet Cloud Run service.
load_balancer_ip_address The external IP address of the HTTP(S) Load Balancer.
mysql_instance_connection_name The connection name for the Cloud SQL instance (used by Cloud SQL Proxy).
mysql_instance_name The name of the Cloud SQL for MySQL instance.
redis_host The host IP address of the Memorystore for Redis instance.
redis_instance_name The name of the Memorystore for Redis instance.
redis_port The port number of the Memorystore for Redis instance.
software_installers_bucket_name The name of the GCS bucket for Fleet software installers.
software_installers_bucket_url The gsutil URL of the GCS bucket for Fleet software installers.
vpc_network_name The name of the VPC network created.
vpc_network_self_link The self-link of the VPC network created.
vpc_subnets_names List of subnet names created in the VPC.