fleet-terraform/addons/logging-destination-snowflake

Logging Destination: Snowflake

This addon configures AWS Kinesis Firehose to send Fleet's osquery logs to Snowflake. It creates:

1. Kinesis Firehose delivery streams for each log type (results, status, and audit)
2. A single S3 bucket for storing all failed delivery attempts
3. IAM roles and policies for the Firehose streams to access the S3 bucket
4. An IAM policy for Fleet to access the Firehose streams

S3 Bucket Policy: Deny Non-HTTPS

This module automatically attaches a bucket policy to the failure S3 bucket that denies any requests made over plain HTTP. No configuration is required.

How to use

module "snowflake-logging" {
  source = "github.com/fleetdm/fleet-terraform//addons/logging-destination-snowflake?depth=1&ref=tf-mod-addon-logging-destination-snowflake-v1.1.0"

  s3_bucket_config = {
    name_prefix  = "fleet-snowflake-failure"
    expires_days = 5
  }
  snowflake_shared = {
    account_url    = "https://<snowflake_url>.snowflakecomputing.com"
    private_key    = "<pass this in securely>"
    key_passphrase = "<pass this in securely>"
    user           = "fleet_user"
    snowflake_role_configuration = {
      enabled        = true
      snowflake_role = "fleet_cloud_rl"
    }
  }

  log_destinations = {
    results = {
      name                   = "fleet-osquery-results-snowflake"
      database               = "fleet_cloud_db"
      schema                 = "fleet_cloud_schema"
      table                  = "osquery_results"
      buffering_size         = 2
      buffering_interval     = 60
      s3_buffering_size      = 10
      s3_buffering_interval  = 400
      s3_buffering_interval  = 400
      s3_error_output_prefix = "results/"
      data_loading_option    = "VARIANT_CONTENT_MAPPING"
      content_column_name    = "results"
    },
    status = {
      name                   = "fleet-osquery-status-snowflake"
      database               = "fleet_cloud_db"
      schema                 = "fleet_cloud_schema"
      table                  = "osquery_status"
      user                   = "fleet"
      buffering_size         = 2
      buffering_interval     = 60
      s3_buffering_size      = 10
      s3_buffering_interval  = 400
      s3_buffering_interval  = 400
      s3_error_output_prefix = "status/"
      data_loading_option    = "VARIANT_CONTENT_MAPPING"
      content_column_name    = "status"
    },
    audit = {
      name                   = "fleet-audit-snowflake"
      database               = "fleet_cloud_db"
      schema                 = "fleet_cloud_schema"
      table                  = "fleet_audit"
      buffering_size         = 2
      buffering_interval     = 60
      s3_buffering_size      = 10
      s3_buffering_interval  = 400
      s3_error_output_prefix = "audit/"
    }
  }
}

Then you can use the module's outputs in your Fleet configuration:

module "fleet" {
  source = "github.com/fleetdm/fleet-terraform?depth=1&ref=tf-mod-root-v1.30.0"
  certificate_arn = module.acm.acm_certificate_arn

  vpc = {
    name = local.vpc_name
    # azs = ["us-east-2a", "us-east-2b", "us-east-2c"]
  }

  fleet_config = {
    image = "fleetdm/fleet:v4.70.1"
    autoscaling = {
      min_capacity = 2
      max_capacity = 5
    }
    mem = 4096
    cpu = 512
    extra_environment_variables = merge(
      local.fleet_environment_variables,
      # Uncomment to enable Snowflake logging
      module.snowflake-logging.fleet_extra_environment_variables
    )
    extra_iam_policies = concat(
      # Uncomment to enable Snowflake logging
      module.snowflake-logging.fleet_extra_iam_policies,
    )
  }

  # ... other Fleet configuration ...
}

Requirements

No requirements.

Providers

Name	Version
aws	6.41.0

Modules

No modules.

Resources

Name	Type
aws_iam_policy.firehose	resource
aws_iam_policy.firehose-logging	resource
aws_iam_role.firehose	resource
aws_iam_role_policy_attachment.firehose	resource
aws_kinesis_firehose_delivery_stream.snowflake	resource
aws_s3_bucket.snowflake-failure	resource
aws_s3_bucket_lifecycle_configuration.snowflake-failure	resource
aws_s3_bucket_policy.deny_insecure_transport	resource
aws_s3_bucket_public_access_block.snowflake-failure	resource
aws_s3_bucket_server_side_encryption_configuration.snowflake-failure	resource
aws_iam_policy_document.deny_insecure_transport	data source
aws_iam_policy_document.firehose-logging	data source
aws_iam_policy_document.firehose_policy	data source
aws_iam_policy_document.osquery_firehose_assume_role	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
compression_format	Compression format for the Firehose delivery stream	string	"UNCOMPRESSED"	no
iam_policy_name	n/a	string	"snowflake-firehose-policy"	no
log_destinations	A map of configurations for Snowflake Firehose delivery streams.	map(object({ name = string database = string schema = string table = string buffering_size = number buffering_interval = number s3_buffering_size = number s3_buffering_interval = number s3_error_output_prefix = optional(string, null) data_loading_option = optional(string, "JSON_MAPPING") content_column_name = optional(string, null) metadata_column_name = optional(string, null) }))	{ "audit": { "buffering_interval": 60, "buffering_size": 2, "database": "fleet", "name": "fleet-audit-snowflake", "s3_buffering_interval": 400, "s3_buffering_size": 10, "schema": "fleet_schema", "table": "fleet_audit" }, "results": { "buffering_interval": 60, "buffering_size": 2, "database": "fleet", "name": "fleet-osquery-results-snowflake", "s3_buffering_interval": 400, "s3_buffering_size": 10, "schema": "fleet_schema", "table": "osquery_results" }, "status": { "buffering_interval": 60, "buffering_size": 2, "database": "fleet", "name": "fleet-osquery-status-snowflake", "s3_buffering_interval": 400, "s3_buffering_size": 10, "schema": "fleet_schema", "table": "osquery_status", "user": "fleet" } }	no
s3_bucket_config	Configuration for the S3 bucket used to store failed Snowflake delivery attempts	object({ name_prefix = optional(string, "fleet-snowflake-failure") expires_days = optional(number, 1) })	{ "expires_days": 1, "name_prefix": "fleet-snowflake-failure" }	no
snowflake_shared	Shared configurations among each logging destination	object({ account_url = string private_key = string key_passphrase = optional(string, null) user = string snowflake_role_configuration = object({ enabled = bool snowflake_role = optional(string, null) }) snowflake_vpc_configuration = optional(object({ private_link_vpce_id = string }), { private_link_vpce_id = null }) })	n/a	yes

Outputs

Name	Description
fleet_extra_environment_variables	Environment variables to configure Fleet to use Snowflake logging via Firehose
fleet_extra_iam_policies	IAM policies required for Fleet to log to Snowflake via Firehose
fleet_s3_snowflake_failure_config	S3 bucket details - snowflake-failure