Discharge7443/fleet-terraform
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
main
fleet-terraform/addons/monitoring
T
History

README.md

Monitoring addon

This addon enables Cloudwatch monitoring for Fleet.

This includes:

  • 5XX Errors on ALB
  • ECS Service Monitoring
  • RDS Monitoring
  • Redis Monitoring
  • ACM Certificate Monitoring
  • A custom Lambda to check the Fleet DB for Cron runs

Preparation

Note: The documented examples and links in this README may assume use of module.fleet instead of module.main. The monitoring example configuration can be modified as documented below, if your fleet module is named fleet instead of main

  • A search and replace of module.fleet -> module.main

Some of the for_each and counts in this module cannot pre-determine the numbers until the main fleet module is applied.

You will need to terraform apply -target module.fleet prior to applying monitoring assuming the use of a configuration matching the example at https://github.com/fleetdm/fleet-terraform/blob/main/example/main.tf.

Multiple alb support was added in order to allow monitoring saml-auth-proxy. See https://github.com/fleetdm/fleet-terraform/tree/main/addons/saml-auth-proxy

Example configuration

This assumes your fleet module is main and is configured with it's default documentation.

https://github.com/fleetdm/fleet-terraform/blob/main/example/main.tf for details.

Note: If you haven't specified defined local.customer or customized service names, the default is "fleet" for anywhere that local.customer is specified below.

If the Fleet database password secret is encrypted with a CMK, also pass mysql_password_secret_kms_key_arn so the cron-monitoring Lambda can decrypt it. When using the byo-vpc module, wire this from module.<fleet_module>.byo-vpc.rds_password_secret_kms_key_arn.

To encrypt the cron-monitoring Lambda and its CloudWatch log group with a CMK, use cron_monitoring.lambda_kms. Supply kms_key_arn to use an existing key, or set cmk_enabled = true and leave kms_key_arn = null to have this module create a key using kms_alias. If you need to customize the base key policy for that module-created CMK, set cron_monitoring.lambda_kms.kms_base_policy.

Important: If the Fleet database password secret key is using a custom kms_base_policy that does not grant kms:* to the account root (for example, a least-privilege policy that only allows specific principals), you must also add the cron-monitoring Lambda role to rds_config.password_secret_kms.extra_kms_policies in the byo-vpc module. AWS KMS requires both the key policy and the IAM policy to allow access. The Lambda role name is predictable (<customer_prefix>-cron-monitoring-lambda), so you can construct the ARN before the role exists. See the byo-vpc module README for a full example.

Upgrading existing cron-monitoring log groups

If you are upgrading from a version where the cron-monitoring Lambda log group was not managed with the Lambda's real function name, AWS may already have auto-created /aws/lambda/<customer_prefix>_cron_monitoring outside Terraform. When this module starts managing that real log group so KMS encryption can be applied, Terraform cannot create it if it already exists.

There are two ways to handle that:

  1. Delete the existing auto-created log group and let Terraform recreate it with the new KMS settings. This avoids carrying forward older log events that were not encrypted with the new CMK.
aws logs delete-log-group --log-group-name "/aws/lambda/<customer_prefix>_cron_monitoring"
terraform apply
  1. Import the existing auto-created log group into Terraform state. Use this only if you need to preserve the existing log events as-is.
terraform state rm 'module.monitoring.aws_cloudwatch_log_group.cron_monitoring_lambda[0]'
terraform import 'module.monitoring.aws_cloudwatch_log_group.cron_monitoring_lambda[0]' '/aws/lambda/<customer_prefix>_cron_monitoring'
terraform apply

If the older unused hyphenated log group still exists, you can remove it after the upgrade:

aws logs delete-log-group --log-group-name "/aws/lambda/<customer_prefix>-cron-monitoring"

module "monitoring" {
  source                 = "github.com/fleetdm/fleet-terraform//addons/monitoring?ref=tf-mod-addon-monitoring-v1.14.0"
  customer_prefix        = local.customer
  fleet_ecs_service_name = module.fleet.byo-vpc.byo-db.byo-ecs.service.name
  albs = [
    {
      name                    = module.fleet.byo-vpc.byo-db.alb.lb_dns_name,
      target_group_name       = module.fleet.byo-vpc.byo-db.alb.target_group_names[0]
      target_group_arn_suffix = module.fleet.byo-vpc.byo-db.alb.target_group_arn_suffixes[0]
      arn_suffix              = module.fleet.byo-vpc.byo-db.alb.lb_arn_suffix
      ecs_service_name        = module.fleet.byo-vpc.byo-db.byo-ecs.service.name
      min_containers          = module.fleet.byo-vpc.byo-db.byo-ecs.appautoscaling_target.min_capacity
      alert_thresholds = {
        HTTPCode_ELB_5XX_Count = {
          period    = 3600
          threshold = 2
        },
        HTTPCode_Target_5XX_Count = {
          period    = 120
          threshold = 0
        }
      }
    },
  ]
  sns_topic_arns_map = {
    log_monitoring   = [var.sns_topic_arn]
    alb_httpcode_5xx = [var.sns_topic_arn]
    cron_monitoring  = [var.sns_topic_arn]
    cron_job_failure_monitoring  = [var.sns_another_topic_arn]
  }
  mysql_cluster_members = module.fleet.byo-vpc.rds.cluster_members
  # The cloudposse module seems to have a nested list here.
  redis_cluster_members = module.fleet.byo-vpc.redis.member_clusters[0]
  acm_certificate_arn   = module.acm.acm_certificate_arn
  cron_monitoring = {
    mysql_host                 = module.fleet.byo-vpc.rds.cluster_reader_endpoint
    mysql_database             = module.fleet.byo-vpc.rds.cluster_database_name
    mysql_user                 = module.fleet.byo-vpc.rds.cluster_master_username
    mysql_password_secret_name = "${local.customer}-database-password"
    mysql_password_secret_kms_key_arn = module.fleet.byo-vpc.rds_password_secret_kms_key_arn
    mysql_tls_config           = "true"
    rds_security_group_id      = module.fleet.byo-vpc.rds.security_group_id
    subnet_ids                 = module.fleet.vpc.private_subnets
    vpc_id                     = module.fleet.vpc.vpc_id
    # Format of https://pkg.go.dev/time#ParseDuration
    delay_tolerance = "4h"
    # Interval format for: https://docs.aws.amazon.com/scheduler/latest/UserGuide/schedule-types.html#rate-based
    run_interval          = "1 hour"
    log_retention_in_days = 365
    # Cron List of Names to Ignore (see below for valid values)
    ignore_list = []
    lambda_kms = {
      cmk_enabled     = true
      kms_alias       = "fleet-cron-monitoring"
      kms_base_policy = local.kms_base_policy_statements
    }
  }
  log_monitoring = {
    invalid-secret = {
      log_group_name = module.fleet.byo-vpc.byo-db.byo-ecs.logging_config.awslogs-group
      pattern = "{ $.internal = \"invalid secret\" }"
      evaluation_periods = 1
      period             = 3600
      threshold          = 1
    }
    duplicate-identifier = {
      log_group_name = module.fleet.byo-vpc.byo-db.byo-ecs.logging_config.awslogs-group
      pattern = "{ $.msg = \"osquery host with duplicate identifier has enrolled in Fleet and will overwrite existing host data\" }"
      evaluation_periods = 1
      period             = 3600
      threshold          = 1
    }
    limit-exceeded = {
      log_group_name = module.fleet.byo-vpc.byo-db.byo-ecs.logging_config.awslogs-group
      pattern = "{ $.err = \"limit exceeded\" }"
      evaluation_periods = 1
      period             = 60
      threshold          = 1
    }
  }
}

Configurable Alert Thresholds

All CloudWatch alarm thresholds, periods, and evaluation periods can be overridden via the alert_thresholds object. Each alarm type is optional — omit any field to use the module default.

Field Alarm Default threshold Default period Default evaluation periods
rds_cpu RDS CPU Utilization 80 300s 1
redis_cpu Redis CPU Utilization 70 300s 1
redis_cpu_engine Redis Engine CPU Utilization 25 300s 1
redis_memory Redis Database Memory % 80 300s 1
acm_cert_expiry ACM Certificate Expiry 30 days 86400s 1
alb_healthyhosts ALB Healthy Host Count 1 60s 1

Example — raise the RDS CPU threshold and require 3 consecutive 5-minute periods before firing (15 minutes total):

module "monitoring" {
  # ...
  alert_thresholds = {
    rds_cpu = {
      threshold          = 90
      period             = 300
      evaluation_periods = 3
    }
  }
}

Anomaly detection alarms (redis_current_connections, redis_replication_lag, target_response_time) are not included — they use CloudWatch anomaly detection bands and do not have a simple threshold/period/evaluation_periods structure.

SNS topic ARNs map

Valid targets for sns_topic_arns_map:

  • acm_certificate_expired
  • alb_healthyhosts
  • alb_httpcode_5xx
  • backend_response_time
  • cron_monitoring (notifications about failures in the cron scheduler)
  • cron_job_failure_monitoring (notifications about errors in individual cron jobs - defaults to value of cron_monitoring)
  • log_monitoring
  • rds_cpu_utilization_too_high
  • rds_db_event_subscription
  • redis_cpu_engine_utilization
  • redis_cpu_utilization
  • redis_current_connections
  • redis_database_memory_percentage
  • redis_replication_lag

If you want to publish to all, use default_sns_topic_arns instead and include your notification ARNs there.

Deprecated (typo) aliases are still accepted for backwards compatibility:

  • alb_helthyhosts (use alb_healthyhosts)
  • rds_cpu_untilizaton_too_high (use rds_cpu_utilization_too_high)

Cron Names

  • apple_mdm_apns_pusher
  • apple_mdm_dep_profile_assigner
  • apple_mdm_iphone_ipad_refetcher
  • apple_mdm_iphone_ipad_reviver
  • automations
  • batch_activity_completion_checker
  • calendar
  • cleanups_then_aggregation
  • host_vitals_label_membership
  • integrations
  • maintained_apps
  • mdm_service_discovery
  • mdm_windows_profile_manager
  • refresh_vpp_app_versions
  • scheduled_batch_activities
  • upcoming_activities_maintenance
  • usage_statistics
  • vulnerabilities

Requirements

Name Version
terraform >= 1.12.0
aws >= 6.37.0

Providers

Name Version
archive 2.7.1
aws 6.39.0
null 3.2.4

Modules

No modules.

Resources

Name Type
aws_cloudwatch_event_rule.cron_monitoring_lambda resource
aws_cloudwatch_event_target.cron_monitoring_lambda resource
aws_cloudwatch_log_group.cron_monitoring_lambda resource
aws_cloudwatch_log_metric_filter.log_monitoring resource
aws_cloudwatch_metric_alarm.acm_certificate_expired resource
aws_cloudwatch_metric_alarm.alb_healthyhosts resource
aws_cloudwatch_metric_alarm.cpu_utilization_too_high resource
aws_cloudwatch_metric_alarm.lb resource
aws_cloudwatch_metric_alarm.log_monitoring resource
aws_cloudwatch_metric_alarm.redis-current-connections resource
aws_cloudwatch_metric_alarm.redis-database-memory-percentage resource
aws_cloudwatch_metric_alarm.redis-replication-lag resource
aws_cloudwatch_metric_alarm.redis_cpu resource
aws_cloudwatch_metric_alarm.redis_cpu_engine_utilization resource
aws_cloudwatch_metric_alarm.target_response_time resource
aws_db_event_subscription.default resource
aws_iam_policy.cron_monitoring_lambda resource
aws_iam_role.cron_monitoring_lambda resource
aws_iam_role_policy_attachment.cron_monitoring_lambda resource
aws_iam_role_policy_attachment.cron_monitoring_lambda_managed resource
aws_kms_alias.cron_monitoring_lambda resource
aws_kms_key.cron_monitoring_lambda resource
aws_lambda_function.cron_monitoring resource
aws_lambda_permission.cron_monitoring_cloudwatch resource
aws_security_group.cron_monitoring resource
aws_security_group_rule.cron_monitoring_to_rds resource
null_resource.cron_monitoring_build resource
archive_file.cron_monitoring_lambda data source
aws_caller_identity.current data source
aws_iam_policy_document.cron_monitoring_lambda data source
aws_iam_policy_document.cron_monitoring_lambda_assume_role data source
aws_iam_policy_document.cron_monitoring_lambda_kms data source
aws_partition.current data source
aws_region.current data source
aws_secretsmanager_secret.mysql_database_password data source

Inputs

Name Description Type Default Required
acm_certificate_arn n/a string null no
albs n/a 
list(object({
    name                    = string
    arn_suffix              = string
    target_group_name       = string
    target_group_arn_suffix = string
    min_containers          = optional(string, 1)
    ecs_service_name        = string
    alert_thresholds = optional(
      object({
        HTTPCode_ELB_5XX_Count = object({
          period    = number
          threshold = number
        })
        HTTPCode_Target_5XX_Count = object({
          period    = number
          threshold = number
        })
      }),
      {
        HTTPCode_ELB_5XX_Count = {
          period    = 120
          threshold = 0
        },
        HTTPCode_Target_5XX_Count = {
          period    = 120
          threshold = 0
        }
      }
    )
  }))
 [] no
alert_thresholds CloudWatch alarm threshold overrides. Each alarm type is optional; omitted fields use the defaults below. 
object({
    rds_cpu = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 80
      period             = 300
      evaluation_periods = 1
    })
    redis_cpu = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 70
      period             = 300
      evaluation_periods = 1
    })
    redis_cpu_engine = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 25
      period             = 300
      evaluation_periods = 1
    })
    redis_memory = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 80
      period             = 300
      evaluation_periods = 1
    })
    acm_cert_expiry = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 30
      period             = 86400
      evaluation_periods = 1
    })
    alb_healthyhosts = optional(object({
      threshold          = number
      period             = number
      evaluation_periods = number
      }), {
      threshold          = 1
      period             = 60
      evaluation_periods = 1
    })
  })
 
{
  "acm_cert_expiry": {
    "evaluation_periods": 1,
    "period": 86400,
    "threshold": 30
  },
  "alb_healthyhosts": {
    "evaluation_periods": 1,
    "period": 60,
    "threshold": 1
  },
  "rds_cpu": {
    "evaluation_periods": 1,
    "period": 300,
    "threshold": 80
  },
  "redis_cpu": {
    "evaluation_periods": 1,
    "period": 300,
    "threshold": 70
  },
  "redis_cpu_engine": {
    "evaluation_periods": 1,
    "period": 300,
    "threshold": 25
  },
  "redis_memory": {
    "evaluation_periods": 1,
    "period": 300,
    "threshold": 80
  }
}
 no
cron_monitoring n/a 
object({
    mysql_host                        = string
    mysql_database                    = string
    mysql_user                        = string
    mysql_password_secret_name        = string
    mysql_password_secret_kms_key_arn = optional(string, null)
    mysql_tls_config                  = optional(string, "true")
    vpc_id                            = string
    subnet_ids                        = list(string)
    rds_security_group_id             = string
    delay_tolerance                   = string
    run_interval                      = string
    log_retention_in_days             = optional(number, 7)
    ignore_list                       = optional(list(string), [])
    lambda_kms = optional(object({
      cmk_enabled = optional(bool, false)
      kms_key_arn = optional(string, null)
      kms_alias   = optional(string, "fleet-cron-monitoring")
      kms_base_policy = optional(list(object({
        sid    = string
        effect = string
        principals = object({
          type        = string
          identifiers = list(string)
        })
        actions   = list(string)
        resources = list(string)
        conditions = optional(list(object({
          test     = string
          variable = string
          values   = list(string)
        })), [])
      })), null)
      extra_kms_policies = optional(list(any), [])
      }), {
      cmk_enabled        = false
      kms_key_arn        = null
      kms_alias          = "fleet-cron-monitoring"
      kms_base_policy    = null
      extra_kms_policies = []
    })
  })
 null no
customer_prefix n/a string "fleet" no
default_sns_topic_arns n/a list(string) [] no
fleet_ecs_service_name n/a string null no
log_monitoring Map of CloudWatch log monitors to create. Key is used as a suffix for resources and metric naming. 
map(object({
    log_group_name     = string
    pattern            = string
    evaluation_periods = number
    period             = number
    threshold          = number
  }))
 {} no
mysql_cluster_members n/a list(string) [] no
redis_cluster_members n/a list(string) [] no
sns_topic_arns_map n/a map(list(string)) {} no

Outputs

Name Description
cron_monitoring_lambda_arn n/a
cron_monitoring_lambda_role_arn n/a
Reference in New Issue View Git Blame Copy Permalink