doc: added scripts and scripts for extreme low costs

This commit is contained in:
Leon Xu
2026-08-07 17:50:32 -07:00
parent e123c1895a
commit 60bd5066ef
5 changed files with 575 additions and 0 deletions
+117
View File
@@ -0,0 +1,117 @@
# Deployment Guide: Fleet with no ALB, no NAT, no Redis, no logs
`extreme-no-logs.tfvars` deploys Fleet at **~$48-52/month** by removing every
component that isn't strictly required for a ~10-device deployment:
| Component | Config | Monthly cost |
| ------------------ | ----------------------------------- | ------------ |
| Aurora Serverless | 0.5 ACU min / 1.0 ACU max | ~$36 |
| ECS Fargate Spot | 0.25 vCPU / 1GB, 1 task, 100% Spot | ~$9-13 |
| Secrets Manager | 2 secrets | ~$0.80 |
| S3 installers | 1 bucket, no versioning | ~$0.50 |
| Data transfer | light | ~$1 |
| ALB | **removed** | $0 |
| NAT Gateway | **removed** | $0 |
| ElastiCache | **removed** | $0 |
| CloudWatch Logs | **removed** | $0 |
| **Total** | | **~$48-52** |
## Architecture
```
internet
│ :8080 (0.0.0.0/0)
┌───────────────────────┐
│ ECS Fargate Spot task │ public subnet, public IP
│ fleetdm/fleet │
└───────────┬───────────┘
│ :3306 (VPC-internal)
┌───────────────────────┐
│ Aurora Serverless v2 │ database subnet (private)
│ 0.5 - 1.0 ACU │
└───────────────────────┘
```
There is no TLS termination point. Fleet serves **plain HTTP on :8080**.
> ⚠️ **Security note**: traffic between you/agents and Fleet is unencrypted.
> For a personal/lab deployment this is usually acceptable; the Fleet server
> private key and DB credentials stay inside AWS. If you need TLS, see
> "Adding TLS" below.
## Prerequisites
- Terraform ≥ 1.12, AWS CLI, credentials with admin-ish rights
- No ACM certificate needed (there is no HTTPS listener)
## Deploy
```bash
terraform init
terraform apply -var-file=extreme-no-logs.tfvars
```
Wait ~5-10 minutes for Aurora + the first ECS task.
## Access Fleet
### Option A — direct IP
```bash
# discover the task public IP
TASK=$(aws ecs list-tasks --cluster fleet --service-name fleet --query 'taskArns[0]' --output text)
ENI=$(aws ecs describe-tasks --cluster fleet --tasks "$TASK" \
--query "tasks[0].attachments[?status=='ATTACHED']|[0].details[?name=='networkInterfaceId'].value | [0]" --output text)
IP=$(aws ec2 describe-network-interfaces --network-interface-ids "$ENI" \
--query 'NetworkInterfaces[0].Association.PublicIp' --output text)
echo "http://$IP:8080"
```
Then set Fleet's server URL (Settings → Organization) to `http://<IP>:8080`
so newly enrolled agents get the right address.
### Option B — Route53 (recommended)
```bash
./scripts/setup-route53.sh fleet.example.com # one-shot
# or keep in sync automatically:
*/5 * * * * /path/to/scripts/setup-route53.sh fleet.example.com >>/tmp/fleet-dns.log 2>&1
```
The record uses TTL=60s; the script re-resolves the task IP on each run, so
DNS follows Spot restarts within ~5 minutes.
## Debugging (no logs by default)
```bash
./scripts/emergency-logging.sh enable # adds awslogs driver, /ecs/fleet, 3-day retention
aws logs tail /ecs/fleet --follow
./scripts/emergency-logging.sh disable # removes driver + deletes log group
```
Each enable/disable registers a new task-definition revision and forces a
redeployment (~1 min downtime).
## Adding TLS (optional)
The cheapest TLS option without an ALB is Fleet's built-in TLS
(`FLEET_SERVER_TLS=true` plus cert files). That requires baking certs into the
image or mounting them, which is out of scope for this configuration. A
pragmatic middle ground:
1. Put a **CloudFront distribution** in front of the task IP (~$1-3/month at
low traffic) — CloudFront terminates TLS with its own cert and forwards to
the origin over HTTP inside the AWS backbone. Note CloudFront adds latency
for osquery's long-polling; test before committing.
2. Or re-enable the ALB (`alb_config.enabled = true` + `certificate_arn`)
which restores managed TLS for ~$16/month.
## Reverting to a load balancer
Set `alb_config.enabled = true` in your tfvars (and provide
`certificate_arn`) and re-apply. Terraform moves existing state automatically
via the `moved` blocks in `byo-vpc/byo-db/moved.tf` — the ALB is recreated,
not orphaned.
+114
View File
@@ -0,0 +1,114 @@
# Migration Guide: ALB → No-ALB (extreme-no-logs)
This guide covers migrating an existing ALB-based Fleet deployment to the
`extreme-no-logs.tfvars` configuration (~$48-52/month).
> **Fresh install?** Skip this document and read
> [DEPLOYMENT-NO-ALB.md](./DEPLOYMENT-NO-ALB.md) instead.
## What changes
| Component | Before (extreme-low-cost) | After (extreme-no-logs) |
| ------------------ | ------------------------- | ---------------------------- |
| ALB | ✔ created (~$16/mo) | ❌ destroyed |
| ElastiCache | ✔ t4g.micro (~$12/mo) | ❌ destroyed |
| CloudWatch Logs | 1-day retention | ❌ no log groups / no driver |
| ECS networking | private subnet (ALB → task) | public subnet + public IP |
| Fleet access | `https://fleet.example.com` (ALB + ACM cert) | `http://<task-ip>:8080` or Route53 A record |
| TLS | terminated at ALB | none (plain HTTP) by default |
## 1. Pre-migration checklist
1. **Backup state**
```bash
terraform state pull > terraform.state.backup.$(date +%Y%m%d-%H%M%S).json
```
2. **Note the current ALB DNS name** and any Route53 records pointing at it:
```bash
terraform output -json | python3 -c "import json,sys; d=json.load(sys.stdin); print(d)"
```
3. **Plan for downtime**: the ECS service is redeployed; expect ~2-5 minutes
of unavailability.
4. **Notify device owners** (optional): osquery agents buffer results and
reconnect automatically; nothing is lost during a short outage.
## 2. Automated migration (recommended)
```bash
./scripts/migrate-from-alb.sh
```
The script backs up state, shows the plan, asks for confirmation, applies,
discovers the new task public IP, and optionally syncs a Route53 record.
## 3. Manual migration
```bash
terraform plan -var-file=extreme-no-logs.tfvars -out=migration.plan
# review: ALB, NAT GW, ElastiCache and log groups should show as destroyed
terraform apply migration.plan
```
## 4. Post-migration
1. **Get the new endpoint**:
```bash
# direct IP
./scripts/setup-route53.sh --help # shows discovery logic, or:
aws ecs list-tasks --cluster fleet --service-name fleet
```
2. **(Optional) Route53**: keep a low-TTL A record in sync:
```bash
./scripts/setup-route53.sh fleet.example.com
# and via cron:
*/5 * * * * /path/to/scripts/setup-route53.sh fleet.example.com >>/tmp/fleet-dns.log 2>&1
```
3. **Update Fleet server settings**: In Fleet UI → Settings → Organization,
set the server URL to the new address so agents enroll with the right URL.
4. **Verify agents reconnect** within ~10 minutes.
## 5. Rollback
If anything goes wrong, restore the pre-migration state and re-apply the old
configuration:
```bash
# Restore state backup (local backend)
cp terraform.state.backup.<timestamp>.json terraform.tfstate
# Re-apply previous config
terraform apply -var-file=extreme-low-cost.tfvars
```
With a remote backend (S3), use the backend's versioning to restore the
previous state version, then apply the previous tfvars.
## 6. Troubleshooting without logs
Fleet runs with no log driver in this configuration. To debug:
```bash
# Enable logs temporarily (default 3-day retention)
./scripts/emergency-logging.sh enable
aws logs tail /ecs/fleet --follow
# When done, resume cost savings:
./scripts/emergency-logging.sh disable
```
Health check without logs:
```bash
curl -s http://<task-ip>:8080/healthz
```
+115
View File
@@ -0,0 +1,115 @@
#!/bin/bash
# ============================================================================
# emergency-logging.sh - Temporarily enable/disable CloudWatch Logs for Fleet
# ============================================================================
# The extreme-no-logs configuration ships with NO container logging to save
# ~$3-5/month. When you need to debug Fleet, run this script to temporarily
# attach the awslogs driver to the running service, then disable it again
# when finished (to resume full cost savings).
#
# Usage:
# ./scripts/emergency-logging.sh enable [retention-days] # default 3 days
# ./scripts/emergency-logging.sh disable
# ./scripts/emergency-logging.sh status
#
# Environment overrides:
# FLEET_CLUSTER (default: fleet)
# FLEET_SERVICE (default: fleet)
# FLEET_LOG_GROUP (default: /ecs/fleet)
# AWS_REGION (default: aws cli default)
# ============================================================================
set -euo pipefail
CLUSTER="${FLEET_CLUSTER:-fleet}"
SERVICE="${FLEET_SERVICE:-fleet}"
LOG_GROUP="${FLEET_LOG_GROUP:-/ecs/fleet}"
REGION="${AWS_REGION:-$(aws configure get region 2>/dev/null || echo us-east-2)}"
RETENTION="${2:-3}"
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
require() { command -v "$1" >/dev/null 2>&1 || { echo -e "${RED}❌ '$1' not found${NC}"; exit 1; }; }
require aws
require python3
current_task_def() {
aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" --region "$REGION" \
--query 'services[0].taskDefinition' --output text
}
register_task_def() {
# $1 = "with-logs" | "without-logs" ; prints new task definition ARN
local mode="$1" td new_td_json
td=$(current_task_def)
if [ -z "$td" ] || [ "$td" = "None" ]; then
echo -e "${RED}❌ Could not find service ${SERVICE} in cluster ${CLUSTER}${NC}" >&2
exit 1
fi
new_td_json=$(aws ecs describe-task-definition --task-definition "$td" --region "$REGION" \
| MODE="$mode" LOG_GROUP="$LOG_GROUP" REGION="$REGION" python3 -c '
import json, os, sys
td = json.load(sys.stdin)["taskDefinition"]
mode, log_group, region = os.environ["MODE"], os.environ["LOG_GROUP"], os.environ["REGION"]
for c in td["containerDefinitions"]:
if mode == "with-logs":
c["logConfiguration"] = {
"logDriver": "awslogs",
"options": {
"awslogs-group": log_group,
"awslogs-region": region,
"awslogs-stream-prefix": "fleet",
},
}
else:
c.pop("logConfiguration", None)
# register-task-definition only accepts a subset of describe output fields
out = {k: td[k] for k in (
"family", "taskRoleArn", "executionRoleArn", "networkMode",
"containerDefinitions", "requiresCompatibilities", "cpu", "memory",
) if k in td}
for opt in ("volumes", "placementConstraints", "pidMode", "ipcMode", "ephemeralStorage", "runtimePlatform"):
if opt in td and td[opt]:
out[opt] = td[opt]
print(json.dumps(out))
')
aws ecs register-task-definition --cli-input-json "$new_td_json" --region "$REGION" \
--query 'taskDefinition.taskDefinitionArn' --output text
}
case "${1:-}" in
enable)
echo -e "${YELLOW}📝 Enabling CloudWatch Logs for ${SERVICE} (${LOG_GROUP}, ${RETENTION}-day retention)...${NC}"
aws logs create-log-group --log-group-name "$LOG_GROUP" --region "$REGION" 2>/dev/null || true
aws logs put-retention-policy --log-group-name "$LOG_GROUP" --retention-in-days "$RETENTION" --region "$REGION"
NEW_TD=$(register_task_def with-logs)
echo " New task definition: $NEW_TD"
aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
--task-definition "$NEW_TD" --force-new-deployment --region "$REGION" >/dev/null
echo -e "${GREEN}✅ Logs enabled. Tail with:${NC}"
echo " aws logs tail $LOG_GROUP --follow --region $REGION"
echo -e "${YELLOW}⚠️ Remember to run '$0 disable' when done to resume cost savings.${NC}"
;;
disable)
echo -e "${YELLOW}🧹 Disabling CloudWatch Logs for ${SERVICE}...${NC}"
NEW_TD=$(register_task_def without-logs)
echo " New task definition: $NEW_TD"
aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
--task-definition "$NEW_TD" --force-new-deployment --region "$REGION" >/dev/null
aws logs delete-log-group --log-group-name "$LOG_GROUP" --region "$REGION" 2>/dev/null || true
echo -e "${GREEN}✅ Logs disabled and log group deleted. Cost savings resumed.${NC}"
;;
status)
TD=$(current_task_def)
aws ecs describe-task-definition --task-definition "$TD" --region "$REGION" \
--query 'taskDefinition.containerDefinitions[0].logConfiguration' --output json
;;
*)
echo "Usage: $0 <enable [retention-days]|disable|status>"
exit 1
;;
esac
+128
View File
@@ -0,0 +1,128 @@
#!/bin/bash
# ============================================================================
# migrate-from-alb.sh - Migrate an existing ALB-based Fleet deployment to the
# ALB-free extreme-no-logs configuration (~$48-52/month).
# ============================================================================
# What it does:
# 1. Backs up the Terraform state file
# 2. Runs terraform plan with extreme-no-logs.tfvars
# 3. On confirmation, applies (this DESTROYS the ALB, NAT GW, ElastiCache,
# and CloudWatch log groups, and recreates the ECS service with a
# public IP and no log driver)
# 4. Prints the new direct access URL
# 5. Optionally syncs a Route53 record via setup-route53.sh
#
# Usage:
# ./scripts/migrate-from-alb.sh [--tfvars path] [--yes]
#
# Prerequisites: terraform >= 1.12, aws cli, valid AWS credentials.
# ============================================================================
set -euo pipefail
TFVARS="extreme-no-logs.tfvars"
ASSUME_YES=false
while [ $# -gt 0 ]; do
case "$1" in
--tfvars) TFVARS="$2"; shift 2 ;;
--yes) ASSUME_YES=true; shift ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m'
command -v terraform >/dev/null 2>&1 || { echo -e "${RED}❌ terraform not found${NC}"; exit 1; }
command -v aws >/dev/null 2>&1 || { echo -e "${RED}❌ aws cli not found${NC}"; exit 1; }
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
cd "$REPO_ROOT"
echo -e "${BLUE}🚀 Fleet migration: ALB -> direct public IP (${TFVARS})${NC}"
echo ""
# --- Step 1: State backup ---------------------------------------------------
echo -e "${YELLOW}Step 1/5: Backing up Terraform state...${NC}"
BACKUP="terraform.state.backup.$(date +%Y%m%d-%H%M%S).json"
if terraform state pull > "$BACKUP" 2>/dev/null && [ -s "$BACKUP" ]; then
echo -e "${GREEN}✅ State backed up to ${BACKUP}${NC}"
else
echo -e "${YELLOW}⚠️ No existing state found (fresh deployment?) - continuing.${NC}"
rm -f "$BACKUP"
fi
echo ""
# --- Step 2: Plan ------------------------------------------------------------
echo -e "${YELLOW}Step 2/5: Planning changes...${NC}"
terraform plan -var-file="$TFVARS" -input=false -out=migration.plan
echo ""
# --- Step 3: Confirm ---------------------------------------------------------
echo -e "${RED}⚠️ This will DESTROY the ALB, NAT gateway(s), ElastiCache cluster and"
echo -e " CloudWatch log groups. Fleet will briefly be unreachable while the"
echo -e " ECS service is redeployed with a public IP.${NC}"
if [ "$ASSUME_YES" = false ]; then
read -r -p "Continue? Type 'yes' to proceed: " CONFIRM
if [ "$CONFIRM" != "yes" ]; then
rm -f migration.plan
echo "Migration cancelled."
exit 1
fi
fi
# --- Step 4: Apply -----------------------------------------------------------
echo -e "${YELLOW}Step 3/5: Applying...${NC}"
terraform apply -input=false migration.plan
rm -f migration.plan
echo -e "${GREEN}✅ Applied.${NC}"
echo ""
# --- Step 5: Access info -----------------------------------------------------
echo -e "${YELLOW}Step 4/5: Discovering new Fleet endpoint...${NC}"
REGION="${AWS_REGION:-$(aws configure get region 2>/dev/null || echo us-east-2)}"
CLUSTER="fleet"; SERVICE="fleet"
for i in $(seq 1 24); do
TASK_ARN=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" --region "$REGION" \
--query 'taskArns[0]' --output text 2>/dev/null || true)
if [ -n "$TASK_ARN" ] && [ "$TASK_ARN" != "None" ]; then
ENI_ID=$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ARN" --region "$REGION" \
--query "tasks[0].attachments[?status=='ATTACHED']|[0].details[?name=='networkInterfaceId'].value | [0]" \
--output text 2>/dev/null || true)
if [ -n "$ENI_ID" ] && [ "$ENI_ID" != "None" ]; then
PUBLIC_IP=$(aws ec2 describe-network-interfaces --network-interface-ids "$ENI_ID" --region "$REGION" \
--query 'NetworkInterfaces[0].Association.PublicIp' --output text 2>/dev/null || true)
fi
[ -n "${PUBLIC_IP:-}" ] && [ "${PUBLIC_IP:-}" != "None" ] && break
fi
echo " waiting for task public IP... ($i/24)"
sleep 10
done
if [ -z "${PUBLIC_IP:-}" ] || [ "${PUBLIC_IP:-}" = "None" ]; then
echo -e "${RED}❌ Could not determine task public IP yet. Check the ECS console,${NC}"
echo " then run scripts/setup-route53.sh once the task is running."
exit 1
fi
echo -e "${GREEN}✅ Fleet is reachable at: http://${PUBLIC_IP}:8080${NC}"
echo ""
# --- Optional Route53 sync ----------------------------------------------------
echo -e "${YELLOW}Step 5/5: Route53 setup (optional)${NC}"
if [ "$ASSUME_YES" = false ]; then
read -r -p "Point a DNS record at this IP now? (yes/no): " SETUP_DNS
else
SETUP_DNS="no"
fi
if [ "$SETUP_DNS" = "yes" ]; then
read -r -p "Domain (e.g. fleet.example.com): " DOMAIN
read -r -p "Hosted zone ID (blank to auto-detect): " ZONE_ID
"${SCRIPT_DIR}/setup-route53.sh" "$DOMAIN" ${ZONE_ID:+"$ZONE_ID"}
echo ""
echo -e "${YELLOW}Tip: add a cron entry to keep DNS in sync on task restarts:${NC}"
echo " */5 * * * * ${SCRIPT_DIR}/setup-route53.sh ${DOMAIN} ${ZONE_ID:-} >>/tmp/fleet-dns.log 2>&1"
fi
echo ""
echo -e "${GREEN}🎉 Migration complete. Estimated new cost: ~\$48-52/month.${NC}"
+101
View File
@@ -0,0 +1,101 @@
#!/bin/bash
# ============================================================================
# setup-route53.sh - Point a Route53 DNS record at the Fleet ECS task public IP
# ============================================================================
# With no ALB, the Fleet task's public IP changes whenever the task restarts
# (Spot reclaim, deployment, etc.). This script discovers the current public
# IP and upserts an A record with a low TTL (60s).
#
# Usage:
# ./scripts/setup-route53.sh <fleet.example.com> [hosted-zone-id]
#
# Environment overrides:
# FLEET_CLUSTER (default: fleet)
# FLEET_SERVICE (default: fleet)
# RECORD_TTL (default: 60)
# AWS_REGION (default: aws cli default)
#
# Tip: run from cron every 5 minutes to keep the record fresh:
# */5 * * * * /path/to/scripts/setup-route53.sh fleet.example.com Z123456 >>/tmp/fleet-dns.log 2>&1
# ============================================================================
set -euo pipefail
DOMAIN="${1:-}"
ZONE_ID="${2:-}"
CLUSTER="${FLEET_CLUSTER:-fleet}"
SERVICE="${FLEET_SERVICE:-fleet}"
TTL="${RECORD_TTL:-60}"
REGION="${AWS_REGION:-$(aws configure get region 2>/dev/null || echo us-east-2)}"
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
if [ -z "$DOMAIN" ]; then
echo "Usage: $0 <fleet.example.com> [hosted-zone-id]"
exit 1
fi
command -v aws >/dev/null 2>&1 || { echo -e "${RED}❌ aws cli not found${NC}"; exit 1; }
echo "🔍 Looking up current Fleet task public IP (cluster=$CLUSTER service=$SERVICE region=$REGION)..."
TASK_ARN=$(aws ecs list-tasks --cluster "$CLUSTER" --service-name "$SERVICE" --region "$REGION" \
--query 'taskArns[0]' --output text)
if [ -z "$TASK_ARN" ] || [ "$TASK_ARN" = "None" ]; then
echo -e "${RED}❌ No running tasks found for service ${SERVICE}${NC}"
exit 1
fi
ENI_ID=$(aws ecs describe-tasks --cluster "$CLUSTER" --tasks "$TASK_ARN" --region "$REGION" \
--query "tasks[0].attachments[?status=='ATTACHED']|[0].details[?name=='networkInterfaceId'].value | [0]" \
--output text)
if [ -z "$ENI_ID" ] || [ "$ENI_ID" = "None" ]; then
echo -e "${RED}❌ Could not find network interface for task ${TASK_ARN}${NC}"
exit 1
fi
PUBLIC_IP=$(aws ec2 describe-network-interfaces --network-interface-ids "$ENI_ID" --region "$REGION" \
--query 'NetworkInterfaces[0].Association.PublicIp' --output text)
if [ -z "$PUBLIC_IP" ] || [ "$PUBLIC_IP" = "None" ]; then
echo -e "${RED}❌ Task has no public IP (assign_public_ip not enabled?)${NC}"
exit 1
fi
echo -e "${GREEN}✅ Task public IP: ${PUBLIC_IP}${NC}"
# Resolve hosted zone ID from the domain if not provided
if [ -z "$ZONE_ID" ]; then
# Walk up the domain labels until a hosted zone matches (e.g. fleet.example.com -> example.com)
NAME="$DOMAIN"
while [ -n "$NAME" ]; do
ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name "${NAME}." \
--query "HostedZones[?Name=='${NAME}.'] | [0].Id" --output text 2>/dev/null | cut -d/ -f3)
[ -n "$ZONE_ID" ] && [ "$ZONE_ID" != "None" ] && break
NAME="${NAME#*.}" # strip left-most label
[[ "$NAME" != *.* ]] && { ZONE_ID=""; break; }
done
if [ -z "$ZONE_ID" ] || [ "$ZONE_ID" = "None" ]; then
echo -e "${RED}❌ No Route53 hosted zone found for ${DOMAIN}${NC}"
echo " Pass the zone ID explicitly: $0 $DOMAIN <zone-id>"
exit 1
fi
fi
echo "🌐 Upserting A record: ${DOMAIN} -> ${PUBLIC_IP} (zone ${ZONE_ID}, TTL ${TTL})"
aws route53 change-resource-record-sets --hosted-zone-id "$ZONE_ID" --change-batch "{
\"Changes\": [{
\"Action\": \"UPSERT\",
\"ResourceRecordSet\": {
\"Name\": \"${DOMAIN}\",
\"Type\": \"A\",
\"TTL\": ${TTL},
\"ResourceRecords\": [{\"Value\": \"${PUBLIC_IP}\"}]
}
}]
}" >/dev/null
echo -e "${GREEN}✅ Done. Access Fleet at: http://${DOMAIN}:8080${NC}"
echo -e "${YELLOW}⚠️ Note: plain HTTP only unless Fleet TLS certs are configured.${NC}"