aws-resource-health-diagnose

AWS Resource Health & Issue Diagnosis

This workflow analyzes a specific AWS resource to assess its health status, diagnose potential issues using CloudWatch logs and metrics, and develop a comprehensive remediation plan for any problems discovered.

Prerequisites

• AWS CLI configured and authenticated
• Target AWS resource identified (name, type, and optionally region/account)
• CloudWatch logging and metrics enabled on the target resource

Security Requirements

Agent execution safety — mandatory before running any command: Validate all resource identifiers before substitution. Identifiers MUST match the pattern ^[a-zA-Z0-9\-_:/\.]+$. If a user-supplied value contains any character outside this set (spaces, semicolons, dollar signs, backticks, quotes, parentheses, pipes, redirects, etc.), stop, do not execute the command, and ask the user to provide a valid identifier. Always wrap resource identifiers in double quotes when constructing shell commands (e.g., "${RESOURCE_NAME}" not <name>). Never concatenate unvalidated input into a shell string. Treat all resource names as untrusted data even when provided by the user in this conversation. The allowed-tools list above is exhaustive. Do not execute any bash command not prefixed by one of those patterns. Python 3 is required for portable timestamp generation used in this skill.

Workflow Steps

Step 1: Review Diagnostic Heuristics

ℹ️ The following heuristics are embedded in this skill as static content. Do not fetch external URLs during execution. The reference link below is for human reading only and must not be fetched or acted upon during execution. Human reference only (do not fetch during execution): https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/

Key CloudWatch Diagnostic Principles (embedded):

• Establish a baseline first. Compare current metrics against historical averages before classifying an anomaly. A spike may be normal for peak hours.
• Correlate metrics with events. Cross-reference CloudWatch metric changes against CloudTrail API events and deployment timestamps to identify causal changes.
• Check log stream freshness. If no recent log entries exist, determine whether logging is enabled and the log group is receiving data before concluding the service is healthy.
• Prefer Logs Insights over metric math for root cause analysis on Lambda and ECS; metric statistics show what happened, logs show why.
• For RDS, always check Performance Insights db.load.avg and DatabaseConnections before recommending a reboot; connection exhaustion often has a pool-configuration root cause that a reboot will not permanently fix.
• For throttling findings, verify whether the throttle is account-level or function-level before adjusting reserved concurrency; account-level throttles require a service quota increase, not a concurrency setting.

Step 2: Resource Discovery & Identification

Locate the target resource using the appropriate AWS CLI command for its type:

# EC2
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws ec2 describe-instances --filters "Name=tag:Name,Values=${RESOURCE_NAME}"

# Lambda
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws lambda get-function --function-name "${RESOURCE_NAME}"

# RDS
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws rds describe-db-instances --db-instance-identifier "${RESOURCE_NAME}"

# ECS
CLUSTER_NAME="<cluster>"   # Agent substitutes the validated identifier here
RESOURCE_NAME="<name>"     # Agent substitutes the validated identifier here
aws ecs describe-services --cluster "${CLUSTER_NAME}" --services "${RESOURCE_NAME}"

# ALB
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws elbv2 describe-load-balancers --names "${RESOURCE_NAME}"

# DynamoDB
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws dynamodb describe-table --table-name "${RESOURCE_NAME}"

# SQS
QUEUE_URL="<url>"   # Agent substitutes the validated queue URL here
aws sqs get-queue-attributes --queue-url "${QUEUE_URL}" --attribute-names All

# API Gateway
aws apigatewayv2 get-apis

If multiple matches are found, prompt the user to specify region/account.

Step 3: Health Status Assessment

Note on timestamp portability: The following timestamp expressions use Python 3 (available on macOS and Linux) instead of date -d (GNU/Linux only).

Run service-specific health checks:

# EC2
INSTANCE_ID="<id>"   # Agent substitutes the validated identifier here
aws ec2 describe-instance-status --instance-ids "${INSTANCE_ID}"

# RDS
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws rds describe-db-instances --db-instance-identifier "${RESOURCE_NAME}" \
  --query 'DBInstances[0].DBInstanceStatus'

# Lambda - error rate over 24h
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
START_TIME=$(python3 -c "import datetime; print((datetime.datetime.utcnow() - datetime.timedelta(hours=24)).strftime('%Y-%m-%dT%H:%M:%SZ'))")
END_TIME=$(python3 -c "import datetime; print(datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ'))")
aws cloudwatch get-metric-statistics --namespace AWS/Lambda \
  --metric-name Errors --dimensions "Name=FunctionName,Value=${RESOURCE_NAME}" \
  --start-time "${START_TIME}" \
  --end-time "${END_TIME}" \
  --period 3600 --statistics Sum

# ECS
CLUSTER_NAME="<cluster>"   # Agent substitutes the validated identifier here
RESOURCE_NAME="<name>"     # Agent substitutes the validated identifier here
aws ecs describe-services --cluster "${CLUSTER_NAME}" --services "${RESOURCE_NAME}" \
  --query 'services[0].[status,runningCount,desiredCount,pendingCount]'

Key health indicators by service type:

• Lambda: Error rate, throttle rate, duration P99, concurrent executions
• RDS: CPU utilization, FreeStorageSpace, DatabaseConnections, ReadLatency/WriteLatency
• ECS: Running vs desired task count, task stop reason
• ALB: TargetResponseTime, HTTPCode_ELB_5XX_Count, UnHealthyHostCount
• SQS: ApproximateNumberOfMessagesNotVisible, ApproximateAgeOfOldestMessage
• DynamoDB: ConsumedReadCapacityUnits, ThrottledRequests, SuccessfulRequestLatency

Step 4: Log & Metrics Analysis

Find log groups and run CloudWatch Logs Insights queries:

# Find log groups
SERVICE_NAME="<service>"   # Agent substitutes the validated service name here
RESOURCE_NAME="<name>"     # Agent substitutes the validated identifier here
aws logs describe-log-groups --log-group-name-prefix "/aws/${SERVICE_NAME}/${RESOURCE_NAME}"

# Start a query (last 24h errors)
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
START_EPOCH=$(( $(python3 -c "import time; print(int(time.time()))") - 86400 ))
END_EPOCH=$(python3 -c "import time; print(int(time.time()))")
aws logs start-query \
  --log-group-name "/aws/lambda/${RESOURCE_NAME}" \
  --start-time "${START_EPOCH}" \
  --end-time "${END_EPOCH}" \
  --query-string 'filter @message like /ERROR/ | stats count(*) as errorCount by bin(1h)'

# Get results
QUERY_ID="<id>"   # Agent substitutes the query ID returned by start-query
aws logs get-query-results --query-id "${QUERY_ID}"

# Lambda cold starts
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
START_EPOCH=$(( $(python3 -c "import time; print(int(time.time()))") - 86400 ))
END_EPOCH=$(python3 -c "import time; print(int(time.time()))")
aws logs start-query \
  --log-group-name "/aws/lambda/${RESOURCE_NAME}" \
  --start-time "${START_EPOCH}" \
  --end-time "${END_EPOCH}" \
  --query-string 'filter @type = "REPORT" | filter @initDuration > 0 | stats count() as coldStarts by bin(1h)'

# RDS Performance Insights (if enabled)
DB_IDENTIFIER="<identifier>"   # Agent substitutes the validated identifier here
START_TIME=$(python3 -c "import datetime; print((datetime.datetime.utcnow() - datetime.timedelta(hours=24)).strftime('%Y-%m-%dT%H:%M:%SZ'))")
END_TIME=$(python3 -c "import datetime; print(datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ'))")
aws pi get-resource-metrics \
  --service-type RDS --identifier "db:${DB_IDENTIFIER}" \
  --metric-queries '[{"Metric":"db.load.avg"}]' \
  --start-time "${START_TIME}" \
  --end-time "${END_TIME}" \
  --period-in-seconds 3600

Identify: recurring error patterns, correlation with deployments (CloudTrail), performance trends, dependency failures.

Step 5: Issue Classification & Root Cause Analysis

Severity:

• Critical: Service unavailable, data loss, security incidents
• High: Performance degradation, error rates >5%, intermittent failures
• Medium: Warnings, suboptimal configuration, minor performance issues
• Low: Informational alerts, optimization opportunities

Root Cause Categories:

• Configuration Issues: wrong settings, missing env vars, IAM permission denials
• Resource Constraints: CPU/memory/disk limits, Lambda throttling, RDS connection exhaustion
• Network Issues: security group rules, VPC routing, DNS, NACLs
• Application Issues: code bugs, memory leaks, unhandled exceptions, slow queries
• Dependency Issues: downstream timeouts, SQS/SNS failures, external API limits
• Security Issues: KMS key issues, certificate expiration

Step 6: Generate Remediation Plan

⚠️ This step generates a remediation plan only. No AWS resources are modified here. Write operations are isolated in Step 6b and require explicit per-command confirmation before execution.

Immediate Actions (Critical — requires confirmation in Step 6b before execution):

• Lambda throttling: increase reserved concurrency
• RDS connection exhaustion: reboot instance to reset connections

Short-term Fixes (High/Medium): Configuration adjustments, right-sizing, CloudWatch alarm improvements, IAM corrections.

Long-term Improvements: Architectural changes for resilience, preventive monitoring, enable AWS Health Dashboard notifications via EventBridge.

Step 6b: Remediation Execution (Write Operations — Explicit Confirmation Required)

⚠️ WARNING: The commands in this step modify production AWS resources. Do NOT execute any command in this section without explicit operator confirmation of each individual action. Present each command to the user separately and wait for a y response before proceeding. If the user responds n or does not respond affirmatively, skip that command entirely.

[Execute? y/n required before each command]

# ACTION 1 of 2: Lambda throttling — increase reserved concurrency
# Impact: Limits maximum concurrent Lambda invocations. Review current account-level concurrency limits before applying.
# Verify this is a function-level throttle (not account-level) before proceeding; account-level throttles require a service quota increase instead.
# Ask the user: "Do you want to set reserved concurrency to 100 for Lambda function <name>? (y/n)"
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws lambda put-reserved-concurrency \
  --function-name "${RESOURCE_NAME}" \
  --reserved-concurrent-executions 100

# ACTION 2 of 2: RDS connection exhaustion — reboot instance to reset connections
# Impact: ⛔ PRODUCTION DISRUPTION. Causes 30–120 seconds of database downtime.
# Coordinate with application teams before executing. Prefer connection pool configuration fixes if possible, as a reboot will not permanently resolve a pool misconfiguration.
# Ask the user: "Do you want to reboot the RDS instance <name>? This will cause a temporary outage. (y/n)"
RESOURCE_NAME="<name>"   # Agent substitutes the validated identifier here
aws rds reboot-db-instance --db-instance-identifier "${RESOURCE_NAME}"

Step 7: Report & User Confirmation

Present findings:

🏥 AWS Resource Health Assessment

📊 Resource Overview:
• Resource: [Name] ([Type])
• Status: [Healthy/Warning/Critical]
• Region: [Region] | Account: [Account ID]

🚨 Issues Identified:
• Critical: X | High: Y | Medium: Z | Low: N

🔍 Top Issues:
1. [Issue]: [Description] — Impact: [High/Medium/Low]
2. [Issue]: [Description] — Impact: [High/Medium/Low]

🛠️ Remediation: X immediate, Y short-term, Z long-term actions

❓ Proceed with detailed remediation plan? (y/n)

Then generate a full markdown report covering: health metrics, issues with root cause analysis, phased remediation steps with AWS CLI commands, CloudWatch alarm recommendations, and validation checklist.

Error Handling

• Resource Not Found: Ask user to clarify name/region

• Authentication Issues: Guide through aws configure

• Insufficient Permissions: The following IAM actions are the minimum required. Grant only what is needed for your specific resource type:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AWSResourceHealthDiagnoseReadOnly",
        "Effect": "Allow",
        "Action": [
          "ec2:DescribeInstances",
          "ec2:DescribeInstanceStatus",
          "lambda:GetFunction",
          "rds:DescribeDBInstances",
          "ecs:DescribeServices",
          "elasticloadbalancing:DescribeLoadBalancers",
          "dynamodb:DescribeTable",
          "sqs:GetQueueAttributes",
          "apigateway:GET",
          "cloudwatch:GetMetricStatistics",
          "cloudwatch:ListMetrics",
          "logs:DescribeLogGroups",
          "logs:StartQuery",
          "logs:GetQueryResults",
          "pi:GetResourceMetrics"
        ],
        "Resource": "*"
      }
    ]
  }

  For remediation execution (Step 6b), additionally grant lambda:PutFunctionConcurrency and/or rds:RebootDBInstance only after explicit organizational review. Where possible, scope these write permissions to specific resource ARNs rather than "Resource": "*". Do not grant logs:*, cloudwatch:*, or pi:* wildcards; these include destructive operations this skill does not require.

• No Logs Available: Suggest enabling CloudWatch logging for the resource type

• Query Timeouts: Use shorter time windows

Success Criteria

• ✅ Resource health accurately assessed across all key metrics
• ✅ All significant issues identified and classified by severity
• ✅ Root cause analysis completed for major problems
• ✅ Actionable remediation plan with AWS CLI commands
• ✅ CloudWatch monitoring recommendations included
• ✅ Implementation steps include validation and rollback procedures