Assist In Root Cause Analysis | 🏗️ Build A Debugging Dashboard

Assist In Root Cause Analysis | 🏗️ Build A Debugging Dashboard



Tópico: Assist In Root Cause Analysis | 🏗️ Build A Debugging Dashboard
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
Exam Guide: Developer - Associate

🏗️ Domain 4:  Troubleshooting And Optimization

📘 Task 1: Assist In A Root Cause Analysis

When something breaks in production, you need to find out why. Fast. This task tests your ability to use CloudWatch Logs, Logs Insights, CloudWatch metrics, dashboards, and CloudTrail to debug application issues. Writing Logs Insights queries. Understanding Lambda-specific log fields. Using Embedded Metric Format for custom metrics. And recognizing common failure patterns across AWS service integrations.



📘 Concepts




CloudWatch Logs Architecture


Component
What It Is
Example

Log Group
A collection of log streams that share retention and access settings
/aws/lambda/my-function

Log Stream
A sequence of log events from a single source
2024/01/15/[$LATEST]abc123

Log Event
A single log entry with a timestamp and message
{"level":"ERROR","message":"DynamoDB timeout"}

Retention
How long logs are kept (1 day to 10 years, or never expire)

30 days for production

Subscription Filter
Real-time stream of log events to another destination
Stream to Kinesis, Lambda, or OpenSearch

Metric filter
Extract metric values from log text patterns
Count ERROR occurrences

💡 Lambda automatically creates a log group named /aws/lambda/{function-name}. Each function instance creates a new log stream. Log retention defaults to "Never expire". Always set a retention policy to control costs. You need logs:CreateLogGroup and logs:PutLogEvents permissions in the Lambda execution role.



CloudWatch Logs Insights Query Syntax


Logs Insights is a query language for searching and analysing log data. The exam tests specific query commands:

Command
Purpose
Example

fields
Select which fields to display
fields @timestamp, @message

filter
Filter log events by condition
filter @message like /ERROR/

stats
Aggregate data
stats count(*) by bin(5m)

sort
Order results
sort @timestamp desc

limit
Cap the number of results
limit 20

parse
Extract fields from unstructured log text
parse @message "user=* action=*" as user, action

display
Choose which fields appear in results
display @timestamp, @message

Common Query Patterns

Find the most recent errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Count errors per 5-minute window

filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc

Find slow Lambda invocations

filter @duration > 5000
| fields @requestId, @duration, @billedDuration, @memorySize, @maxMemoryUsed
| sort @duration desc
| limit 10

Parse custom log fields

parse @message '"level":"*","requestId":"*","error":"*"' as level, reqId, error
| filter level = "ERROR"
| stats count(*) by error

Find cold starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @memorySize
| sort @initDuration desc
| limit 20

💡The parse command uses glob patterns (*) to extract fields from unstructured text. The stats command supports count(), sum(), avg(), min(), max(), pct() (percentiles). Use bin() to group by time intervals. The @ prefix denotes system-generated fields like @timestamp, @message, @logStream, etc.



Lambda-Specific Log Fields


Lambda automatically adds these fields to REPORT lines:

Field
What It Contains
Why It Matters

@type
Log line type: START, END, REPORT

Filter for REPORT to get invocation summaries

@requestId
Unique invocation ID
Correlate logs for a single invocation

@duration
Actual execution time (ms)
Performance monitoring

@billedDuration
Rounded up to nearest ms (or 1ms minimum)
Cost tracking

@memorySize
Configured memory (MB)
Right-sizing analysis

@maxMemoryUsed
Peak memory used (MB)
Detect memory pressure

@initDuration
Cold start initialization time (ms)
Only present on cold starts

@xrayTraceId
X-Ray trace ID
Link logs to traces

Memory right-sizing query

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMem,
max(@maxMemoryUsed) as peakMem,
avg(@memorySize) as configuredMem,
avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

💡 If @maxMemoryUsed is consistently close to @memorySize, the function may be running out of memory. If it's consistently low (under 50%), you're over-provisioned and wasting money. The @initDuration field only appears on cold starts. If you see it frequently, consider provisioned concurrency.



CloudWatch Embedded Metric Format (EMF) vs PutMetricData


There are two ways to publish custom metrics.

Aspect
Embedded Metric Format (EMF)
PutMetricData API

How It Works

Write structured JSON to stdout: CloudWatch extracts metrics automatically
Call the CloudWatch API directly

Latency

Async (written with logs)
Sync (API call blocks execution)

Cost

Log ingestion cost only (no per-metric API charge)
$0.01 per 1,000 PutMetricData API calls

Dimensions
Up to 30 dimensions
Up to 30 dimensions

Resolution
Standard (60s) or high (1s)
Standard (60s) or high (1s)

Batching

Automatic (one log line = one metric data point)
Manual (up to 1,000 values per API call)

Lambda Overhead

None. Just a print statement
API call adds latency to each invocation

Best For
Lambda functions, high-throughput applications
EC2 applications, infrequent metric publishing

EMF format

import json

def lambda_handler(event, context):
# Your business logic here
order_total = process_order(event)

# Emit EMF metric. CloudWatch extracts this automatically
print(json.dumps({
"_aws": {
"Timestamp": 1234567890000,
"CloudWatchMetrics": [{
"Namespace": "OrderService",
"Dimensions": [["Environment", "OrderType"]],
"Metrics": [
{"Name": "OrderTotal", "Unit": "None"},
{"Name": "ProcessingTime", "Unit": "Milliseconds"}
]
}]
},
"Environment": "production",
"OrderType": "standard",
"OrderTotal": order_total,
"ProcessingTime": 145
}))

PutMetricData approach

import boto3

cloudwatch = boto3.client('cloudwatch')

def lambda_handler(event, context):
order_total = process_order(event)

# Sync API call — adds latency
cloudwatch.put_metric_data(
Namespace='OrderService',
MetricData=[{
'MetricName': 'OrderTotal',
'Value': order_total,
'Unit': 'None',
'Dimensions': [
{'Name': 'Environment', 'Value': 'production'},
{'Name': 'OrderType', 'Value': 'standard'}
]
}]
)

💡> EMF is the preferred approach for Lambda because it adds zero latency. You're just writing to stdout, which Lambda already sends to CloudWatch Logs. CloudWatch automatically extracts the metrics from the structured JSON. Use PutMetricData when you need to publish metrics from EC2 or when you need more control over metric timing.



Common Integration Failure Patterns


Scenarios to identify the root cause. Know these patterns:

Symptom
Likely Cause
How to Debug
Fix

502 Bad Gateway (API GW)
Lambda returned invalid response format
Check Lambda logs for response structure
Return statusCode, headers, body

504 Gateway Timeout (API GW)
Lambda exceeded API Gateway's 29-second timeout
Check @duration in logs
Reduce Lambda timeout, optimize code, or use async

429 Too Many Requests
Lambda concurrency throttled or API GW rate limit
Check Throttles metric
Increase reserved concurrency or request limit increase

Access Denied
Missing IAM permissions
Check CloudTrail for AccessDenied events
Update IAM policy

Task timed out (Lambda)
Function exceeded its configured timeout
Check @duration vs timeout setting
Increase timeout or optimize code

Out of memory
Function exceeded configured memory
Check @maxMemoryUsed vs @memorySize

Increase memory allocation

Connection timeout
Lambda in VPC can't reach internet or service
Check VPC config, NAT Gateway, security groups
Add NAT Gateway or VPC endpoint

ResourceNotFoundException
DynamoDB table or resource doesn't exist in the region
Check region configuration
Verify resource exists in the correct region



502 Bad Gateway Deep Dive


The most common Lambda + API Gateway error. API Gateway expects this exact response format:

# CORRECT: API Gateway accepts this
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json'},
'body': json.dumps({'message': 'success'})  # body must be a STRING
}

# WRONG: causes 502
return {
'statusCode': 200,
'body': {'message': 'success'}  # body is a dict, not a string!
}

# WRONG: causes 502
return "success"  # not the expected format at all

# WRONG: causes 502 (unhandled exception)
def handler(event, context):
raise Exception("oops")  # no try/except = 502

💡 A 502 from API Gateway almost always means the Lambda function returned a response that doesn't match the expected format, or the function threw an unhandled exception. Check the Lambda logs first: the error will be there. The API Gateway execution logs (if enabled) will show "Malformed Lambda proxy response."



CloudTrail for Debugging Permission Issues


CloudTrail Feature
What It Does
Use Case

Event History

Last 90 days of management events (free)
Quick lookup of recent API calls

Trail

Continuous logging to S3 (all events)
Long-term audit, compliance

Data Events
S3 object-level and Lambda invocation events
Debug specific resource access

Insights
Detect unusual API activity
Identify anomalous patterns

Finding AccessDenied events

1. Open CloudTrail console → Event history

2. Filter by Event name or Error code

3. Look for events with errorCode: AccessDenied or errorCode: UnauthorizedAccess

4. The event details show:

•
userIdentity: who made the call (which role/user)

•
eventName: what API was called (e.g., PutItem)

•
requestParameters: what resource was targeted

•
errorMessage: the specific permission that's missing

💡 CloudTrail logs every AWS API call. When you get an AccessDenied error, CloudTrail tells you exactly which principal tried to call which API on which resource and which permission was missing. This is the fastest way to debug IAM issues. Event history is free and covers the last 90 days.



Deployment Failure Troubleshooting


Failure Type
Where to Look
Common Causes

CloudFormation stack failure
CloudFormation Events tab
IAM permissions, resource limits, invalid template

CodeBuild failure
CodeBuild build logs
Missing dependencies, test failures, buildspec syntax

CodeDeploy failure
CodeDeploy deployment logs
Hook script errors, health check failures, timeout

Lambda deployment failure
CloudWatch Logs + CodeDeploy
New version crashes, alarm triggers rollback

SAM deploy failure
CloudFormation Events + SAM CLI output
Transform errors, packaging issues, capability missing

💡 CloudFormation failures show the specific resource and error in the Events tab. Look for CREATE_FAILED or UPDATE_FAILED status. The most common cause is insufficient IAM permissions because the CloudFormation role doesn't have permission to create the resource. For SAM, make sure you include CAPABILITY_IAM and CAPABILITY_AUTO_EXPAND when deploying.



💡 Build A Debugging Dashboard


Build a Debugging Dashboard from scratch using the AWS Console:

• A Lambda function that emits structured logs and EMF custom metrics

• CloudWatch Logs Insights queries that find errors, slow invocations, and cold starts

• A CloudWatch dashboard displaying key application metrics

• Simulated failures (timeout, permission denied, 502) with debugging walkthroughs

• CloudTrail queries to find access denied events



Prerequisites


• An AWS Account

• Python



Part I




Create a Lambda Function with Structured Logging and EMF


Step 01: Create the Function

Open the Lambda console → Create function

•
Function name: DebugDemoFunction

•
Runtime: Python 3.13

Click Create function

Step 02: Add Structured Logging with EMF

import json
import time
import os
import random

def emit_emf_metric(metric_name, value, unit, dimensions):
"""Emit a CloudWatch metric using Embedded Metric Format."""
print(json.dumps({
"_aws": {
"Timestamp": int(time.time() * 1000),
"CloudWatchMetrics": [{
"Namespace": "DebugDemo",
"Dimensions": [list(dimensions.keys())],
"Metrics": [{"Name": metric_name, "Unit": unit}]
}]
},
**dimensions,
metric_name: value
}))

def lambda_handler(event, context):
start_time = time.time()
action = event.get('action', 'process')
request_id = context.aws_request_id

# Structured log entry
log_entry = {
"level": "INFO",
"requestId": request_id,
"action": action,
"timestamp": time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
"functionName": context.function_name,
"memoryLimit": context.memory_limit_in_mb
}

try:
if action == 'slow':
# Simulate a slow operation
delay = random.uniform(2, 5)
log_entry["level"] = "WARN"
log_entry["message"] = f"Slow operation detected: {delay:.1f}s"
time.sleep(delay)

elif action == 'error':
# Simulate an error
raise ValueError("Invalid order data: missing required field 'customerId'")

elif action == 'oom':
# Simulate high memory usage
log_entry["message"] = "Processing large dataset"
data = "x" * (50 * 1024 * 1024)  # 50MB string

else:
# Normal processing
log_entry["message"] = "Order processed successfully"
time.sleep(random.uniform(0.1, 0.5))

print(json.dumps(log_entry))

# Emit EMF metrics
processing_time = (time.time() - start_time) * 1000
emit_emf_metric("ProcessingTime", processing_time, "Milliseconds",
{"Environment": "demo", "Action": action})
emit_emf_metric("SuccessCount", 1, "Count",
{"Environment": "demo", "Action": action})

return {
'statusCode': 200,
'body': json.dumps({
'message': 'Processed successfully',
'action': action,
'requestId': request_id,
'processingTime': f"{processing_time:.0f}ms"
})
}

except Exception as e:
processing_time = (time.time() - start_time) * 1000
log_entry["level"] = "ERROR"
log_entry["error"] = str(e)
log_entry["errorType"] = type(e).__name__
print(json.dumps(log_entry))

emit_emf_metric("ErrorCount", 1, "Count",
{"Environment": "demo", "Action": action})

return {
'statusCode': 500,
'body': json.dumps({
'error': str(e),
'requestId': request_id
})
}

Click Deploy

Step 03: Generate Test Data

Go to the Test tab

Create and run these test events multiple times:

Normal processing

{"action": "process"}

Slow operation

{"action": "slow"}

Error case

{"action": "error"}

⚠️ Run each test event 5-10 times to generate enough log data



Part II




Write CloudWatch Logs Insights Queries


Step 01: Open Logs Insights

Open the CloudWatch console → ** ▼ Logs** → Log Management

Select the log group: /aws/lambda/DebugDemoFunction

⚠️ Set the time range to the last 1 hour

→ View in Logs Insights

Query 01: Find All Errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 25

Click Run query. You should see the structured error logs from the error test events.

Query 02: Analyze Invocation Performance

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
max(@duration) as maxDuration,
min(@duration) as minDuration,
count(*) as invocations

Query 03: Find Cold Starts

filter @type = "REPORT" and @initDuration > 0
| fields @requestId, @initDuration, @duration, @maxMemoryUsed
| sort @initDuration desc

Query 04: Parse Structured Logs for Error Breakdown

parse @message '"level":"*","requestId":"*"' as level, reqId
| filter level = "ERROR"
| stats count(*) as errorCount by level

Query 05: Memory Utilization Analysis

filter @type = "REPORT"
| stats avg(@maxMemoryUsed) as avgMemUsed,
max(@maxMemoryUsed) as peakMemUsed,
avg(@memorySize) as configuredMem,
avg(@maxMemoryUsed) / avg(@memorySize) * 100 as utilizationPct

Query 06: Slow Invocations Over Time

filter @type = "REPORT"
| stats avg(@duration) as avgDuration,
pct(@duration, 95) as p95Duration,
pct(@duration, 99) as p99Duration
by bin(5m)
| sort bin(5m) asc

💡 Save useful queries by clicking **Save in Logs Insights. You can also add query results directly to a CloudWatch dashboard. The pct() function calculates percentiles: pct(@duration, 95) gives you the 95th percentile duration, which is more useful than averages for identifying tail latency.**



Part III




Create a CloudWatch Dashboard


Step 01: Build the Dashboard

Open the CloudWatch console → Dashboards → Create dashboard

Dashboard name: DebugDemo-Dashboard

Click Create dashboard

Step 02: Add an Invocation Count Widget

Click Add widget → Number

Select Lambda → By Function Name → DebugDemoFunction → Invocations

Period: 5 minutes

Click Create widget

Step 03: Add an Error Rate Widget

Click Add widget → Line

Add two metrics:

• Lambda → By Function Name → DebugDemoFunction → Errors

• Lambda → By Function Name → DebugDemoFunction → Invocations

Click Create widget

Step 04: Add a Duration Widget

Click Add widget → Line

Select Lambda → By Function Name → DebugDemoFunction → Duration

Add statistics: Average, p99, Maximum

Click Create widget

Step 05: Add a Custom EMF Metric Widget

Click Add widget → Number

Select DebugDemo (custom namespace) → Environment, Action → ProcessingTime

Click Create widget

Step 06: Add a Logs Insights Query Widget

Click Add widget → Logs table

Select log group: /aws/lambda/DebugDemoFunction

Enter query:

filter @message like /ERROR/
| fields @timestamp, @message
| sort @timestamp desc
| limit 10

Click Create

Step 07:* Click **Save dashboard



Part IV




Simulate and Debug Common Failures


Step 01: Simulate a 502 Bad Gateway

Create a new Lambda function: BadResponseFunction and add this intentionally broken code

import json

def lambda_handler(event, context):
# BUG: body should be a string, not a dict
return {
'statusCode': 200,
'body': {'message': 'this will cause a 502'}  # Wrong!
}

Create an API Gateway REST API pointing to this function

💡 Test the API endpoint. You'll get a 502

⚠️ Debug steps:

• Check Lambda logs → the function executed successfully (no error)

• Check API Gateway execution logs → "Malformed Lambda proxy response"

•
Fix: Change 'body': {'message': '...'} to 'body': json.dumps({'message': '...'})

Step 02: Simulate a Timeout

On DebugDemoFunction, go to Configuration → General configuration → Edit

Set Timeout to 3 seconds

Run the slow test event (which sleeps 2-5 seconds)

⚠️ Debug steps:

• Check logs → Task timed out after 3.00 seconds

• Check the REPORT line → @duration close to 3000ms

• Query in Logs Insights:

filter @message like /Task timed out/
| fields @requestId, @duration, @memorySize

Fix: Increase timeout or optimize the slow operation

Step 03: Simulate Permission Denied

On DebugDemoFunction, update the code to call DynamoDB:

import boto3
import json

dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
table = dynamodb.Table('NonExistentTable')
try:
table.put_item(Item={'PK': 'test', 'SK': 'test'})
except Exception as e:
print(json.dumps({
"level": "ERROR",
"error": str(e),
"errorType": type(e).__name__
}))
return {
'statusCode': 500,
'body': json.dumps({'error': str(e)})
}

💡Run the test. You'll get an AccessDeniedException

Debug steps:

• Check Lambda logs → AccessDeniedException

• Open CloudTrail → Event history

• Filter by Event name: PutItem

• Find the event with errorCode: AccessDenied

• The event shows which role tried to access which resource



Part V




Use CloudTrail to Find Access Denied Events


Step 01: Search Event History

Open the CloudTrail console → Event history

Change the filter to Error code

Type AccessDenied and press Enter

You should see the PutItem event from the previous step

Step 02: Examine the Event Details

Click on the event to expand it

Key fields to examine

•
userIdentity.arn: the Lambda execution role that made the call

•
eventName: PutItem

•
requestParameters: the table name and item

•
errorCode: AccessDeniedException

•
errorMessage: the specific permission needed

💡To fix: add dynamodb:PutItem permission to the Lambda execution role for the specific table ARN

Step 03: Create a Logs Insights Query for CloudTrail

If you have a CloudTrail trail sending logs to CloudWatch:

fields eventTime, eventName, errorCode, errorMessage,
userIdentity.arn as principal,
requestParameters.tableName as table
| filter errorCode = "AccessDenied"
| sort eventTime desc
| limit 20

💡 CloudTrail Event history is free and covers the last 90 days of management events. For longer retention or data events (S3 object access, Lambda invocations), you need a trail. When debugging permission issues, CloudTrail is your first stop, it tells you exactly what permission is missing, who tried to use it, and on which resource.



🏗️ What You Built | 📘 Exam Concepts Recap


What You Built
Exam Concept

Emitted EMF metrics from structured logs
Zero-latency custom metrics via Embedded Metric Format

Wrote CloudWatch Logs Insights queries

fields, filter, stats, parse, sort, pct() syntax

Queried @initDuration for cold starts
Reading Lambda REPORT fields for performance

Built a CloudWatch dashboard with mixed widgets
Single pane of glass for application health

Reproduced and fixed a 502 Bad Gateway
Malformed Lambda proxy response (body must be string)

Reproduced a timeout and read the REPORT line
Diagnosing Task timed out from logs

Found AccessDenied events in CloudTrail
Debugging permission issues: principal, API, resource



⚠️ Clean Up Protocol


•
Lambda → Delete DebugDemoFunction and BadResponseFunction

•
API Gateway → Delete any test APIs created

•
CloudWatch → Delete the DebugDemo-Dashboard

•
CloudWatch → Delete log groups for the Lambda functions

•
IAM → Delete Lambda execution roles



Key Takeaways for the Exam


•
CloudWatch Logs Insights uses fields, filter, stats, parse, sort, and limit commands. Know the syntax for each.

•
Lambda REPORT lines contain @duration, @billedDuration, @memorySize, @maxMemoryUsed, and @initDuration (cold starts only). Use these for performance analysis.

•
EMF is preferred over PutMetricData for Lambda because it adds zero latency. Metrics are extracted from structured JSON written to stdout. Use PutMetricData for EC2 or infrequent publishing.

•
502 Bad Gateway from API Gateway almost always means the Lambda response format is wrong: body must be a string, not a dict.

•
504 Gateway Timeout means Lambda exceeded API Gateway's 29-second limit. The Lambda timeout must be less than 29 seconds for synchronous API Gateway integrations.

•
CloudTrail logs every AWS API call. Use Event history to find AccessDenied errors. It shows the principal, API, resource, and missing permission.

•
Metric filters extract CloudWatch metrics from log text patterns. Subscription filters stream log events to Kinesis, Lambda, or OpenSearch in real time.

•
CloudWatch dashboards can include metrics widgets, Logs Insights query results, and alarms giving you a single pane of glass for application health.

• The parse command in Logs Insights extracts fields from unstructured text using glob patterns (*). The stats command supports count(), avg(), sum(), min(), max(), and pct() for percentiles.

• When debugging deployment failures, check CloudFormation Events (stack failures), CodeBuild logs (build failures), and CodeDeploy logs (deployment failures). Each service has its own log location.



Additional Resources


• CloudWatch Logs Insights language query syntax

• Specification: Embedded metric format

• Monitoring, debugging, and troubleshooting Lambda functions

• Working with CloudTrail event history

• Creating a customized CloudWatch dashboard

• Troubleshooting issues in Lambda

🏗️