Introduction

AWS delivers CloudWatch metrics in OpenTelemetry format via Firehose, but AppSignal uses its own internal format. Building the parser to bridge these two formats presented several technical challenges.

The metrics arriving through this pipe power AWS automated dashboards. When AppSignal detects metrics from a supported AWS service, it creates a dashboard for it automatically, with pre-built charts grouped by category: compute, databases, networking, messaging, storage, and others. The rest of this post is what we had to do inside the parser to make the data on those dashboards line up with what AWS shows.

Data format

The Firehose delivery format wraps everything in a JSON envelope with base64-encoded records. Those records might be gzip-compressed, or they might not be. The parser needs to handle both. Inside each record are one or more OpenTelemetry protobuf messages, concatenated together with varint length prefixes (LEB128 encoding). The structure is deeply nested: resource_metrics contains scope_metrics, which contains metrics, which finally contains data_points.

The Dimensions problem

CloudWatch doesn't send one data point per metric. For something like TotalMetricUpdate, you get an aggregate data point with no Dimensions attribute, plus multiple dimensional data points with specific Dimensions like MetricStreamName.

The aggregate is not the sum of the dimensional values. They're separate pre-aggregated data points that CloudWatch sends independently. The initial implementation combined them, so dashboard values didn't match what AWS showed.

The solution was to extract Dimensions as tags, creating separate time series for each tag combination. The aggregate and dimensional data points stay properly separated.

Unit conversions

Duration metrics arrive in whatever unit the service decided to use. Lambda sends seconds, other services might send milliseconds or microseconds. We normalize everything to milliseconds. Percentage metrics needed special handling too. CloudWatch represents them as numeric values with a "%" unit, but AppSignal has a dedicated Percentage type with total_count and subset_count.

We use gauges instead of counters because CloudWatch data is already pre-aggregated. Using counters would double-count everything.

Metric naming

AWS uses conventions like AWS/Lambda/Duration, but we transform those to snake*case like aws_lambda_duration. Same for metric names with dots: DeliveryToHttpEndpoint.Bytes becomes delivery_to_http_endpoint_bytes. This isn't just about style. It enables wildcard matching in dashboards, so you can show all Lambda metrics with aws_lambda*\*.

Where the metrics land

Once the parser has normalized the data, AppSignal matches the metric names against its list of supported AWS services and creates a automated dashboard per service: AWS/Lambda, AWS/RDS, AWS/SQS, and so on. The full list of supported services and namespaces (across compute, databases, networking, messaging, storage, security, AI/ML, developer tools, and more) is in the AWS dashboards docs.

Each chart uses wildcard tag filters that take advantage of the naming and tagging the parser produces. The Lambda Errors chart, for example, shows one line per function tagged by FunctionName, and you can narrow it from the chart options. Charts are ordered by urgency of signal rather than resource type: health and errors first, then performance, then capacity. Chart descriptions say what values indicate a problem and what to do about it, instead of restating the metric name.

Setting it up

The pipeline itself is standard AWS plumbing: a CloudWatch Metric Stream emitting OpenTelemetry 1.0 to a Firehose delivery stream that posts to AppSignal's HTTPS endpoint. There are two ways to wire it up:

• Console setup, one resource at a time.
• CloudFormation, one template covering the S3 failure bucket, IAM roles, Firehose stream, and metric stream.

By default the stream covers every namespace in the account. To narrow it, add IncludeFilters to the metric stream resource:

IncludeFilters:
  - Namespace: AWS/EC2
  - Namespace: AWS/RDS
  - Namespace: AWS/ELB

Refer to the docs for the full template, deployment steps, and multi-region notes.

CloudWatch logs

The same Firehose-based delivery works for CloudWatch logs, with a CloudFormation template for the automated path. The setup mirrors the metrics side: an S3 failure bucket, IAM roles for Firehose and CloudWatch, a Firehose delivery stream, and a subscription filter on the log group you want to forward. The only meaningful differences are the destination URL (/logs/aws-kinesis instead of /metrics/aws-cloudwatch) and the credential it expects (a log source API key instead of an App-level Push API key).

Get involved

AWS dashboards are currently in preview. The underlying pipeline that ingests CloudWatch metrics and normalizes them into the metric primitive that powers AppSignal is generally available, and we're using the dashboards work to shape which services to cover and how charts should be laid out.

• Try the AWS dashboards. Wire up a metric stream (console or CloudFormation) and tell us which services and charts you'd want to see, and what's missing from the ones you got.
• Join a UX research session. A 30-minute video call where you share what's working and what isn't. Reach us on Discord.
• Send feedback. Use the in-feature feedback button, or join us on Discord.