How to setup Astra Traffic Monitoring with AWS API Gateway

Here's how to configure AWS Rest and HTTP API Gateway with OpenTelemetry (Otel) for traffic monitoring. It also covers troubleshooting common issues encountered during configuration.

AWS APIGW integration

This integration is supported for the following type of AWS API Gateway:
REST API Gateway
HTTP API Gateway

AWS supports detailed tracing only for REST API Gateway and not for HTTP API Gateway. This means that limited information like URL Path, Status code, Domain and Protocol can be observed for HTTP API Gateway. Request/Response headers and body which can only be observed with detailed tracing can't be observed for HTTP API Gateway.

Quick Installation

Pre-requisites

Configure tracing and logging for the AWS REST API Gateway by following this and for AWS HTTP API Gateway by following this

Keep the AWS AccessKeyID and SecretAccessKey ready.

Launch a VM (EC2 instance) with following minimal hardware configuration in the same VPC where the Astra-Traffic-Collector is running.
- Memory: 2GB RAM, CPU: 2, Disk: 10GB

SSH into your VM and install Docker and Docker Compose in your VM by following the official doc.

Create a directory for the Astra Log Parser
Create logs folder and assign user role for volume mounting.

mkdir -p /opt/astra-log-parser/logs && chown -R 10001:10001 /opt/astra-log-parser/logs && cd /opt/astra-log-parser

Create docker-compose.yaml file

version: '3.3'
services:
  astra-log-parser:
    image: getastra/log-parser
    container_name: astra-log-parser
    volumes:
      - /opt/astra-log-parser/logs:/astra/logs
    network_mode: host
    env_file:
      - .env
    restart: always

Create .env file

Replace <sensorID> with the SENSOR_ID displayed during the creation of AWS API Gateway integration
Replace <host:port> with the address of astra-traffic-collector.
Replace <AWSAccessKeyID> with the AWS Access Key Id.
Replace <AWSSecretAccessKey> with the AWS Secret Access Key.
Replace <AWSRegion> with the AWS Region.
Replace <AWSCloudWatchLogGroupName> with the AWS CloudWatch Log Group Name associated with API Gateway.

SENSOR_ID=<sensorID>
OTLP_ENDPOINT=<host:port>
AWS_ACCESS_KEY_ID=<AWSAccessKeyID>
AWS_SECRET_ACCESS_KEY=<AWSSecretAccessKey>
AWS_REGION=<AWSRegion>
LOG_GROUP_NAME=<AWSCloudWatchLogGroupName>

Create linux service file under /etc/systemd/system/astra-log-parser.service
The service file allows the Astra Log Parser to be managed as a systemd service, enabling easy start, stop, restart, and automatic start on boot.

[Unit]
Description=Start Astra Log Parser
After=docker.service
Requires=docker.service

[Service]
Type=simple
ExecStart=/bin/bash -c "docker compose -f /opt/astra-log-parser/docker-compose.yaml up"
ExecStop=/bin/bash -c "docker compose -f /opt/astra-log-parser/docker-compose.yaml down"
Restart=always

[Install]
WantedBy=multi-user.target

Manage the service by using systemctl

# Enable the service to start on boot
sudo systemctl enable astra-log-parser

# Manage the service
sudo systemctl start | stop | restart | status astra-log-parser

# Disable the service
sudo systemctl disable astra-log-parser

Upgrade

Docker container upgrade

Execute the following commands to stop, upgrade and then start the service

systemctl stop astra-log-parser
cd /opt/astra-log-parser/
docker-compose pull
systemctl start astra-log-parser

Troubleshooting

I do not see my astra-log-parser running

Symptoms

Error seen when systemctl status astra-log-parser is executed

Following or similar error seen when docker logs astra-log-parser is executed

astra-log-parser   | Error parsing UUID SENSOR_ID: invalid UUID length: 0
astra-log-parser   | LOG_GROUP_NAME empty
astra-log-parser   | AWS_ACCESS_KEY_ID empty
astra-log-parser   | AWS_SECRET_ACCESS_KEY empty
astra-log-parser   | AWS_REGION empty

Cause

lack of required env variables

Solution

Edit /opt/astra-log-parser/.env and update it with right information.

Restart astra-log-parser
sudo systemctl stop astra-log-parser
sudo systemctl start astra-log-parser

Unable to see entries in inventory

Symptoms

No entries in inventory/ inventory not getting updated

No error in astra-log-parser or astra-traffic-collector log

Cause

Unregistered hostname

Solution

Double check if the hostname is registered under Scope URI for Report in Target setup page

FAQ (Frequently Asked Questions)

Can I see what trace are sent from my environment?

Yes, one can see the traces sent by traffic-collector by inspecting docker logs. Run docker logs astra-traffic-collector

Updated on: 31/01/2025