How to set up Astra Traffic Monitoring with Nginx in Cloud VM

Here's how to configure nginx with OpenTelemetry (Otel) for traffic monitoring, in cloud VM environments such as AWS, GCP, Azure and Digital Ocean. It also covers troubleshooting common issues encountered during configuration.

Nginx integration

Case 1: Nginx not present in my VM

This section details the steps required to install and configure the Nginx load balancer to successfully instrument incoming HTTP requests.

Pre-requisite:

Set up a VM to host the Nginx load balancer.

Install Docker and Docker-Compose on the VM.

Prepare SSL certificates for the Nginx server. If you don’t have them, generate a self-signed SSL certificate .

Steps:

SSH into the VM and navigate to your working directory.

mkdir ~/astra-obs
cd ~/astra-obs

Create nginx.conf file with following content.

Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where traffic collector is hosted.

load_module modules/ngx_http_js_module.so;
load_module modules/ngx_otel_module.so;

user nginx;
worker_processes auto;

pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;

events {
      worker_connections 1024;
      # multi_accept on;
}

http {
      # Basic Settings
      sendfile on;
      tcp_nopush on;
      types_hash_max_size 2048;

      include /etc/nginx/mime.types;
      default_type application/octet-stream;

      # SSL Settings
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
      ssl_prefer_server_ciphers on;

      # Logging Settings
      access_log /var/log/nginx/access.log;

      # Gzip Settings
      gzip on;

      # Virtual Host Configs
      include /etc/nginx/conf.d/*.conf;
      include /etc/nginx/sites-enabled/*;

      js_import ga_nginx_script.js;
      js_set $req_headers ga_nginx_script.req_headers_json;
      js_set $resp_headers ga_nginx_script.resp_headers_json;
      otel_exporter {
          endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint

          interval    5s;
          batch_size  512;
          batch_count 4;
      }
          
      server {
          listen 19443 ssl;
          server_name nginx;
          ssl_certificate     /etc/nginx/server.crt;
          ssl_certificate_key /etc/nginx/server.key;

          location / {
            #...
            otel_trace on;
            otel_trace_context propagate;
            otel_span_attr sensor.version $nginx_version;

            otel_span_attr sensor.id <your_sensor_id>;

            otel_span_attr url.query $query_string;
            otel_span_attr http.request.body $request_body;
            otel_span_attr http.user_agent $http_user_agent;
            otel_span_attr http.referer $http_referer;
            otel_span_attr http.request.headers $req_headers;
            otel_span_attr http.response.headers $resp_headers;
            
            #...
            #proxy_pass http://backend;
            #...
        }
      }
}

Edit the nginx.conf created in previous step to serve the incoming requests by giving the address of backend servers.

Configure reverse proxy - http
Configure reverse proxy - https

Create ga_nginx_script.js file with following content.

function req_headers_json(r) {
     return JSON.stringify(r.headersIn);
}

function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
}

export default { req_headers_json, resp_headers_json };

Create docker-compose.yaml file with following content.
Replace /path/to/server.crt and /path/to/server.key with the absoulte path to your certificate and key.
Replace ~/astra-obs/nginx.conf and ~/astra-obs/ga_nginx_script.js with absolute path to nginx.conf and ga_nginx_script.js respectively.

version: '3.7'

services:

 otel-collector:
   image: nginx:otel
   volumes:
     - /path/to/server.crt:/etc/nginx/server.crt
     - /path/to/server.key:/etc/nginx/server.key
     - ~/astra-obs/nginx.conf:/etc/nginx/nginx.conf
     - ~/astra-obs/ga_nginx_script.js:/etc/nginx/ga_nginx_script.js
   network_mode: host
   restart: always

Start the nginx container by executing sudo docker compose up -d from working directory.

Expect the following output when you exec: sudo docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29f430a5ac1f nginx:otel "/docker-entrypoint.…" About a minute ago Up About a minute nginx-lb-container


To stop the nginx container executing sudo docker compose down from working directory.

Case 2: Nginx already present in my VM

Take backup of current nginx configuration. Usually found in /etc/nginx/nginx.conf

It's essential to inject custom configuration into nginx config to have nginx successfully instrument the incoming request and response. Although these customization doesn't alter the functionality of nginx, we strongly recommend to take the backup of existing nginx configuration

Register nginx repository in your VM by following this . After registering, run following to install required modules

Run update: sudo apt update
Install otel module: sudo apt install -y nginx-module-otel
Install js module: sudo apt install -y nginx-module-njs

Create ga_nginx_script.js file with following content under /etc/nginx/

function req_headers_json(r) {
     return JSON.stringify(r.headersIn);
 }

 function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
 }

 export default { req_headers_json, resp_headers_json };

Edit the /etc/nginx/nginx.conf file with following content to instrument the incoming requests.
Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where Traffic collector is hosted.

load_module modules/ngx_http_js_module.so;
load_module modules/ngx_otel_module.so;

user  nginx;
#...
#...events{}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    #...
    js_import ga_nginx_script.js;
    js_set $req_headers ga_nginx_script.req_headers_json;
    js_set $resp_headers ga_nginx_script.resp_headers_json;
    otel_exporter {
        endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint

        interval    5s;
        batch_size  512;
        batch_count 4;
    }
    #...
    server {
        #...

        location / {
            #...
            otel_trace on;
            otel_trace_context propagate;
            otel_span_attr sensor.version $nginx_version;

            otel_span_attr sensor.id <your_sensor_id>;

            otel_span_attr url.query $query_string;
            otel_span_attr http.request.body $request_body;
            otel_span_attr http.user_agent $http_user_agent;
            otel_span_attr http.referer $http_referer;
            otel_span_attr http.request.headers $req_headers;
            otel_span_attr http.response.headers $resp_headers;
            #...
            #proxy_pass http://backend;
            #...
        }
    }
}

Restart nginx

systemctl restart nginx

Troubleshooting

Unable to send trace from nginx to traffic collector

Symptoms

No entries in inventory/ inventory not getting updated

Following or similar error seen in nginx log

[Error] File: /tmp/build/opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "DNS resolution failed for ...

Cause

Nginx is unable to resolve traffic-collector address

Solution

If Address of the traffic collector given in nginx.conf is incorrect, locate the http block in your nginx configuration file, find the otel_exporter {...} block, update IPAddressOfTrafficCollector as shown below

otel_exporter {
  endpoint  <IPAddressOfTrafficCollector>:4317
  ...
}

IPAddressOfTrafficCollector should be the your public ip of the collector server instance

Restart nginx by running nginx -s reload

Unable to see entries in inventory

Symptoms

No entries in inventory/ inventory not getting updated

No error in nginx/traffic-collector log

Cause

Unregistered hostname

Solution

Double check if the hostname is registered under Scope URI for Report in Target setup page

FAQ (Frequently Asked Questions)

Can I see what trace are sent from my environment?

Yes, one can see the traces sent by traffic-collector by inspecting docker logs. Run docker logs astra-traffic-collector

How to get IP Address Of Traffic Collector in Non Kubernetes environment?

As container network is the same network as host network, the IP of the container would be the same as the Virtual Machine IP.

Updated on: 02/12/2024