How to set up Astra Traffic Monitoring with Nginx in Cloud VM

Here's how to configure nginx with OpenTelemetry (Otel) for traffic monitoring, in cloud VM environments such as AWS, GCP, Azure and Digital Ocean. It also covers troubleshooting common issues encountered during configuration.

Nginx integration

Case 1: Nginx not present in my VM

This section details the steps required to install and configure the Nginx load balancer to successfully instrument incoming HTTP requests.

Pre-requisite:

Set up a VM to host the Nginx load balancer.

Install Docker and Docker-Compose on the VM.

Prepare SSL certificates for the Nginx server. If you don’t have them, generate a self-signed SSL certificate .

Steps:

SSH into the VM and navigate to your working directory.

mkdir ~/astra-obs
cd ~/astra-obs

Create a bare minimum nginx.conf file with following content.

user nginx;

worker_processes auto;

pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;

events {
      worker_connections 1024;
      # multi_accept on;
}

http {
      # Basic Settings
      sendfile on;
      tcp_nopush on;
      types_hash_max_size 2048;
      # server_tokens off;
      # server_names_hash_bucket_size 64;
      # server_name_in_redirect off;

      include /etc/nginx/mime.types;
      default_type application/octet-stream;

      # SSL Settings
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
      ssl_prefer_server_ciphers on;

      # Logging Settings
      access_log /var/log/nginx/access.log;

      # Gzip Settings
      gzip on;
      # gzip_vary on;
      # gzip_proxied any;
      # gzip_comp_level 6;
      # gzip_buffers 16 8k;
      # gzip_http_version 1.1;
      # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

      # Virtual Host Configs
      include /etc/nginx/conf.d/*.conf;
      include /etc/nginx/sites-enabled/*;
          
      server {
          #log_subrequest on;
      }
}

Edit the nginx.conf created in previous step to serve the incoming requests.

Configure nginx server
Certificates will be mounted later while creating docker-compose.yaml . Set ssl_certificate /etc/nginx/server.crt and ssl_certificate_key /etc/nginx/server.key in nginx.conf .
Configure reverse proxy - https
Configure reverse proxy - http

Edit the nginx.conf updated in the previous step with the following content to instrument the incoming requests.
Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where traffic collector is hosted.

load_module modules/ngx_http_js_module.so;
  load_module modules/ngx_otel_module.so;

  #user  nginx;
  #...
  http {
      #include       /etc/nginx/mime.types;
      #default_type  application/octet-stream;
      #...
      js_import ga_nginx_script.js;
      js_set $req_headers ga_nginx_script.req_headers_json;
      js_set $resp_headers ga_nginx_script.resp_headers_json;
      otel_exporter {
          endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint
          interval    5s;
          batch_size  512;
          batch_count 4;
      }
      #...
      server {
          #...

          location / {
              #...
              otel_trace on;
              otel_trace_context propagate;
              otel_span_attr sensor.version $nginx_version;
              otel_span_attr sensor.id <your_sensor_id>;
              otel_span_attr url.query $query_string;
              otel_span_attr http.request.body $request_body;
              otel_span_attr http.user_agent $http_user_agent;
              otel_span_attr http.referer $http_referer;
              otel_span_attr http.request.headers $req_headers;
              otel_span_attr http.response.headers $resp_headers;
              #...
              #proxy_pass http://backend;
              #...
          }
      }
  }

Create ga_nginx_script.js file with following content.

function req_headers_json(r) {

     return JSON.stringify(r.headersIn);
 }

 function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
 }

 export default { req_headers_json, resp_headers_json };

Create docker-compose.yaml file with following content.
Replace /path/to/server.crt and /path/to/server.key with the absoulte path to your certificate and key.

version: '3.7'

services:

 otel-collector:
   image: nginx:otel
   volumes:
     - /path/to/server.crt:/etc/nginx/server.crt
     - /path/to/server.key:/etc/nginx/server.key
     - /astra-obs/nginx.conf:/etc/nginx/nginx.conf
     - /astra-obs/ga_nginx_script.js:/etc/nginx/ga_nginx_script.js
   network_mode: host
   restart: always

Start the nginx container by executing docker-compose up -d from working directory.

To stop the nginx container executing docker-compose down from working directory.

Case 2: Nginx already present in my VM

Take backup of current nginx configuration. Usually found in /etc/nginx/nginx.conf

It's essential to inject custom configuration into nginx config to have nginx successfully instrument the incoming request and response. Although these customization doesn't alter the functionality of nginx, we strongly recommend to take the backup of existing nginx configuration

Register nginx repository in your VM

Install otel module: sudo apt install -y nginx-module-otel
Install js module: sudo apt install -y nginx-module-njs

Create ga_nginx_script.js file with following content under /etc/nginx/

function req_headers_json(r) {
     return JSON.stringify(r.headersIn);
 }

 function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
 }

 export default { req_headers_json, resp_headers_json };

Edit the nginx.conf file with following content to instrument the incoming requests.
Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where Traffic collector is hosted.

load_module modules/ngx_http_js_module.so;

  load_module modules/ngx_otel_module.so;

  user  nginx;
  #...
  http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;
      #...
      js_import ga_nginx_script.js;
      js_set $req_headers ga_nginx_script.req_headers_json;
      js_set $resp_headers ga_nginx_script.resp_headers_json;
      otel_exporter {
          endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint
          interval    5s;
          batch_size  512;
          batch_count 4;
      }
      #...
      server {
          #...

          location / {
              #...
              otel_trace on;
              otel_trace_context propagate;
              otel_span_attr sensor.version $nginx_version;
              otel_span_attr sensor.id <your_sensor_id>;
              otel_span_attr url.query $query_string;
              otel_span_attr http.request.body $request_body;
              otel_span_attr http.user_agent $http_user_agent;
              otel_span_attr http.referer $http_referer;
              otel_span_attr http.request.headers $req_headers;
              otel_span_attr http.response.headers $resp_headers;
              #...
              #proxy_pass http://backend;
              #...
          }
      }
  }

Restart nginx

systemctl restart nginx

Troubleshooting

Unable to send trace from nginx to traffic collector

Symptoms

No entries in inventory/ inventory not getting updated

Following or similar error seen in nginx log

[Error] File: /tmp/build/opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "DNS resolution failed for ...

Cause

Nginx is unable to resolve traffic-collector address

Solution

If Address of the traffic collector given in nginx.conf is incorrect, locate the http block in your nginx configuration file, find the otel_exporter {...} block, update IPAddressOfTrafficCollector as shown below

otel_exporter {
  endpoint  <IPAddressOfTrafficCollector>:4317
  ...
}

IPAddressOfTrafficCollector should be the your public ip of the collector server instance

Restart nginx by running nginx -s reload

Unable to see entries in inventory

Symptoms

No entries in inventory/ inventory not getting updated

No error in nginx/traffic-collector log

Cause

Unregistered hostname

Solution

Double check if the hostname is registered under Scope URI for Report in Target setup page

FAQ (Frequently Asked Questions)

Can I see what trace are sent from my environment?

Yes, one can see the traces sent by traffic-collector by inspecting docker logs. Run docker logs astra-traffic-collector

How to get IP Address Of Traffic Collector in Non Kubernetes environment?

As container network is the same network as host network, the IP of the container would be the same as the Virtual Machine IP.

Updated on: 11/11/2024