How to set up Astra Traffic Monitoring with Nginx in Cloud VM

Here's how to configure NGINX with OpenTelemetry (OTEL) for traffic monitoring, in cloud VM environments such as AWS, GCP, Azure and Digital Ocean. It also covers troubleshooting common issues encountered during configuration.

Nginx integration

Case 1: Nginx not present in my VM

This section details the steps required to install and configure the Nginx load balancer to successfully instrument incoming HTTP requests.

Pre-requisite:

Set up a VM to host the Nginx load balancer.
Install Docker and Docker-Compose on the VM.
Prepare SSL certificates for the Nginx server. If you don’t have them, generate a self-signed SSL certificate.

Steps:

SSH into the VM and navigate to your working directory.

mkdir /astra-obs
cd /astra-obs

Create a bare minimum nginx.conf file with following content.

user nginx;

worker_processes auto;

pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;

events {
      worker_connections 1024;
      # multi_accept on;
}

http {
      # Basic Settings
      sendfile on;
      tcp_nopush on;
      types_hash_max_size 2048;
      # server_tokens off;
      # server_names_hash_bucket_size 64;
      # server_name_in_redirect off;

      include /etc/nginx/mime.types;
      default_type application/octet-stream;

      # SSL Settings
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
      ssl_prefer_server_ciphers on;

      # Logging Settings
      access_log /var/log/nginx/access.log;

      # Gzip Settings
      gzip on;
      # gzip_vary on;
      # gzip_proxied any;
      # gzip_comp_level 6;
      # gzip_buffers 16 8k;
      # gzip_http_version 1.1;
      # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

      # Virtual Host Configs
      include /etc/nginx/conf.d/*.conf;
      include /etc/nginx/sites-enabled/*;
          
      server {
          #log_subrequest on;
      }
}

Edit the nginx.conf created in previous step to serve the incoming requests.

Configure nginx server
Certificates will be mounted later while creating docker-compose.yaml. Set ssl_certificate /etc/nginx/server.crt and ssl_certificate_key /etc/nginx/server.key in nginx.conf.
Configure reverse proxy - https
Configure reverse proxy - http

Edit the nginx.conf updated in the previous step with the following content to instrument the incoming requests.
Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where Traffic collector is hosted.

load_module modules/ngx_http_js_module.so;

  load_module modules/ngx_otel_module.so;

  #user  nginx;
  #...
  http {
      #include       /etc/nginx/mime.types;
      #default_type  application/octet-stream;
      #...
      js_import ga_nginx_script.js;
      js_set $req_headers ga_nginx_script.req_headers_json;
      js_set $resp_headers ga_nginx_script.resp_headers_json;
      otel_exporter {
          endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint
          interval    5s;
          batch_size  512;
          batch_count 4;
      }
      #...
      server {
          #...

          location / {
              #...
              otel_trace on;
              otel_trace_context propagate;
              otel_span_attr sensor.version $nginx_version;
              otel_span_attr sensor.id <your_sensor_id>;
              otel_span_attr url.query $query_string;
              otel_span_attr http.request.body $request_body;
              otel_span_attr http.user_agent $http_user_agent;
              otel_span_attr http.referer $http_referer;
              otel_span_attr http.request.headers $req_headers;
              otel_span_attr http.response.headers $resp_headers;
              #...
              #proxy_pass http://backend;
              #...
          }
      }
  }

Create ga_nginx_script.js file with following content.

function req_headers_json(r) {

     return JSON.stringify(r.headersIn);
 }

 function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
 }

 export default { req_headers_json, resp_headers_json };

Create docker-compose.yaml file with following content.
Replace /path/to/server.crt and /path/to/server.key with the absoulte path to your certificate and key.

version: '3.7'

services:

 otel-collector:
   image: nginx:otel
   volumes:
     - /path/to/server.crt:/etc/nginx/server.crt
     - /path/to/server.key:/etc/nginx/server.key
     - /astra-obs/nginx.conf:/etc/nginx/nginx.conf
     - /astra-obs/ga_nginx_script.js:/etc/nginx/ga_nginx_script.js
   network_mode: host
   restart: always

Start the nginx container by executing docker-compose up -d from working directory.

To stop the nginx container executing docker-compose down from working directory.

Case 2: Nginx already present in my VM

Register nginx repository in your VM

Install otel module: sudo apt install -y nginx-module-otel
Install js module: sudo apt install -y nginx-module-njs

Create ga_nginx_script.js file with following content under /etc/nginx/

function req_headers_json(r) {

     return JSON.stringify(r.headersIn);

 }

 function resp_headers_json(r) {
     return JSON.stringify(r.headersOut);
 }

 export default { req_headers_json, resp_headers_json };

Edit the nginx.conf file with following content to instrument the incoming requests.
Replace <your_sensor_id> with the integration ID displayed for your nginx integration in Integrations page in UI.
Replace <IPAddressOfTrafficCollector> with the public IP Address of the virtual machine where Traffic collector is hosted.

load_module modules/ngx_http_js_module.so;

  load_module modules/ngx_otel_module.so;

  user  nginx;
  #...
  http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;
      #...
      js_import ga_nginx_script.js;
      js_set $req_headers ga_nginx_script.req_headers_json;
      js_set $resp_headers ga_nginx_script.resp_headers_json;
      otel_exporter {
          endpoint  <IPAddressOfTrafficCollector>:4317; # OTEL collector's gRPC endpoint
          interval    5s;
          batch_size  512;
          batch_count 4;
      }
      #...
      server {
          #...

          location / {
              #...
              otel_trace on;
              otel_trace_context propagate;
              otel_span_attr sensor.version $nginx_version;
              otel_span_attr sensor.id <your_sensor_id>;
              otel_span_attr url.query $query_string;
              otel_span_attr http.request.body $request_body;
              otel_span_attr http.user_agent $http_user_agent;
              otel_span_attr http.referer $http_referer;
              otel_span_attr http.request.headers $req_headers;
              otel_span_attr http.response.headers $resp_headers;
              #...
              #proxy_pass http://backend;
              #...
          }
      }
  }

Restart nginx

systemctl restart nginx

Traffic Collector integration

Fresh Installation

Spin up a new instance of traffic collector by chosing the right image from marketplace.

SSH into the VM.

Locate the following two editable files under /opt/astra-traffic-collector/
.env
config_custom.yaml

Update the secrets by editing .env file:
Replace <collectorId> with the integrationId displayed for your traffic collector integration in Integrations page in UI.
Replace <clientId> with the clientId displayed during the creation of traffic collector integration.
Replace <clientSecret> with the clientSecret displayed during the creation of traffic collector integration.

COLLECTOR_ID=<collectorId>
CLIENT_ID=<clientId>
CLIENT_SECRET=<clientSecret>
TOKEN_URL=https://kc-admin.getastra.dev/realms/astra_api_scanner/protocol/openid-connect/token

To customize the traffic collector edit the config_custom.yaml.

Please refer to this article to know more about customizing the traffic collector configuration.

Upgrade

Docker container upgrade

This process updates the running container to a new version of the collector while retaining any customization you've made via configuration files, such as customconfig.yaml_.

Change directory where docker-compose.yaml is present: cd /opt/astra-traffic-collector/

Run docker-compose down on the docker-compose.yaml to stop the container

Edit the image version field in services.otel-sensor.image of docker-compose.yaml to the latest image version and run the docker-compose up command to spin up the new container.

Image upgrade

Refers to upgrading the virtual machine image used for the Traffic Collector. This process typically involves creating a new VM image with updated software. Customizations made on the existing VM will be lost as the VM is replaced with a new image.

You can use GUI and also use cli tools like awscli, gcloud, az to pull a new VM using the Image from the cloud provider's marketplace.

To upgrade the image using AWS CLI:

aws ec2 describe-images --filters "Name=name,Values=YourImageName"
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0
aws ec2 run-instances --image-id ami-0123456789abcdef0

To upgrade the image using Google Cloud CLI:

gcloud compute instances stop INSTANCE_NAME
gcloud compute instances delete INSTANCE_NAME
gcloud compute instances create INSTANCE_NAME --image-family=IMAGE_FAMILY --image-project=IMAGE_PROJECT

To upgrade the image using Azure CLI:

az vm deallocate --resource-group myResourceGroup --name myVM
az vm delete --resource-group myResourceGroup --name myVM
az vm create --resource-group myResourceGroup --name myVM --image IMAGE_URN

To upgrade the image using DigitalOcean CLI:

doctl compute droplet-action shutdown DROPLET_ID
doctl compute droplet delete DROPLET_ID
doctl compute droplet create --image IMAGE_SLUG --size droplet_size --region region_slug --ssh-keys key_id

Troubleshooting

Unable to send trace from nginx to traffic collector

Symptoms

No entries in inventory/ inventory not getting updated

Following or similar error seen in nginx log

[Error] File: /tmp/build/opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "DNS resolution failed for ...

Cause

Nginx is unable to resolve traffic-collector address

Solution

Non kubernetes based env

If address of the traffic collector given in nginx.conf is incorrect, Locate the http block in your NGINX configuration file, find the otel_exporter {} block, update the endpoint with your FQDN or Public IP and PORT, then restart NGINX

How to get right Address? The Address should be the your public ip of the collector server instance. Address could also be a configured Fully Qualified Domain Name (FQDN)

Kubernetes based env

Address of the traffic collector given in the configmap ga-obs-config is incorrect

Assign following values in ga-obs-config configmap

host = "astra-traffic-collector.obs-client.svc.cluster.local"
              port = 4317

Unable to send traces from traffic collector to ga collector


Symptoms

No entries in inventory/ inventory not getting updated

Following error is seen in astra-traffic-collector container log

error	exporterhelper/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get security token from token endpoint (endpoint \"https://kc-admin.getastra.dev/realms/astra_api_scanner/protocol/openid-connect/token\"); oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\"", "dropped_items": 1}

Cause

Authenication fails with IAM server

Solution

Non kubernetes based env

Edit /opt/astra-traffic-collector/.env and update it with right credentials.

Run start traffic-collector service

Kubernetes based env

update the values.yaml with right crdentials and then run helm upgrade:

helm upgrade --install traffic-collector getastra/traffic-collector-chart --namespace obs-client --debug --values values.yaml

Unable to see entries in inventory

Symptoms

No entries in inventory/ inventory not getting updated

No error in nginx/traffic-collector log

Cause

Unregistered hostname

Solution

Double check is the hostname is registered under "*target to be scanned**"

Add the hostname under extra hosts to be scanned if it's not registered in the first place

FAQ (Frequently Asked Questions)

Can I see what trace are sent from my environment?

Yes, one can see the traces sent by traffic-collector. For non kuberenetes based deployment, run docker logs astra-traffic-collector. For kubernetes based deployment, run kubectl logs astra-traffic-collector-0 -n obs-client

How to get IP Address Of Traffic Collector in Non Kubernetes environment?

As container network is the same network as host network, the IP of the container would be the same as the Virtual Machine IP.

Updated on: 03/10/2024