Overview

This guide explains how to

• Install ingress-nginx (if not already installed)

• Instrument ingress-nginx to send telemetry via OpenTelemetry

• Troubleshoot common issues

• Verify that traces are reaching Astra Traffic Collector

Illustration: High-level integration flow between Ingress Nginx and Astra Traffic Collector

Prerequisites

Before beginning, ensure you have:

• Astra Traffic Collector is available and reachable.

  📄 How to setup Astra Traffic Collector in Kubernetes

• Nginx integration is created and sensorID is copied

  📄 How to Create Sensor Integration for API Observability

Quick Installation

Step 1 — Install ingress-nginx

To install ingress-nginx in your kubernetes infrastructure, please refer to these article:

• Install Ingress-nginx

• Configure k8s ingress resource

Step 2 — Verify installation

kubectl get all -n ingress-nginx

Sample output in successful installation scenario

NAME                                                    READY    STATUS    RESTARTS   AGE

    pod/nginx-ingress-ingress-nginx-controller-xxx-xx       1/1      Running   0          1m

    NAME                                                       TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE

    service/nginx-ingress-ingress-nginx-controller             LoadBalancer   10.100.58.201               80:32404/TCP,443:31635/TCP   1m
    service/nginx-ingress-ingress-nginx-controller-admission   ClusterIP      10.100.66.225             443/TCP                      1m

    NAME                                                         READY   UP-TO-DATE   AVAILABLE  AGE
    deployment.apps/nginx-ingress-ingress-nginx-controller       1/1     1            1          1m

    NAME                                                         DESIRED   CURRENT   READY   AGE
    replicaset.apps/nginx-ingress-ingress-nginx-controller-xxx   1         1         1       1m

Step 3 — Create working directory and Backup current Helm values

mkdir -p ~/getastra/ingress_nginx_instrumentation && cd ~/getastra/ingress_nginx_instrumentation

Take the backup of the existing ingress-nginx helm values if they are already customized.

sudo helm get values ingress-nginx -n ingress-nginx -o yaml | sudo tee ingress-nginx-original-values.yaml

Step 4 — Modify nginx.tmpl for response header instrumentation

Ensure that ingress-nginx deployment is upgraded to it's latest version to get latest nginx.tmpl. If the ingress-nginx deployment is not latest, then this might cause problem in later step as the template file may go out of sync.

sudo helm upgrade ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx

• Fetch the nginx.tmpl from ingress-nginx pod by executing:

POD_NAME=$(sudo kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
sudo kubectl cp -n ingress-nginx $POD_NAME:/etc/nginx/template/nginx.tmpl ori_nginx.tmpl

• Keep the original file as backup. Copy the file:

sudo cp ori_nginx.tmpl nginx.tmpl

• Open the nginx.tmpl in text editor and then search for keyword header_filter_by_lua_block. Add the lines as shown below

set $resp_headers '';
header_filter_by_lua_block {
                local cjson = require "cjson"
                ngx.var.resp_headers = cjson.encode(ngx.resp.get_headers())
 }

Step 5 — Create ConfigMap

• Create config map for nginx-template-astra-instrumented

sudo kubectl create configmap nginx-template-astra-instrumented --from-file=nginx.tmpl -n ingress-nginx

• Create file called as ga-obs-config.yaml with following content

apiVersion: v1
kind: ConfigMap
metadata:
     name: ga-obs-config
     namespace: ingress-nginx
data:
     ga_obs_nginx.toml: |
            exporter = "otlp"
            processor = "batch"
            [exporters.otlp]
            host = "astra-traffic-collector.astra-collector.svc.cluster.local"
            port = 4317
            # Optional: enable SSL, for endpoints that support it
            # use_ssl = true
            # Optional: set a filesystem path to a pem file to be used for SSL encryption
            # (when use_ssl = true)
            # ssl_cert_path = "/path/to/cacert.pem"
            [processors.batch]
            max_queue_size = 2048
            schedule_delay_millis = 5000
            max_export_batch_size = 512
            [service]
            # Can also be set by the OTEL_SERVICE_NAME environment variable.
            name = "nginx-proxy" # Opentelemetry resource name
            [sampler]
            name = "AlwaysOn" # Also: AlwaysOff, TraceIdRatioBased
            ratio = 1.0
            parent_based = false

• Create configmap for ga-obs-config.yaml

sudo kubectl apply -f ga-obs-config.yaml

• Verify the configmap created

sudo kubectl get cm -n ingress-nginx

Step 6 — Create values.yaml

Create values.yaml with following content.

controller:

        extraVolumeMounts:
          - mountPath: /etc/nginx/template/nginx.tmpl
            subPath: nginx.tmpl
            name: nginx-template-volume
            readOnly: true
          - mountPath: /etc/nginx/ga-obs
            name: ga-obs
            readOnly: true

        extraVolumes:
          - name: nginx-template-volume
            configMap:
                name: nginx-template-astra-instrumented
                defaultMode: 420
          - name: ga-obs
            configMap:
                name: ga-obs-config
                defaultMode: 420
        
        config:
            allow-snippet-annotations: "true"
            http-snippet: |
                opentelemetry_config /etc/nginx/ga-obs/ga_obs_nginx.toml;
                opentelemetry_ignore_paths /is-dynamic-lb-initialized|/health|/metric;
            location-snippet: |
                opentelemetry_operation_name ga-otel-nginx;
                opentelemetry_attribute sensor.version $nginx_version;

                opentelemetry_attribute sensor.id mySensorID;

                opentelemetry_attribute http.request.body $request_body;
                opentelemetry_attribute net.sock.peer.addr $remote_addr;
                opentelemetry_attribute net.sock.peer.port $remote_port;
                access_by_lua_block {
                    local cjson = require "cjson"
                    ngx.var.req_headers = cjson.encode(ngx.req.get_headers())
                }
                opentelemetry_attribute http.request.headers $req_headers;
                opentelemetry_attribute http.response.headers $resp_headers;
            main-snippet: |
                load_module /etc/nginx/modules/otel_ngx_module.so;
            server-snippet: |
                set $req_headers '';

Step 7 — Update Helm values

Preview: Verify the changes (using diff). Following commands installs the diff and then performs a diff command to show what are being changed

sudo helm plugin install https://github.com/databus23/helm-diff
sudo helm diff upgrade ingress-nginx -n ingress-nginx ingress-nginx/ingress-nginx -f values.yaml --debug 

Apply the changes

sudo helm upgrade ingress-nginx -n ingress-nginx ingress-nginx/ingress-nginx -f values.yaml --debug

Troubleshooting

Unable to send trace from nginx to traffic collector

Symptoms

• No entries in inventory OR inventory not getting updated

• Following or similar error seen in nginx log - /var/log/etc/nginx.log

[Error] File: /tmp/build/opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:66 [OTLP TRACE GRPC Exporter] Export() failed with status_code: "UNAVAILABLE" error_message: "DNS resolution failed for ...

Cause

• Nginx is unable to resolve traffic-collector address

Solution

• Address of the traffic collector given in the configmap ga-obs-config is incorrect

• Assign following values in ga-obs-config configmap

host = "astra-traffic-collector.{namespace}.svc.cluster.local"
port = 4317

FAQ (Frequently Asked Questions)

1. Can I see what trace are sent from my environment?

   📄 Verifying Traces in Astra Traffic Collector

2. How to get IP Address Of Traffic Collector in Non Kubernetes environment?

   As container network is the same network as host network, the IP of the container would be the same as the Virtual Machine IP.