Kong Instrumentation with OpenTelemetry Plugin

Overview

Kong Instrumentation enhances API monitoring by capturing detailed HTTP traffic data through OpenTelemetry. It provides granular visibility into requests and responses while maintaining performance through selective data capture and size limits.

Architecture

Prerequisites

Kong Gateway installed and running
OpenTelemetry Collector configured
Access to Kong configuration (Kong Manager)
A service and route pre-configured to setup tracing

Ensure Kong Manager GUI is running and accessible (default port: 8002) If Kong Manager is not accessible:

# Update your Kong container configuration to expose port 8002
  docker run -d --name kong \
    -e "KONG_DATABASE=postgres" \
    -e "KONG_PG_HOST=kong-database" \
    -p 8000:8000 \
    -p 8443:8443 \
    -p 8001:8001 \
    -p 8002:8002 \  # Add this line if missing
    kong:latest

Quick Installation

Set Required Environment Variable

# First, access your Kong container
docker exec -it kong bash
   
# Then set the environment variable for HTTP instrumentation
export KONG_TRACING_INSTRUMENTATIONS=http_client

Access Kong Manager Interface
- Open your web browser
- Navigate to http://<kong-ip>:8002
- From the main dashboard, click on "Plugins" in the left sidebar
- Click the "New Plugin" button
- In the plugin catalog, search for and select "OpenTelemetry"

Choose between the two Plugin Scopes

Global: Plugin will apply to all services, routes, and consumers
- Recommended for system-wide tracing

Scoped: Plugin will apply to specific targets
- Select specific Gateway Services
- Select specific Routes
- Select specific Consumers
- Useful for granular control of tracing

Configure OpenTelemetry Settings
Required Endpoints (at least one must be configured):
- Traces Endpoint: Enter your OpenTelemetry collector endpoint
- Example: https://astra-traffic-collector-ip-address:4318/v1/traces
- Logs Endpoint (optional): Enter if you want to collect logs

Protocol Configuration:
- Select all applicable protocols: grpc, grpcs, http, https
- These determine which types of requests will be traced

Propagation Settings:
- Set Default Format to "w3c" (recommended standard)
- Header Type: "w3c"
- Configure any custom headers if needed

Resource Attributes:
- sensor.version: Set to your current version to "1.0.0"
- service.name: Name of the service should be set to "astra-otel-plugin"
- sensor.id: Set sensor.id using the value provided at the time of creating the Apigee Integration (e.g., "05927cc2-cf2f-4508-a11e-3001964b9113")

Advanced Settings (adjust based on your needs) :
Batch Span Count: 1
Connect Timeout: 1000 ms
Max Batch Size: 200
Max Entries: 10000
Initial Retry Delay: 0.01
Max Retry Delay: 60
Max Retry Time: 60
Sampling Rate: Set between 0.0 and 1.0
Read/Send Timeouts: 5000 ms

Review and Save
- Click "View Configuration" to review all settings
- Verify all parameters are correctly set
- Click "Save" to apply the configuration
- The plugin will be immediately active based on your scope selection

Verify Configuration
- Check the plugin appears in your plugins list
- Status should show as "Enabled"
- Make a test request through your Kong gateway
- Verify traces are appearing in your OpenTelemetry collector

Troubleshooting
- Verify your collector endpoint is accessible from Kong
- Check Kong's error logs for connection issues
- Ensure sampling rate is greater than 0
- Verify protocols match your service configuration

After setting up the OpenTelemetry plugin, you need to configure the Kong Functions plugin to properly instrument the tracing:

Create Kong Functions Plugin
- Return to Kong Manager ( http://<kong-ip>:8002)
- Navigate to "Plugins"
- Click "New Plugin"
- Search and select "Kong Functions"
- Choose the same scope as your OpenTelemetry plugin (Global or Scoped)

Configure Access Phase Function
- Find the "Access" field and paste the following code

-- Access phase handler
   local root_span = kong.tracing.active_span()
   if not root_span then
       kong.log.err("No active span found")
       return
   end

   -- Get direct client IP (load balancer or actual client)
   local peer_addr = kong.client.get_ip()
   kong.log.debug("Direct client IP: ", peer_addr)

   if peer_addr then
       root_span:set_attribute("net.sock.peer.addr", peer_addr)
   end

   -- Add http.target
   root_span:set_attribute("http.target", kong.request.get_path_with_query())

   local function escape_json_value(v)
       if type(v) == "string" then
           return (v:gsub('"', '\\"'))
       end
       return v
   end

   local headers = kong.request.get_headers()
   if headers then
       local headers_str = "{"
       local first = true
       for k, v in pairs(headers) do
           if not first then
               headers_str = headers_str .. ", "
           end
           
           headers_str = headers_str .. '"' .. k .. '": '
           
           if type(v) == "table" then
               headers_str = headers_str .. '"' .. escape_json_value(table.concat(v, ",")) .. '"'
           elseif type(v) == "number" then
               headers_str = headers_str .. tostring(v)
           else
               headers_str = headers_str .. '"' .. escape_json_value(tostring(v)) .. '"'
           end
           
           first = false
       end
       headers_str = headers_str .. "}"
       root_span:set_attribute("http.request.headers", headers_str)
   end

   -- Add request body in access phase
   local ok, body = pcall(kong.request.get_raw_body)
   if ok and body then
       root_span:set_attribute("http.request.body", body)
   end

Configure Body Filter Function
- Find the "Body Filter" field and paste the following code:

-- Constants
   local MAX_BODY_SIZE = 1024 * 1024  -- 1MB in bytes

   -- Body filter phase handler
   local root_span = kong.tracing.active_span()
   if not root_span then
       return
   end

   -- Get chunk and eof flag safely
   local chunk, eof
   if ngx.arg[1] ~= nil then
       chunk = ngx.arg[1]
   end
   if ngx.arg[2] ~= nil then
       eof = ngx.arg[2]
   end

   -- Initialize response_chunks and total size if needed
   kong.ctx.shared.response_chunks = kong.ctx.shared.response_chunks or {}
   kong.ctx.shared.response_size = kong.ctx.shared.response_size or 0

   -- Track size and collect chunks for tracing only
   if chunk and type(chunk) == "string" then
       kong.ctx.shared.response_size = kong.ctx.shared.response_size + #chunk
       if kong.ctx.shared.response_size <= MAX_BODY_SIZE then
           table.insert(kong.ctx.shared.response_chunks, chunk)
       end
   end

   -- Always pass through the original chunk unmodified
   ngx.arg[1] = chunk

   -- On EOF, set the complete body for tracing
   if eof then
       local body = ""
       if kong.ctx.shared.response_size <= MAX_BODY_SIZE then
           body = table.concat(kong.ctx.shared.response_chunks)
       end
       root_span:set_attribute("http.response.body", body)
       kong.ctx.shared.response_chunks = nil
       kong.ctx.shared.response_size = nil
   end

Configure Header Filter Function
- Find the "Header Filter" field and paste the following code:

-- Header filter phase handler
   local root_span = kong.tracing.active_span()
   if not root_span then
       return
   end

   -- Function to escape JSON string values
   local function escape_json_value(v)
       if type(v) == "string" then
           return (v:gsub('"', '\\"'))
       end
       return v
   end

   -- Add response headers in JSON-like string format
   local headers = kong.response.get_headers()
   if headers then
       local headers_str = "{"
       local first = true
       for k, v in pairs(headers) do
           if not first then
               headers_str = headers_str .. ", "
           end
           
           headers_str = headers_str .. '"' .. escape_json_value(k) .. '": '
           
           if type(v) == "table" then
               headers_str = headers_str .. '"' .. escape_json_value(table.concat(v, ",")) .. '"'
           elseif type(v) == "number" then
               headers_str = headers_str .. tostring(v)
           else
               headers_str = headers_str .. '"' .. escape_json_value(tostring(v)) .. '"'
           end
           
           first = false
       end
       headers_str = headers_str .. "}"
       root_span:set_attribute("http.response.headers", headers_str)
   end

Save Configuration
- Review all three function configurations
- Click "Save" to apply the Kong Functions plugin

Updated on: 03/03/2025