Here's how to configure Astra Traffic Collector for traffic monitoring, in Kubernetes environments such as EKS, GKE, AKE. It also covers troubleshooting common issues encountered during configuration.

Traffic Collector integration

Quick Installation

1. Create a working directory and switch to it.

mkdir -p /opt/astra-traffic-collector && cd /opt/astra-traffic-collector

2. Create astra-collector namespace for traffic collector installation by running

kubectl create ns astra-collector

3. **Add astra traffic collector helm repository by running **

helm repo add getastra https://raw.githubusercontent.com/getastra/obs-deployments/gh-pages/

• Update the repo

helm repo update

4. Create values.yaml

• Replace with the COLLECTOR_ID displayed during the creation of astra traffic collector integration.

• Replace with the CLIENT_ID displayed during the creation of astra traffic collector integration.

• Replace with the CLIENT_SECRET displayed during the creation of astra traffic collector integration.

secret:
      name: astra-collector-secrets
      collectorId: 
      clientId: 
      clientSecret: 
      tokenUrl: https://auth.getastra.com/realms/astra_api_scanner/protocol/openid-connect/token
      remoteAddrIdentifierHeader: x-forwarded-for

volumes:
  - configMap:
      defaultMode: 444
      name: astra-collector-custom-config
    name: custom-config
  
volumeMounts:
  - name: collector-message
    mountPath: /var/lib/otelcol/file_storage
  - name: custom-config
    mountPath: /etc/otelcol-contrib/config_custom.yaml
    subPath: config_custom.yaml

5. Create config_custom.yaml file as shown below.

processors:
  
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
  filter/custom:
    error_mode: ignore
    traces:
      span:
        ## allowing traces based on hostname regex pattern. Following will drop all traces originated from host other than: localhost*
        # - IsMatch(attributes["url.host"], "localhost*") == false
        ## excluding traces based on hostname regex pattern. Following will drop all traces originated from host: localhost*
        # - IsMatch(attributes["url.host"], "localhost*")
        ## excluding traces based on template regex pattern. Following will drop all traces having url_template: _wdt*
        # - IsMatch(attributes["url.template"], "_wdt*")
        ## exclude traces with method set to OPTIONS. Comment below line to allow the traces with http method OPTIONS
        - ConvertCase(attributes["http.method"], "upper") == "OPTIONS"
  
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
  transform/custom:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          ## Templatize url path regex pattern by keyword: "slug". Following will templatize the url Path: /api/v1/chinchikrqwertyuiop/ -> /api/v1/{{slug}}/
          - replace_pattern(attributes["url.template"], "chinchikrqwertyuiop", "{{slug}}")
          ## Redact MasterCard credit card number 
          #- replace_all_patterns(attributes, "value", "^5[1-5][0-9]{14}$", "{{card}}")

6. Create configmap containing content of config_custom.yaml file

kubectl create configmap astra-collector-custom-config --from-file=./config_custom.yaml -n astra-collector

7. **Install the helm chart by running: **

helm upgrade --install traffic-collector getastra/traffic-collector-chart --namespace astra-collector --debug --values values.yaml

8. That's it!. You should now see the astra-traffic-collector pod running under astra-collector namespace. Create "sensor" integrations from here and integrate it with astra-traffic-collector to seamlessly monitor traffic.

Customizing the behavior of astra-traffic-collector

1. Edit /opt/astra-traffic-collector/config_custom.yaml. Refer here to know how to filter, redact and templatize

2. Update configmap containing content of config_custom.yaml file

kubectl create configmap astra-collector-custom-config --from-file=/opt/astra-traffic-collector/config_custom.yaml -n astra-collector --dry-run=client -o yaml | kubectl apply -f -
kubectl delete po astra-traffic-collector -n astra-collector

Upgrade

1. Upgrading traffic collector helm chart to latest version:

• Update the helm repo

helm repo update

• Upgrade the chart to latest version

helm upgrade --install traffic-collector getastra/traffic-collector-chart --namespace astra-collector --debug --values values.yaml

Troubleshooting

1. Unable to send traces from traffic collector to ga-collector

Symptoms

• No entries in inventory/ inventory not getting updated

• Following error is seen in astra-traffic-collector container log

error	exporterhelper/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: failed to get security token from token endpoint (endpoint \"https://auth.getastra.com/realms/astra_api_scanner/protocol/openid-connect/token\"); oauth2: \"unauthorized_client\" \"Invalid client or Invalid client credentials\"", "dropped_items": 1}

Cause

• Authenication fails with IAM server

  Solution

• update the values.yaml with right credentials and then run helm upgrade:

helm upgrade --install traffic-collector getastra/traffic-collector-chart --namespace astra-collector --debug --values values.yaml

2. Unable to see entries in inventory

Symptoms

• No entries in inventory/ inventory not getting updated

• No error in nginx/traffic-collector log

  Cause

• Unregistered hostname

  Solution

• Double check if the hostname is registered under Scope URI for Report in Target setup page

• Add the hostname under extra hosts to be scanned if it's not registered in the first place

FAQ (Frequently Asked Questions)

1. Can I see what trace are sent from my environment?

Yes, one can see the traces sent by astra-traffic-collector by inspecting logs. Run kubectl logs astra-traffic-collector-0 -n astra-collector to see the logs.

2. How to regenerate client secret for astra-traffic-collector integration?

• Go to integrations.

• Click on "vertical three dots" of the astra-traffic-collector integration for which client secrets should be regenerated

• Click on "Regenerate client secret"

• Copy down the secrets shown

• Update the values.yaml with new secrets. After updating your values.yaml should look something like this:

secret:
  name: astra-collector-secrets
  collectorId: 
  clientId: 
  clientSecret: 
  tokenUrl: https://auth.getastra.com/realms/astra_api_scanner/protocol/openid-connect/token
  remoteAddrIdentifierHeader: x-forwarded-for

#any other changes here

• Run the helm upgrade command with latest valuues.yaml which has updated client secret

helm upgrade --install traffic-collector getastra/traffic-collector-chart --namespace astra-collector --debug --values values.yaml