Integrating monoscope with Kubernetes

1. Add the Helm Repository

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

2. Create a Secret for Your API Key

kubectl create secret generic monoscope-secrets \
  --from-literal=api-key=YOUR_API_KEY

3. Agent Collector (DaemonSet) — Logs and Node Metrics

Create values-agent.yaml:

mode: daemonset

image:
  # Pin a specific tag so chart upgrades don't silently roll a new collector.
  # Current as of April 2026 — bump after reviewing the release notes:
  # https://github.com/open-telemetry/opentelemetry-collector-releases/releases
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
  tag: "0.149.0"

# Pod-level limits — required so the chart's GOMEMLIMIT helper activates.
# Without limits.memory, GOMEMLIMIT is silently disabled and the Go runtime
# can spike past memory_limiter under burst load. Tune for your workload.
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    memory: 512Mi

presets:
  logsCollection:
    enabled: true          # filelog — reads /var/log/pods on each node
  kubeletMetrics:
    enabled: true          # kubeletstats — local kubelet
  kubernetesAttributes:
    enabled: true          # Provides RBAC + a rich metadata extractor (workload kinds,
                           # container.image.*, k8s.cluster.uid, service.* via OTel
                           # annotations, per-node filtering on agents). Pipelines
                           # reference it as `k8sattributes` to match the chart's
                           # internal alias — that alias is deprecated upstream and
                           # logs a harmless warning until the chart maintainers
                           # rename it.
  # clusterMetrics and kubernetesEvents intentionally OFF here —
  # they belong on the cluster collector (one replica only).

extraEnvs:
  - name: MONOSCOPE_API_KEY
    valueFrom:
      secretKeyRef:
        name: monoscope-secrets
        key: api-key

config:
  # health_check must be declared explicitly: when you override config:, the
  # chart's default service.extensions block is replaced, and the readiness
  # probe (port 13133) starts failing.
  extensions:
    health_check:
      endpoint: 0.0.0.0:13133

  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    # filelog is configured by the logsCollection preset, which already
    # adds the `container` operator to strip the CRI envelope. Add a
    # `recombine` operator after it so app multi-line events (Java stack
    # traces, Python tracebacks, pretty-printed JSON) become one record
    # instead of N separate per-line records. Receiver-level `multiline`
    # is the wrong layer here — it sees CRI-wrapped lines, every one of
    # which begins with a timestamp, so it never merges. Recombine acts
    # on the post-container body where indentation actually correlates
    # with continuations.
    filelog:
      operators:
        - id: container-parser
          type: container
          max_log_size: 102400
        - id: recombine-multiline
          type: recombine
          combine_field: body
          combine_with: "\n"
          # New entry = first character is non-whitespace; indented lines
          # are continuations. Tighten to a per-language regex if your
          # apps log indented content that should NOT be merged.
          is_first_entry: 'body matches "^\\S"'
          # Keep streams from different containers from getting mixed.
          source_identifier: attributes["log.file.path"]

  processors:
    batch: {}
    memory_limiter:
      check_interval: 1s
      limit_mib: 4000
      spike_limit_mib: 800
    resource:
      attributes:
        - key: x-api-key
          value: ${env:MONOSCOPE_API_KEY}
          action: upsert

  exporters:
    # tls.insecure is safe here: the monoscope ingest endpoint terminates TLS at
    # its load balancer; the collector connects over a plaintext-tolerant L4 path.
    otlp_grpc:
      endpoint: "otelcol.monoscope.tech:4317"
      tls:
        insecure: true

  # IMPORTANT: when you override service.pipelines, you must list every receiver
  # and processor you want active — including the ones the presets contribute
  # (filelog, kubeletstats, k8sattributes). The chart merges receivers/processors
  # blocks but does NOT merge pipeline arrays. Same applies to service.extensions:
  # it's replaced wholesale, so health_check must be re-declared here.
  service:
    extensions: [health_check]
    pipelines:
      traces:
        receivers: [otlp]
        processors: [k8sattributes, memory_limiter, batch, resource]
        exporters: [otlp_grpc]
      metrics:
        receivers: [otlp, kubeletstats]
        processors: [k8sattributes, memory_limiter, batch, resource]
        exporters: [otlp_grpc]
      logs:
        receivers: [otlp, filelog]
        processors: [k8sattributes, memory_limiter, batch, resource]
        exporters: [otlp_grpc]

Install it:

helm install monoscope-agent open-telemetry/opentelemetry-collector \
  --values values-agent.yaml

4. Cluster Collector (Deployment) — Events and Cluster Metrics

Create values-cluster.yaml:

mode: deployment
# Must stay 1 — k8s_events and k8s_cluster don't leader-elect, so >1 replica
# produces duplicate events and cluster metrics.
replicaCount: 1

image:
  repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s
  tag: "0.149.0"           # see release notes before bumping

# See agent values for the GOMEMLIMIT linkage. Cluster collector handles less
# data, so smaller limits are usually fine.
resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    memory: 256Mi

presets:
  clusterMetrics:
    enabled: true          # k8s_cluster — cluster-wide metrics from API server
  kubernetesEvents:
    enabled: true          # k8s_events — emits events as logs
  kubernetesAttributes:
    enabled: true          # See agent values for the rationale.

extraEnvs:
  - name: MONOSCOPE_API_KEY
    valueFrom:
      secretKeyRef:
        name: monoscope-secrets
        key: api-key

config:
  # See agent values for why health_check must be re-declared here.
  extensions:
    health_check:
      endpoint: 0.0.0.0:13133

  # Receivers are declared explicitly so the pipelines below are unambiguous.
  # The presets generate matching RBAC; the receiver blocks are merged with
  # whatever the presets inject.
  receivers:
    k8s_cluster:
      collection_interval: 10s
    k8s_events:
      namespaces: []

  processors:
    batch: {}
    memory_limiter:
      check_interval: 1s
      limit_mib: 1000
      spike_limit_mib: 200
    resource:
      attributes:
        - key: x-api-key
          value: ${env:MONOSCOPE_API_KEY}
          action: upsert

  exporters:
    otlp_grpc:
      endpoint: "otelcol.monoscope.tech:4317"
      tls:
        insecure: true     # see note in agent values

  service:
    extensions: [health_check]
    pipelines:
      metrics:
        receivers: [k8s_cluster]
        processors: [k8sattributes, memory_limiter, batch, resource]
        exporters: [otlp_grpc]
      logs:
        receivers: [k8s_events]
        processors: [k8sattributes, memory_limiter, batch, resource]
        exporters: [otlp_grpc]

Install it:

helm install monoscope-cluster open-telemetry/opentelemetry-collector \
  --values values-cluster.yaml

5. Verify

kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector
# Note the `-agent` suffix — the chart appends it to the workload name in
# DaemonSet mode, so this is NOT just `monoscope-agent-opentelemetry-collector`.
kubectl logs daemonset/monoscope-agent-opentelemetry-collector-agent
kubectl logs deployment/monoscope-cluster-opentelemetry-collector

You should see the agent running on every node and a single cluster collector pod. Telemetry will start flowing to your monoscope dashboard shortly after the pods become Ready.

Pods being healthy is necessary but not sufficient — auth failures and dropped exports are silent in the collector logs. To confirm telemetry is actually reaching monoscope, run:

monoscope events search 'resource.k8s.cluster.uid != ""' --since 5m --limit 1
# Or in the dashboard: filter recent events by resource.k8s.namespace.name.

Install the Operator

The OTel Operator’s admission webhook requires cert-manager (or another certificate provider). Skip the first command if you already have cert-manager — or any compatible cert source — installed. For production, replace latest with a pinned release tag from each project.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Create the same API key secret used by the Helm path (skip if already created):

kubectl create secret generic monoscope-secrets \
  --from-literal=api-key=YOUR_API_KEY

Agent CR (DaemonSet)

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: monoscope-agent
spec:
  mode: daemonset
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods
  env:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: MONOSCOPE_API_KEY
      valueFrom:
        secretKeyRef:
          name: monoscope-secrets
          key: api-key
  config:
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
    receivers:
      filelog:
        # Path layout: /var/log/pods/<ns>_<pod>_<uid>/<container>/<restart>.log
        include: [/var/log/pods/*/*/*.log]
        start_at: end          # use 'beginning' only for first-install backfill
        operators:
          - id: container-parser
            type: container
            max_log_size: 102400
          # Merge app multi-line events (stack traces, pretty-printed JSON)
          # after the CRI envelope is stripped. See the Helm path above for
          # why this happens here and not at the receiver-level multiline.
          - id: recombine-multiline
            type: recombine
            combine_field: body
            combine_with: "\n"
            is_first_entry: 'body matches "^\\S"'
            source_identifier: attributes["log.file.path"]
      kubeletstats:
        collection_interval: 10s
        auth_type: serviceAccount
        endpoint: ${env:K8S_NODE_NAME}:10250
        insecure_skip_verify: true   # acceptable for dev; in production mount the kubelet CA
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch: {}
      memory_limiter:
        check_interval: 1s
        limit_mib: 4000
        spike_limit_mib: 800
      k8s_attributes:
        auth_type: serviceAccount
        passthrough: false
        filter:
          node_from_env_var: K8S_NODE_NAME
      resource:
        attributes:
          - key: x-api-key
            value: ${env:MONOSCOPE_API_KEY}
            action: upsert
    exporters:
      otlp_grpc:
        endpoint: "otelcol.monoscope.tech:4317"
        tls:
          insecure: true
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8s_attributes, memory_limiter, batch, resource]
          exporters: [otlp_grpc]
        metrics:
          receivers: [otlp, kubeletstats]
          processors: [k8s_attributes, memory_limiter, batch, resource]
          exporters: [otlp_grpc]
        logs:
          receivers: [filelog, otlp]
          processors: [k8s_attributes, memory_limiter, batch, resource]
          exporters: [otlp_grpc]

Cluster CR (Deployment, 1 replica)

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: monoscope-cluster
spec:
  mode: deployment
  replicas: 1
  env:
    - name: MONOSCOPE_API_KEY
      valueFrom:
        secretKeyRef:
          name: monoscope-secrets
          key: api-key
  config:
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
    receivers:
      k8s_cluster:
        collection_interval: 10s
      k8s_events:
        namespaces: []
    processors:
      batch: {}
      memory_limiter:
        check_interval: 1s
        limit_mib: 1000
        spike_limit_mib: 200
      k8s_attributes:
        auth_type: serviceAccount
        passthrough: false
      resource:
        attributes:
          - key: x-api-key
            value: ${env:MONOSCOPE_API_KEY}
            action: upsert
    exporters:
      otlp_grpc:
        endpoint: "otelcol.monoscope.tech:4317"
        tls:
          insecure: true
    service:
      extensions: [health_check]
      pipelines:
        metrics:
          receivers: [k8s_cluster]
          processors: [k8s_attributes, memory_limiter, batch, resource]
          exporters: [otlp_grpc]
        logs:
          receivers: [k8s_events]
          processors: [k8s_attributes, memory_limiter, batch, resource]
          exporters: [otlp_grpc]

RBAC

Both CRs need read access to pods, namespaces, nodes, events, and the workload APIs. The OTel Operator creates a ServiceAccount named <cr-name>-collector in the CR’s namespace, so the binding below targets monoscope-agent-collector and monoscope-cluster-collector. Apply this once:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monoscope-collector
rules:
- apiGroups: [""]
  resources:
    - pods
    - namespaces
    - nodes
    - nodes/stats
    - nodes/proxy
    - services
    - events
    - replicationcontrollers
    - resourcequotas
    - persistentvolumes
    - persistentvolumeclaims
  verbs: [get, list, watch]
- apiGroups: ["apps"]
  resources: [deployments, replicasets, daemonsets, statefulsets]
  verbs: [get, list, watch]
- apiGroups: ["batch"]
  resources: [jobs, cronjobs]
  verbs: [get, list, watch]
- apiGroups: ["autoscaling"]
  resources: [horizontalpodautoscalers]
  verbs: [get, list, watch]
- apiGroups: ["events.k8s.io"]
  resources: [events]
  verbs: [get, list, watch]
- apiGroups: ["discovery.k8s.io"]
  resources: [endpointslices]
  verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: monoscope-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monoscope-collector
subjects:
- kind: ServiceAccount
  name: monoscope-agent-collector
  namespace: default
- kind: ServiceAccount
  name: monoscope-cluster-collector
  namespace: default