Kubernetes Observability Stack 2026 — From Logs to OpenTelemetry

The three pillars of observability — logs, metrics, traces — were a good start. But in 2026, when the average Kubernetes cluster generates hundreds of gigabytes of telemetry data daily, they are no longer enough. You need signal correlation, eBPF-based instrumentation without code, AI-driven alerting, and most importantly: control over the costs that your observability stack generates. This article is a guide to the modern observability stack for Kubernetes — from OpenTelemetry Collector pipeline through the LGTM stack to eBPF and AI alerting.

1. Why Three Pillars Aren’t Enough¶

The concept of “Three Pillars of Observability” — logs, metrics, traces — defined the direction of the entire industry in 2018. But practice has shown that the pillars alone, without mutual correlation, are like three books in three different languages. Each tells the same story, but no one can read them all at once.

The problem of isolated signals: You have an alert on high latency (metric). You open a trace and see that the bottleneck is in a database call. But what was that call doing? You need logs from that specific request. And now try to find the right log among millions of lines — without a trace ID in the log context, you’re lost. This is the reality of 60% of Kubernetes deployments we’ve audited over the past two years.

72% Of organizations have 3+ observability tools without correlation (CNCF Survey 2025)

43 min Average MTTR with manual signal correlation

35% Of observability budget goes to storing data that nobody reads

Modern observability is transitioning from three pillars to unified telemetry. Every signal — log, metric, trace, profile — carries a shared context: trace ID, span ID, resource attributes (cluster, namespace, pod, container). This allows you to click through from a metric alert directly to the relevant trace and from a trace to the logs of a specific span. And this is exactly what OpenTelemetry enables natively.

Additionally, a fourth signal is gaining traction: continuous profiling. Projects like Pyroscope (now part of Grafana Labs) allow you to correlate CPU and memory profiles directly with traces. You know not only where a request is slow, but also why — which function is consuming CPU, where memory allocation spikes occur. In a Kubernetes environment where you share resources among dozens of services, this is essential for capacity planning.

2. OpenTelemetry Collector Pipeline — The Heart of the Entire Stack¶

OpenTelemetry (OTel) became the de facto standard for instrumentation and telemetry collection in 2026. Traces and metrics APIs are stable (GA), the logs API reached stability in Q3 2025. The key component is the OpenTelemetry Collector — a vendor-neutral agent that receives, processes, and exports telemetry data.

Collector Pipeline Architecture¶

`# otel-collector-config.yaml — production configuration

receivers:

otlp:                    # OTLP gRPC + HTTP

protocols:

grpc: { endpoint: “0.0.0.0:4317” }

http: { endpoint: “0.0.0.0:4318” }

prometheus:              # Scrape existing Prometheus targets

config:

scrape_configs:

- job_name: ‘k8s-pods’

kubernetes_sd_configs: [{ role: pod }]

filelog:                 # Container logs from /var/log/pods

include: [“/var/log/pods///*.log”]

processors:

batch:                   # Batching for throughput

send_batch_size: 8192

timeout: 5s

k8sattributes:           # Enrichment: pod name, namespace, labels

extract:

metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name]

tail_sampling:           # Intelligent sampling — not random

policies:

- { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }

- { name: slow, type: latency, latency: { threshold_ms: 2000 } }

- { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 5 } }

filter:                  # Drop health checks and noise

logs:

exclude:

match_type: regexp

bodies: [“^GET /healthz”, “^GET /readyz”]

exporters:

otlphttp/tempo:          # Traces → Grafana Tempo

endpoint: “http://tempo:4318”

prometheusremotewrite:    # Metrics → Grafana Mimir

endpoint: “http://mimir:9009/api/v1/push”

loki:                    # Logs → Grafana Loki

endpoint: “http://loki:3100/loki/api/v1/push”

service:

pipelines:

traces:  { receivers: [otlp], processors: [batch, k8sattributes, tail_sampling], exporters: [otlphttp/tempo] }

metrics: { receivers: [otlp, prometheus], processors: [batch, k8sattributes], exporters: [prometheusremotewrite] }

logs:    { receivers: [otlp, filelog], processors: [batch, k8sattributes, filter], exporters: [loki] }`

Key principles of a production pipeline:

• Receivers accept data from various sources — OTLP (native OTel protocol), Prometheus scrape for backward compatibility, filelog for container logs
• Processors are the brain of the pipeline — k8sattributes automatically enriches telemetry with Kubernetes metadata, tail_sampling intelligently samples traces (100% of errors, 100% of slow requests, 5% baseline)
• Exporters send data to the backend — in the LGTM stack, that is Tempo for traces, Mimir for metrics, Loki for logs
• Filter processor eliminates noise before export — health checks, readiness probes, debug logs in production. This is the main lever for cost control

DaemonSet vs. Sidecar vs. Gateway

On Kubernetes, deploy the Collector as a DaemonSet for logs and node-level metrics, and as a Deployment (gateway) for centralized trace processing. The sidecar pattern (Collector in every pod) adds 50–100 MB RAM overhead per pod — use it only when you need per-pod configuration or strict tenant isolation.

3. LGTM Stack — Loki, Grafana, Tempo, Mimir¶

The LGTM stack from Grafana Labs became the most popular open-source observability platform for Kubernetes in 2026. And for good reason: all components share architectural principles (horizontal scaling, object storage backend, multi-tenancy) and Grafana unifies them into a single UI with native signal correlation.

Loki — Logs Without Content Indexing¶

Loki is “Prometheus for logs.” It does not index log content (unlike Elasticsearch), it only indexes metadata labels. This means significantly lower storage and compute requirements. In practice: where Elasticsearch needs 3 nodes with 64 GB RAM, Loki can manage with a single node and an S3 bucket. The LogQL query language is similar to PromQL, so the learning curve is minimal for teams that already know Prometheus.

Production tip: Deploy Loki in Simple Scalable or microservices mode. Monolithic mode is OK for dev/staging, but in production you want to scale read and write paths independently. Typical production configuration: 3 write replicas, 2 read replicas, S3/GCS as long-term storage.

Tempo — Distributed Traces¶

Tempo stores traces in object storage (S3, GCS, Azure Blob) without requiring indexing. Search works via trace ID (direct lookup) or via TraceQL — a query language that allows filtering traces by attributes, duration, and status. Thanks to integration with Grafana, you can click through from a metric panel directly to relevant traces using Exemplars.

Mimir — Long-term Metrics Storage¶

Mimir is a horizontally scalable storage for Prometheus metrics. Compatible with PromQL, it supports multi-tenancy, global view across clusters, and long-term retention (months to years) on object storage. For most Kubernetes deployments, it replaces Thanos or Cortex with simpler architecture and better performance.

Grafana — Unified UI¶

Grafana 11+ brings a key feature: Explore Traces → Logs → Metrics correlation. From a single dashboard you see metrics, click through to traces, and from traces to logs of a specific span. This eliminates context switching between tools and dramatically reduces MTTR. Grafana Alerting replaces standalone Alertmanager and enables multi-signal alerts — for example, “alert if error rate > 5% AND p99 latency > 2s AND logs contain ‚connection refused’.”

`# Typical LGTM production architecture on Kubernetes

┌──────────────┐    ┌─────────────────────────────────┐

│  Aplikace     │───▶│  OTel Collector (DaemonSet)    │

│  + OTel SDK   │    │  + k8sattributes + sampling    │

└──────────────┘    └──────────┬──────────────────────┘

│

┌──────────┼──────────────┐

▼          ▼              ▼

┌─────────┐ ┌─────────┐   ┌─────────┐

│  Loki    │ │  Tempo   │   │  Mimir   │

│  (logs)  │ │ (traces) │   │(metrics) │

└────┬────┘ └────┬────┘   └────┬────┘

│         │              │

▼         ▼              ▼

┌─────────────────────────────────┐

│          Grafana 11+            │

│  Dashboards + Alerting + Explore │

└─────────────────────────────────┘

│

▼

┌─────────────────────┐

│  Object Storage     │

│  (S3 / GCS / MinIO) │

└─────────────────────┘`

4. eBPF-based Observability — Instrumentation Without Code¶

eBPF (extended Berkeley Packet Filter) is a game-changer for Kubernetes observability. Instead of instrumentation in application code (SDKs, agents, sidecars), eBPF programs run directly in the Linux kernel and capture system calls, network traffic, and performance — without any application modification. For the SRE team, this means: visibility into services you don’t control (third-party, legacy, compiled binaries).

Cilium Hubble — network observability¶

Cilium is today the most widely used eBPF-based CNI (Container Network Interface) for Kubernetes. Its observability layer Hubble provides L3/L4/L7 visibility into all network traffic without sidecar proxy (goodbye Istio sidecar overhead). What Hubble can do:

• Service map — automatically generated dependency map between services, based on actual network flow, not declarative configurations
• DNS observability — you see every DNS query from every pod, including latency and error rate. DNS issues are the #1 hidden killer of Kubernetes performance
• HTTP/gRPC metrics — L7 metrics (request rate, error rate, duration) without service mesh sidecar. Hubble extracts them directly from eBPF hooks in the kernel
• Network policy audit — you see which connections are allowed, which are blocked, and where you’re missing a NetworkPolicy

In production, we combine Hubble metrics with Prometheus/Mimir and Hubble flows with Grafana dashboards. Result: complete network observability with overhead under 2% CPU (compared to 10–15% with the Istio sidecar model).

Pixie — application-level eBPF observability¶

Pixie (open-source, CNCF sandbox project) goes even further than Hubble. Not just network traffic, but also application protocols — it automatically parses HTTP, gRPC, MySQL, PostgreSQL, Kafka, Redis, and more. Without a single line of instrumentation code, you see:

• Complete request/response payload for every database query
• Flamegraph CPU profiles for any pod in the cluster
• Distributed traces reconstructed from eBPF data (without OTel SDK in the application)
• Network traffic including encrypted TLS (Pixie hooks into SSL_read/SSL_write)

Pixie stores data edge-side (directly in the cluster, not in the cloud), which is essential for regulated environments — sensitive data never leaves your cluster. Retention is typically 24 hours (limited by RAM), but for debugging current issues, this is sufficient. For long-term storage, export to the OTel Collector.

eBPF ≠ replacement for OTel instrumentation

eBPF excels in network and system visibility. But business-level context (user ID, order ID, custom attributes) still needs to be instrumented in code via the OTel SDK. The best architecture combines both: eBPF for the infrastructure layer, OTel SDK for application context.

5. AI-driven Alerting — The End of Alert Fatigue¶

The average SRE team receives 500–2,000 alerts daily. Of these, 80–90% are false positives or duplicates. Alert fatigue is a real problem — when you get 50 alerts per hour, you stop reading them. And that one important alert slips through. AI-driven alerting changes this.

Anomaly Detection Instead of Static Thresholds¶

A static threshold “alert if CPU > 80%” is simple but dumb. Some services run at 90% CPU and that is normal (batch processing). Others should not exceed 30%. ML-based anomaly detection learns normal behavior patterns for each service and alerts only on actual deviations. Grafana ML (part of Grafana Cloud, but a self-hosted alternative via Prophet or seasonal ARIMA also exists) detects:

• Seasonal anomalies — distinguishes “Friday traffic spike” (normal) from “unexpected spike on Wednesday at 3 AM” (incident)
• Trend anomalies — slow latency increase of 5 ms per day, which after a month becomes +150 ms. A static threshold won’t catch it, ML will
• Correlation anomalies — error rate hasn’t changed, but error distribution has — instead of usual 404s, suddenly 500s

Alert grouping a root cause analysis¶

When a database goes down, you get alerts from all services that depend on it. That can be 50 alerts per minute — and you need to find the root cause. AI-powered alert grouping (implemented in Grafana OnCall, PagerDuty AIOps, or the open-source project Keep) automatically:

• Groups alerts by dependency graph — “all these alerts are likely related to database-primary down”
• Prioritizes — the root cause alert gets P1, downstream alerts P3
• Suggests remediation — based on historical incidents and runbooks

Practical result: 2,000 alerts per day become 3–5 meaningful alert groups, each with a suggested root cause and links to relevant dashboards and traces. MTTR drops from 43 minutes to 12.

6. Cost Control — Observability Doesn’t Have to Cost a Fortune¶

The biggest dirty secret of observability in 2026: costs. Enterprise companies commonly pay $50–200K annually for Datadog, Splunk, or New Relic. And as data volume grows, so do the bills — linearly, sometimes exponentially. Here is how to get it under control.

Strategy #1: Intelligent Sampling¶

Don’t send 100% of traces to the backend. Tail sampling in OTel Collector preserves 100% of errors and slow requests, but samples baseline traffic at 1–5%. For most services, this means a 95% reduction in trace volume without loss of diagnostic value. Head sampling (decision at the start of the request) is simpler, but you lose the ability to capture an error you don’t yet know about.

Strategy #2: Log Pipeline Optimization¶

Logs are typically 60–70% of total observability costs. Most logs are never read by anyone. Specific steps:

• Drop debug/info logs in production at the OTel Collector filter processor level — not in the application (you want to enable them during debugging)
• Structured logging — JSON logs instead of plain text. Smaller storage, faster querying, better correlation
• Tiered retention — hot data (7 days) on SSD, warm (30 days) on S3, cold (1 year) on Glacier. Loki supports this natively
• Label cardinality control — in Loki, label cardinality is the main cost driver. Don’t put request ID or user ID in labels, they belong in structured metadata

Strategy #3: Self-hosted vs. SaaS Trade-off¶

Self-hosted LGTM stack on Kubernetes typically costs 30–50% of the price of commercial SaaS solutions. But you add operational overhead — upgrades, scaling, incident response for the observability stack itself. Recommendations for Czech enterprise:

• < 50 services, small team: Grafana Cloud (managed LGTM) — simpler, free tier covers small projects
• 50–500 services, dedicated platform team: Self-hosted LGTM on Kubernetes — best cost/control ratio
• Regulated environment (banks, healthcare): Self-hosted, mandatory — data must not leave the infrastructure

60 % Typical savings switching from Datadog to self-hosted LGTM

95 % Trace volume reduction via tail sampling

10x lower Storage costs Loki vs. Elasticsearch

Conclusion: Observability as a Competitive Advantage¶

Kubernetes observability in 2026 is not about tools — it is about architectural decisions. OpenTelemetry as a universal instrumentation standard. LGTM stack as a cost-effective backend. eBPF for zero-code infrastructure visibility. AI-driven alerting for noise elimination.

Key lesson: start from the signals you need, not from the tools that are trendy. Define SLOs, identify critical user journeys, instrument them — and only then deal with storage, dashboards, and alerts. An observability stack is an investment, not a cost. Companies that do it right have 3x lower MTTR, 40% fewer incidents, and significantly happier engineering teams.

Need help with observability? At CORE SYSTEMS, we design and deploy observability stacks for Kubernetes — from greenfield implementation to migration from legacy monitoring. Get in touch.