Observability at Scale — Metrics, Traces & Logs for 1000+ Services
Target Level: Staff / Principal SRE & Platform Engineer — 15+ Yrs
1. Observability Strategy at Scale
Q1: How is observability different from monitoring? Why does this distinction matter at scale?
Answer: Monitoring tells you whether something is broken (known unknowns). Observability lets you understand why by asking arbitrary questions without shipping new code (unknown unknowns). At 1000+ services with hundreds of engineers making changes daily, you can't write a dashboard for every failure mode. You need structured data (metrics, traces, logs) with rich context (service name, region, version, user cohort) that you can slice and drill into dynamically. The most important pattern: high-cardinality metrics with dimensional labels, stored in a M3/Prometheus/Thanos stack that can handle millions of time-series with hundreds of label dimensions.
Q2: Design an observability architecture for 2000 microservices across 5 regions.
Solution:
Tier 1 — Metrics (Prometheus + Thanos + M3DB): Per-cluster Prometheus servers scrape local services, Thanos sidecars ship data to an object store (S3/GCS). Global Thanos querier provides a unified view. High-cardinality labels: service, team, region, deploy_version, error_type. Retention: 15 days hot (S3), 12 months downsampled.
Tier 2 — Tracing (OpenTelemetry + Jaeger/Tempo): OTel SDKs in every service emit traces with head-based sampling (10% for low-traffic, 1% for high-traffic). Tail-based sampling for critical paths (payment, auth). Trace context propagated via W3C TraceContext. Store in Grafana Tempo (object storage-backed, cheap).
Tier 3 — Logging (Loki + structured logging): JSON-structured logs via OTel collector → Kafka buffer → Loki. Structured fields: trace_id, service, level, error_kind. No unstructured logs allowed. Card labels: service, level, team. Logs cost-optimized: debug logs written to file, info+ to Loki, warn+ to cheap long-term storage.
Tier 4 — Service Graph (Cortex/Mimir): RED metrics (Rate, Errors, Duration) per service. Service dependency graph auto-generated from traces. Unmapped traffic detection identifies unknown service-to-service calls.
Q3: How do you choose sampling strategies for distributed tracing?
Answer: Three-tier approach:
1. Head-based sampling (decided at request start) — use for high-volume services. Adaptive sampler: sample 100% of requests that involve "critical" endpoints (payment, auth), 1% for everything else. Cost: predictable, low.
2. Tail-based sampling (decided after response) — keep traces that are slow, errored, or rare. An "error sampler" keeps every trace with status_code >= 500 or latency > p99.5. A "rare service sampler" ensures low-traffic services get sufficient representation. Cost: higher compute (need a buffer), but dramatically more useful.
3. Dynamic sampling — adjust rates based on error budget consumption. When error budget is healthy, sample lower (0.1%). When it's burning fast, bump to 10% for more signal.
Q4: How do you prevent observability costs from exploding?
Answer: Observability cost at scale is often the largest infrastructure line item after compute. Controls:
1. Rate limits: Enforce per-service metric cardinality limits (e.g., max 500 unique label values per metric, max 50 labels per metric). Prometheus metric_relabel_configs and write_relabel_configs to enforce at collector level.
2. Downsampling: Raw metrics retained 7 days, 10-second resolution. 30-day: 1-minute resolution. 1-year: 1-hour resolution. Use Thanos or M3DB downsampling rules.
3. Log cost controls: Not all logs need to be centralised. Debug logs → local file only. Info logs → cheap object storage. Error logs → long-term. Use log filtering at the collector, not the source.
4. Budget per team: Allocate observability budget to each team. Show them costs in their dashboard. Let them tune their own sampling rates. Human nature: when teams see costs, they optimize.
5. Observability-as-a-platform: Provide a managed OTel collector config that works out-of-the-box with sensible defaults. Teams that exceed defaults need a cost-justified exception.
Q5: You're woken up at 3 AM — latency p99 of the checkout service went from 200ms to 20s in the last 10 minutes. Walk me through your debug process.
Answer:
1. Check deploy dashboard — was a new version deployed in the last hour? (Most common cause.)
2. Check RED metrics — is the increase in latency across all endpoints or specific? Is error rate also up? If latency up + errors up → likely a bug. Latency up + rate flat + errors flat → likely a resource bottleneck.
3. Distributed tracing — open the trace query for checkout slow traces. Which span is the dominant contributor? Database query? Downstream service call? Serialization?
4. Check dependency health — drill into downstream services. Is the database slow? (Check RDS metrics — CPU, connections, replica lag.) Is Redis slow? Check cache hit rate dropped.
5. Check infrastructure metrics — pod CPU throttling? Node disk pressure? Connection pool exhaustion?
6. Check recent traffic changes — traffic spike? New client version? Feature flag turned on?
7. Hypothesis → rollback or mitigate — rollback the deploy if that's the likely cause. Or scale up the database. Or throttle non-critical traffic.
2. OpenTelemetry Architecture Design
Q6: Design an OpenTelemetry-based observability pipeline for a multi-cloud environment.
Solution:
Instrumentation: OTel SDKs in all services (auto-instrumentation for Java, Python, Go, Node.js). Manual instrumentation for business-critical spans.
OTel Collector pipeline: Each K8s node runs an OTel Collector DaemonSet. Collectors are configured with: receivers (OTLP gRPC, Prometheus scrape, filelog), processors (batch, attributes, memory_limiter, filter, sampling), exporters (Prometheus remote write, Jaeger gRPC, Loki push, S3 archive).
Multi-cloud: Each cloud provider runs its own collector tier. A global collector (cross-cloud) aggregates cross-region traces. Deduplication happens via trace_id hash.
Cost control: memory_limiter processor prevents OOM. batch processor reduces export calls. filter processor drops high-volume debug spans. probabilistic_sampler processor for head-based sampling.
Q7: How do you implement a service-level objective (SLO) monitoring system?
Answer: Build on top of the metrics pipeline:
1. Define SLIs: For each service — request latency (p50, p90, p99), error rate (5xx / total), throughput. Service-level objectives are counted over a rolling window (30 days).
2. Multi-window, multi-burn-rate alerts: Use Google's SRE workbook approach — two alerting windows (1h fast burn, 6h slow burn). A fast burn alert fires if the error budget is on track to be exhausted in < 6 hours. A slow burn fires if it's on track for < 3 days.
3. Tools: Prometheus sloth (SLO generator) creates recording rules for budget remaining, burn rate, and alerting rules. Grafana dashboards show burn rate and budget remaining.
4. Error budget policy: If a service consumes > 50% of its budget in a week, deploys are frozen. If > 75%, on-call is alerted and the team is expected to drop everything to investigate.
3. Interview Questions
Q8: Prometheus vs M3DB vs Thanos vs VictoriaMetrics — compare and contrast.
Answer: Prometheus is the ingestion standard but single-node only. Thanos adds global view and long-term storage via object store — battle-tested, but requires significant operational effort (sidecars, compactor, store gateways, querier). M3DB (Uber) handles higher cardinality natively with a clustered architecture but is complex to operate. VictoriaMetrics is simpler to run (single binary), has better disk compression (7x vs Prometheus), and handles high cardinality well — the practical choice for most teams. Recommendation: start with Thanos if you're already on Prometheus and need global view. Consider VictoriaMetrics for new deployments — cheaper and simpler. Avoid raw Prometheus for > 10M time-series.
Q9: How do you handle the cardinality explosion problem?
Solution: pod and user_id in metric labels cause cardinality explosion. Mitigations: (1) Aggregation at scrape time — replace user_id label with a user_tier label (free/premium/enterprise). (2) Record rules in Prometheus that aggregate raw high-cardinality metrics into low-cardinality summaries. (3) Monitoring as code — review all new metrics in code review, with a linter that warns if a metric has > 10 unique label values. (4) Use exemplars instead of labels for trace-level detail — embed trace_id in the metric via exemplars, not labels.
Q10: Design a multi-tenant observability system for a platform serving 50+ internal teams.
Answer: Tenant isolation patterns:
Option A (Cortex/Mimir): Prometheus-compatible API with built-in tenant isolation. Each team's data is separate by tenant_id header. Queries can only see their own data. Cost: shared infrastructure, per-team rate limits.
Option B (Per-team Prometheus + global Thanos query): Each team runs their own Prometheus. Thanos sidecar/query aggregates across teams. Teams can see each other's metrics for debugging but only edit their own dashboards.
Option C (Grafana Cloud / Datadog): Managed solution with RBAC. Teams self-serve their own dashboards and alerts. Default: all data visible to everyone with "break glass" access control.
Recommendation: Option B for engineering maturity, Option C if you want to avoid operational burden. Always enforce per-tenant rate limits to prevent one team's misconfiguration from affecting others.
4. Real-World Troubleshooting Scenarios
T1: Alert fatigue — on-call gets paged 50 times per night, most alerts are false positives
Symptoms: PagerDuty shows 50+ alerts per shift. Team starts ignoring alerts. A real incident goes unnoticed for 45 minutes.
Diagnosis:
1. Analyze alert-to-incident ratio — what % of pages result in a documented incident? Target: > 30%. If < 10%, your alerting is too noisy.
2. Check alert thresholds — are you alerting on p99 latency when the SLO target is 95th percentile? Adjust thresholds to match SLOs, not arbitrary numbers.
3. Check for flapping alerts — metric oscillates around the threshold (e.g., CPU at 79% when threshold is 80%). Add a "for" duration in Prometheus: for: 5m to require sustained breach.
4. Check for correlated alerts — a node failure triggers 20 pod-level alerts plus the node alert. Group alerts by root cause. Use alert aggregation (Alertmanager inhibition rules).
Remediation: Implement multi-window, multi-burn-rate SLO-based alerting. Each service has exactly 2 alert rules (fast burn, slow burn). If you need more than 5 Prometheus rules per service, you're over-alerting. Run a monthly "alert hygiene" review — remove alerts that haven't fired a genuine incident in 3 months.
T2: Distributed tracing shows gaps — spans missing for 30% of requests
Symptoms: Jaeger UI shows incomplete traces. Some spans are present, some are missing. Trace waterfall has "gaps" with no span for 200ms.
Diagnosis:
1. Check sampling configuration — are you using head-based sampling (probability at root span) or tail-based (sample after complete trace)? Head-based sampling: if the root span is sampled but a downstream service's sampler says "don't sample", that service's spans are missing. Solution: use consistent probability sampling (all services use the same sampler config), or switch to tail-based sampling with a backend like Grafana Tempo.
2. Check trace context propagation — is traceparent / x-request-id / b3 header properly propagated via HTTP headers or message queue headers? Common issue: async event processing (Kafka, SQS) drops the trace context when the message is serialized/deserialized. Configure the messaging library to propagate W3C trace context.
3. Check instrumentation coverage — is the service instrumented with OpenTelemetry SDK? Not all libraries are auto-instrumented. Manual instrumentation may be needed for custom frameworks or older libraries.
4. Check span exporter errors — otel-collector --validate or check collector logs for export errors. If the collector can't reach the tracing backend, spans are dropped.
Remediation: Standardize on W3C trace context across all services. Configure consistent sampling at the collector level (tail-based). Add trace context propagation tests in CI. Monitor span export success rate as a service-level indicator.
T3: Prometheus server is OOM-killed every 6 hours
Symptoms: Prometheus pod restarts. OOMKilled exit code 137. tsdb WAL replay takes 10+ minutes after restart.
Diagnosis:
1. Check cardinality — Prometheus memory usage ≈ 1-2KB per time-series. 10M time-series = 10-20GB RAM. Query count({__name__=~".+"}) to get total series count. Also check: topk(10, count by (__name__) ({__name__=~".+"})) to find metrics with the most series.
2. Check label cardinality — count(count by (pod, namespace) (kube_pod_info)) can reveal if a label like user_id, request_id, or container_id has millions of unique values.
3. Check scrape targets — are there too many targets scraped in a single Prometheus? A single Prometheus should handle < 500K time-series. Beyond that, use Thanos (vertical scaling) or shard by namespace/region.
4. Check retention — --storage.tsdb.retention.time=15d. If you have high ingestion rate and long retention, disk fills + memory pressure.
Remediation: Identify and fix high-cardinality metrics. Record rules to aggregate. Reduce retention to 7 days for raw data, rely on Thanos/VictoriaMetrics for long-term. Increase Prometheus memory to 16GB. Set --storage.tsdb.max-block-duration=2h to reduce WAL replay time.