Prometheus Interview Questions and Answers

Core Concepts & Architecture

1. What is Prometheus and how does it differ from traditional monitoring systems?

Answer:

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability in modern, dynamic environments. It was originally built at SoundCloud and is now a standalone open-source project maintained by the Cloud Native Computing Foundation (CNCF).

Key Differences from Traditional Monitoring Systems:

• Pull vs. Push Model: Prometheus primarily uses a pull-based model, where it actively scrapes metrics from configured HTTP endpoints on monitored targets. Traditional systems often use a push-based model, where agents on the monitored systems push metrics to a central server. The pull model simplifies configuration and makes it easier to monitor ephemeral services in environments like Kubernetes.
• Multi-dimensional Data Model: Prometheus stores all data as time series, with each time series being identified by a metric name and a set of key-value pairs called labels. This multi-dimensional data model allows for powerful and flexible querying.
• Powerful Query Language (PromQL): Prometheus has its own query language, PromQL, which is designed for working with time-series data. It allows for complex queries, aggregations, and alerting.
• Service Discovery: Prometheus has built-in service discovery mechanisms that allow it to dynamically discover and monitor new targets, which is essential in cloud-native environments.
• No Dependencies: The Prometheus server is a single binary with no external dependencies, making it easy to deploy and manage.

2. Explain the architecture of Prometheus and its key components.

Answer:

The Prometheus architecture consists of several core components:

• Prometheus Server: The central component that scrapes and stores time-series data, processes PromQL queries, and generates alerts.
• Exporters: These are agents that run on monitored targets and expose metrics in a Prometheus-compatible format. There are exporters for a wide variety of systems, such as the Node Exporter for host metrics, the Blackbox Exporter for endpoint probing, and various database exporters.
• Pushgateway: An intermediary service for pushing metrics from short-lived or batch jobs that cannot be scraped.
• Alertmanager: Handles alerts generated by Prometheus, including deduplicating, grouping, and routing them to various notification channels like email, Slack, or PagerDuty.
• Service Discovery: Automatically discovers targets to be scraped. It has built-in support for discovering targets in Kubernetes, AWS, and other cloud providers.
• Grafana: While not a part of Prometheus itself, Grafana is a popular open-source visualization tool that is often used with Prometheus to create dashboards and visualize metrics.

PromQL

3. What is PromQL and how is it used?

Answer:

PromQL (Prometheus Query Language) is a powerful functional query language that allows you to select and aggregate time-series data stored in Prometheus. It is used for:

• Ad-hoc querying: You can use PromQL to explore your metrics and to troubleshoot problems.
• Dashboards: You can use PromQL to create dashboards in Grafana to visualize your metrics.
• Alerting: You can use PromQL to create alerting rules that will notify you when a certain condition is met.

4. What are the four metric types in Prometheus?

Answer:

• Counter: A cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors.
• Gauge: A metric that represents a single numerical value that can arbitrarily go up and down. For example, you can use a gauge to represent the current memory usage or the number of concurrent requests.
• Histogram: A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
• Summary: Similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window.

5. Explain the concept of labels in Prometheus metrics and their importance.

Answer:

Labels are key-value pairs that are associated with time-series data in Prometheus. They provide a multi-dimensional data model, allowing for powerful filtering, aggregation, and grouping of metrics. For example, the metric http_requests_total could have labels like method="GET", path="/api/users", and status="200".

Importance of Labels:

• Granular Analysis: Labels allow you to distinguish metrics from different instances, services, or environments.
• Flexible Querying: You can use labels to easily slice and dice data to focus on specific subsets.
• Service Discovery: Labels are often populated by service discovery mechanisms to identify targets.

Alerting

6. How does Prometheus handle alerting, and what is the role of Alertmanager?

Answer:

Prometheus handles alerting in two parts:

1. Alerting Rules: You define alerting rules in Prometheus that specify a condition that must be met for an alert to be triggered. These rules are written in PromQL.
2. Alertmanager: The Alertmanager is a separate component that is responsible for handling alerts generated by Prometheus. It can deduplicate, group, and route alerts to various notification channels like email, Slack, or PagerDuty.

7. What is an alerting rule in Prometheus? Provide an example.

Answer:

An alerting rule is a PromQL expression that is evaluated at regular intervals. If the expression evaluates to true, an alert is generated.

Example:

This alerting rule will fire if the memory usage of a node is greater than 80% for more than 5 minutes:

alert: HighMemoryUsage
expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 80
for: 5m
labels:
  severity: warning
annotations:
  summary: "High memory usage on {{ $labels.instance }}"
  description: "{{ $labels.instance }} has been using more than 80% of its memory for the last 5 minutes."

Configuration

8. What is the purpose of the prometheus.yml file?

Answer:

The prometheus.yml file is the main configuration file for Prometheus. It is used to configure the following:

• Global settings: Such as the scrape interval and the evaluation interval.
• Scrape configs: Which targets to scrape and how to scrape them.
• Alerting rules: The alerting rules that Prometheus should evaluate.
• Remote write: Where to send metrics for long-term storage.

9. How do you configure Prometheus to scrape metrics from a target?

Answer:

You can configure Prometheus to scrape metrics from a target by adding a scrape_config to the prometheus.yml file.

Example:

This scrape_config will scrape metrics from a target at localhost:9090:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

10. How do you configure service discovery in Prometheus?

Answer:

Prometheus supports a number of service discovery mechanisms, such as:

• Static configs: You can manually specify the targets to scrape.
• File-based service discovery: You can specify the targets to scrape in a file.
• Kubernetes service discovery: You can use the Kubernetes API to discover the targets to scrape.
• Cloud provider service discovery: You can use the APIs of cloud providers like AWS, Azure, and GCP to discover the targets to scrape.

Example (Kubernetes Service Discovery):

This scrape_config will discover and scrape all pods in a Kubernetes cluster that have the annotation prometheus.io/scrape: 'true':

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

Troubleshooting

11. How do you troubleshoot a Prometheus target that is not being scraped?

Answer:

1. Check the Prometheus UI: Go to the Status > Targets page in the Prometheus UI. This will show you the status of all of your targets. If a target is not being scraped, it will be in the DOWN state.
2. Check the target's logs: Check the logs of the target to see if there are any errors.
3. Check the network connectivity: Make sure that there is network connectivity between the Prometheus server and the target.
4. Check the scrape configuration: Make sure that the scrape configuration in the prometheus.yml file is correct.

12. How do you troubleshoot an alert that is not firing?

Answer:

1. Check the alerting rule: Make sure that the alerting rule is correct and that the PromQL expression is evaluating to true.
2. Check the Alertmanager: Make sure that the Alertmanager is running and that it is configured to receive alerts from Prometheus.
3. Check the notification channel: Make sure that the notification channel is configured correctly and that it is able to receive alerts from the Alertmanager.

13. How do you troubleshoot a slow PromQL query?

Answer:

1. Use the query_range API: The query_range API allows you to specify a time range for a query. This can be useful for troubleshooting slow queries, as you can use it to narrow down the time range that is causing the problem.
2. Use the promtool command-line tool: The promtool command-line tool can be used to check the syntax of a PromQL query and to see how it will be evaluated.
3. Use a recording rule: A recording rule is a PromQL expression that is evaluated at regular intervals and whose result is stored as a new time series. This can be useful for pre-calculating expensive queries.

14. What do you do when Prometheus alerts fail to route?

Answer:

When Prometheus alerts fail to route, you should:

1. Validate Alertmanager configuration: Check alertmanager.yml for correct routing trees, receivers, and notification configurations.
2. Check connectivity: Ensure the Alertmanager can reach its configured receivers (e.g., Slack API, email server).
3. Test in staging: If possible, test alert configurations in a staging environment.
4. Review Alertmanager logs: Look for errors or warnings related to alert processing and routing.
5. Prometheus server logs: Check Prometheus logs to ensure alerts are being generated and sent to Alertmanager.
6. Check firewall rules: Ensure no network policies or firewalls are blocking communication between Prometheus and Alertmanager, or between Alertmanager and its receivers.
7. Version control: Ensure configurations are versioned in a repository for traceability.

15. Have you used Prometheus in Kubernetes? How?

Answer:

Yes, Prometheus is widely used in Kubernetes environments. This typically involves:

• Deployment: Deploying Prometheus using Helm charts or the Prometheus Operator.
• Service Discovery: Configuring Prometheus to automatically discover pods and services via Kubernetes service discovery mechanisms (e.g., kubernetes_sd_config).
• kube-state-metrics: Deploying kube-state-metrics to expose metrics about the state of Kubernetes objects (pods, deployments, nodes, etc.).
• Node Exporter: Running Node Exporter as a DaemonSet on each node to collect host-level metrics.
• Application instrumentation: Instrumenting applications running in Kubernetes to expose Prometheus-compatible metrics.
• Grafana integration: Setting up Grafana dashboards to visualize Kubernetes cluster health and application performance.

---

16. What is the difference between Prometheus and ELK (Elasticsearch, Logstash, Kibana)?

Answer:

Prometheus is primarily designed for metrics monitoring (e.g., CPU, memory, performance, application-specific metrics) and alerting based on time-series data. ELK (Elasticsearch, Logstash, Kibana) is primarily used for log monitoring, aggregation, and analysis. While both are crucial for observability, Prometheus focuses on numerical time-series data, and ELK focuses on unstructured or semi-structured log data. They are often used together in a comprehensive monitoring stack.

17. Scenario: You receive an alert indicating high CPU usage on a critical production server. How would you use Prometheus to diagnose the issue?

Answer:

1. Identify the target: Use the alert details to pinpoint the specific server.
2. Query relevant metrics: Use PromQL to query CPU usage metrics for that server, such as node_cpu_seconds_total (with mode label for idle, user, system, etc.) or specific CPU core usage metrics.
3. Analyze trends: Look for spikes or sustained high values in CPU usage over time.
4. Correlate with other metrics: Check other system metrics like memory usage (node_memory_MemAvailable_bytes), disk I/O (node_disk_io_time_seconds_total), and network activity (node_network_receive_bytes_total, node_network_transmit_bytes_total) to see if resource constraints are contributing.
5. Application-specific metrics: If available, check application metrics (e.g., request latency, error rates) to see if the high CPU is correlated with application load.
6. Drill down: Based on the analysis, further investigate specific processes or services running on the server that might be consuming excessive CPU.

---

18. What are some best practices for setting up Prometheus in production?

Answer:

• Use Node Exporter on every server: For comprehensive system-level metrics.
• Set proper retention policies: Define how long Prometheus stores metrics using the --storage.tsdb.retention.time flag.
• Optimize scraping intervals: Avoid very low intervals that can strain resources.
• Use relabeling and filtering: To manage data efficiently and refine collected metrics.
• Implement high-availability setups: Run multiple Prometheus instances and use external aggregation for redundancy.
• Configure Alertmanager effectively: Use grouping, silencing, and routing to avoid alert spam and ensure critical issues are notified.
• Utilize service discovery: Especially in dynamic environments like Kubernetes, to automatically discover and monitor targets.
• Integrate with Grafana: For effective visualization and reporting.
• Use recording rules: For heavy or frequently used PromQL queries to improve performance.
• Version control configurations: Store prometheus.yml and alert rules in a versioned repository.

---

19. How do you integrate Prometheus with Grafana for visualization?

Answer:

Integrating Prometheus with Grafana involves configuring Prometheus as a data source in Grafana. Once configured, you can create dashboards and panels in Grafana, using PromQL queries to retrieve and display metrics from Prometheus. Grafana allows for various visualization types (graphs, gauges, tables) and supports templating for dynamic dashboards.

---

20. Describe the function and use cases of the Alertmanager in Prometheus.

Answer:

Alertmanager is a component of Prometheus that processes alerts generated by the Prometheus server based on predefined conditions. Its functions include:

• Grouping: Combines similar alerts into a single notification to reduce noise.
• Deduplication: Prevents sending duplicate notifications for the same alert.
• Silencing: Temporarily mutes alerts for specific time periods or conditions.
• Routing: Directs alerts to different receivers (e.g., email, Slack, PagerDuty) based on their labels.

Common use cases include setting alerts for high CPU usage, memory consumption, or endpoint availability.

---

21. How does Prometheus handle metric cardinality explosion?

Answer:

Cardinality explosion occurs when the number of unique time series grows too large, which can strain resources. Prometheus handles this by using a time-series database, but issues can arise if labels are too dynamic or metrics grow uncontrollably. Strategies to mitigate this include:

• Careful label design: Avoid labels with highly dynamic or unique values (e.g., timestamps, request IDs).
• Relabeling: Use Prometheus's relabeling configurations to drop or modify labels before storage.
• Recording rules: Pre-aggregate frequently queried, high-cardinality metrics into lower-cardinality ones.

---

22. Explain the concept of labels in Prometheus metrics.

Answer:

Labels are key-value pairs associated with time series data in Prometheus. They provide dimensional data modeling, allowing for powerful querying and aggregation capabilities. Each unique combination of a metric name and its labels identifies a distinct time series.

---

Practical Implementation & Troubleshooting

23. Provide a complete prometheus.yml example with mandatory fields.

Answer:

This example demonstrates a basic but complete prometheus.yml configuration file.

# prometheus.yml

# The global configuration block defines parameters that are valid for all other
# configuration contexts.
global:
  # How frequently to scrape targets by default.
  scrape_interval: 15s
  # How frequently to evaluate rules.
  evaluation_interval: 15s
  # A label to attach to all time series or alerts.
  external_labels:
    monitor: 'my-prometheus'

# A list of files from which to load alerting rules.
rule_files:
  - 'alert.rules.yml'

# The alerting block configures the connection to the Alertmanager.
alerting:
  alertmanagers:
    - static_configs:
        # The address of the Alertmanager.
        - targets: ['localhost:9093']

# The scrape_configs block defines a set of targets to scrape.
scrape_configs:
  # The job_name is a label that is added to any time series scraped from this config.
  # It's a mandatory field.
  - job_name: 'prometheus'
    # This job scrapes the Prometheus server itself.
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    # For monitoring a standalone machine (e.g., a VM or bare-metal server).
    # The 'targets' list contains the network addresses of the exporters.
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']
        labels:
          group: 'production-servers'

  - job_name: 'kubernetes-pods'
    # This job uses Kubernetes Service Discovery to find and scrape pods.
    kubernetes_sd_configs:
      # The role determines which Kubernetes objects to discover.
      - role: pod
    relabel_configs:
      # This relabeling rule keeps only pods that have the specific annotation.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # This captures the value of the annotation for the metrics path.
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # This rewrites the address label to use the pod's IP and the discovered port.
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

24. Explain how metrics are scraped from pods and standalone systems with examples.

Answer:

1. Scraping Metrics from Pods (in Kubernetes):

In Kubernetes, Prometheus uses service discovery to automatically find and scrape pods. This is the preferred method because pods are ephemeral and their IPs change frequently.

How it works: 1. Instrumentation: The application running in the pod must expose its metrics on an HTTP endpoint (e.g., /metrics). This is often done using a client library (e.g., for Java, Go, Python). 2. Annotations: You add specific annotations to your Pod or Service's metadata in the Kubernetes deployment YAML. These annotations tell a properly configured Prometheus where and how to scrape the metrics. 3. Prometheus Configuration: The prometheus.yml file is configured with a kubernetes_sd_configs job that watches the Kubernetes API for pods with those specific annotations.

Example: * Pod's Deployment YAML (partial): yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app-deployment spec: template: metadata: annotations: prometheus.io/scrape: 'true' # Tells Prometheus to scrape this pod prometheus.io/port: '8080' # The port to scrape on prometheus.io/path: '/actuator/prometheus' # The path for metrics ... * Prometheus scrape_configs (as seen in the previous question): The kubernetes-pods job will discover this pod, read its annotations, and start scraping metrics from http://<pod-ip>:8080/actuator/prometheus.

2. Scraping Metrics from a Standalone System:

For standalone systems (like VMs or bare-metal servers), you typically use an exporter and a static configuration.

How it works: 1. Install an Exporter: You install an exporter on the standalone machine. A common one is the Node Exporter, which provides system-level metrics (CPU, memory, disk, network). 2. Run the Exporter: The exporter runs as a service on the machine, exposing metrics on a specific port (e.g., port 9100 for Node Exporter). 3. Static Configuration: You add a static_configs block to your prometheus.yml file, manually listing the IP address and port of the exporter.

Example: * On the standalone machine (192.168.1.10): bash # Download and run the Node Exporter ./node_exporter --web.listen-address=":9100" * Prometheus scrape_configs: yaml scrape_configs: - job_name: 'node-exporter' static_configs: - targets: ['192.168.1.10:9100'] Prometheus will now periodically scrape metrics from http://192.168.1.10:9100/metrics.

25. What PromQL queries are used in real-time troubleshooting?

Answer:

Here are a few examples of PromQL queries used for troubleshooting:

1. CPU Usage (Top 5 hosts): Find the top 5 machines by average CPU usage over the last 15 minutes. promql topk(5, 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[15m])) * 100))

2. High Memory Usage: Find instances using more than 85% of their memory. promql (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85

3. High HTTP Error Rate: Calculate the 5xx error rate for a web service over the last 5 minutes. promql sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100 > 5

4. API Request Latency (95th percentile): Find the 95th percentile latency for API requests over the last 10 minutes. promql histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[10m])) by (le, path))

5. Filesystem Space: Find filesystems with less than 20% free space. promql node_filesystem_avail_bytes / node_filesystem_size_bytes * 100 < 20

26. What are the installation steps for Prometheus on Kubernetes and on a standalone machine?

Answer:

1. Installation on Kubernetes (using Helm):

Using the official kube-prometheus-stack Helm chart is the most common and recommended method. It installs Prometheus, Alertmanager, Grafana, and various exporters.

1. Add Helm Repo: bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update
2. Create a Namespace: bash kubectl create namespace monitoring
3. Install the Chart: bash helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring This deploys a full monitoring stack. You can customize the installation by creating a values.yaml file to override default settings (e.g., persistence, service types).

2. Installation on a Standalone Machine (Linux):

1. Download Prometheus: Go to the official Prometheus downloads page and get the latest release for your OS and architecture. bash wget https://github.com/prometheus/prometheus/releases/download/v2.40.1/prometheus-2.40.1.linux-amd64.tar.gz
2. Extract the Files: bash tar xvfz prometheus-2.40.1.linux-amd64.tar.gz cd prometheus-2.40.1.linux-amd64
3. Configure prometheus.yml: Edit the prometheus.yml file to add your scrape targets. By default, it's configured to scrape itself.
4. Start Prometheus: bash ./prometheus --config.file="prometheus.yml" Prometheus will now be running and accessible at http://localhost:9090. For production use, you should run it as a systemd service.

27. How do I discover targets for scraping? Does pom.xml help?

Answer:

Target discovery depends on your environment. Build files like pom.xml (for a Java project) do not directly tell Prometheus about scrape targets. Their role is indirect: they are used to build the application, which includes the code that exposes a metrics endpoint. The discovery happens at runtime.

1. Discovering Targets for static_configs (Standalone):

For static_configs, you need to know the IP address and port of the service you want to scrape. * How do you find this? This information comes from your infrastructure provisioning or network configuration. If you set up a VM with an application, you know its IP address. If the application documentation says it exposes metrics on port 8080, then your target is <ip-address>:8080. * Example: You have a Java application running on a server at 10.0.1.5. The Spring Boot Actuator is configured to expose Prometheus metrics on port 9001. Your static_configs target would be 10.0.1.5:9001.

2. Discovering Pod Targets in Kubernetes:

This is where automatic service discovery shines. You don't need to know the pod's IP address. * How it works: You add annotations to your Kubernetes objects (Deployments, Services, etc.). The kubernetes_sd_config in Prometheus watches the Kubernetes API for objects that match its criteria (e.g., have the prometheus.io/scrape: 'true' annotation). * What you need to know: You only need to know the annotations to use and the port name or number your application is exposing metrics on. This information should be part of your team's development and deployment standards. * The actual target address (<pod-ip>:<port>) is discovered and managed entirely by Prometheus. You never need to hardcode it in the Prometheus configuration. In the Prometheus UI (Status -> Targets), you will see the dynamically discovered pod IPs listed as targets.

---