The 50 Grafana questions interviewers actually ask, with direct answers, real queries and config, and what the interviewer is listening for. Grouped for freshers, intermediate, and experienced rounds.
50 questions with answersKey Takeaways
Grafana is an open-source platform for visualizing, querying, and alerting on data from many sources. You point it at a backend that already holds your data (Prometheus for metrics, Loki for logs, a SQL database, a cloud monitoring service) and Grafana runs queries against that source and draws the results as panels on a dashboard. The thing candidates miss most often is that Grafana doesn't store your metrics or logs itself; it's a read layer over other systems. The official Grafana introduction documentation describes it as a way to query, visualize, and alert on data no matter where it's stored, which is the mental model to carry into the interview. In interviews, Grafana questions probe how you reason about the pieces: what a data source really is, how a panel query differs across backends, why a variable makes a dashboard reusable, how alert rules and notification routing fit together, and what you'd do when a dashboard is slow or an alert won't stop firing. This page collects the 50 questions that come up most, each with a direct answer plus real queries and config. Pair this bank with our AI interview preparation guides for the live-interview format, since the first monitoring round is increasingly an AI-driven technical screen.
Watch: Server Monitoring // Prometheus and Grafana Tutorial
Video: Server Monitoring // Prometheus and Grafana Tutorial (Christian Lempa, YouTube)
Test yourself and earn a certificate
10 quick questions. Score 70%+ to download your Grafana certificate.
The fundamentals every entry-level round checks: what Grafana is, data sources, panels, dashboards, variables, and the query basics underneath. If any answer here surprises you, that's your study list.
Grafana is an open-source tool for visualizing, querying, and alerting on data from many sources. You it connects to a backend that already holds your data, like Prometheus or a SQL database, and Grafana queries that source and draws the results as dashboards.
It solves the problem of scattered, hard-to-read operational data. Instead of logging into five systems to check health, you get one place that pulls metrics and logs from all of them, visualizes trends, and alerts when something's wrong. The key point: Grafana visualizes data, it doesn't store it.
Key point: Lead with 'it visualizes data from other sources, it doesn't store it'. That single sentence is what separates people who understand Grafana from people who've only seen a dashboard.
Watch a deeper explanation
Video: Grafana Tutorial for Beginners: What is Grafana (Tech With Diego, YouTube)
No, and this trips people up. Grafana is a visualization and query layer, not a database. The actual metrics live in a data source like Prometheus, InfluxDB, or a SQL database, and Grafana runs a fresh query against that source every time you load a panel or evaluate an alert.
The only thing Grafana stores itself is its own configuration: dashboards, users, data source connections, alert rules, and settings, in an internal database (SQLite by default, or an external MySQL/PostgreSQL). Your time-series data stays in the backend.
Key point: Interviewers ask this to catch a common misconception. Being clear that Grafana stores config, not metrics, shows you understand the architecture.
A data source is a configured connection to a backend that Grafana can query: Prometheus, Loki, Elasticsearch, a SQL database, a cloud monitoring service. Each type has its own query editor, so writing a Prometheus query looks different from writing a SQL query.
You add a data source once (URL, auth, options), then panels reference it. One Grafana can hold many data sources, and a single dashboard can mix them, one panel from Prometheus, another from a SQL database, side by side.
Key point: Mention that one dashboard can mix multiple data sources. That detail shows you understand Grafana's data-source-agnostic design, a common follow-up.
A panel is a single visualization: one graph, table, gauge, or stat, driven by one or more queries against a data source. A dashboard is a collection of panels arranged on a grid, usually telling one story like 'the health of this service'.
You build a dashboard by adding panels, each with its own query and visualization type. Dashboards can also carry variables, a time range, and a refresh interval that apply across all their panels.
Key point: Keep the hierarchy crisp: query feeds a panel, panels make a dashboard. Muddling these is a giveaway that you haven't actually built one.
The everyday ones are the time series graph (values over time, the default for metrics), the stat panel (one big number, like current error rate), the gauge (a value against a range), the bar chart, the table (raw rows), and the state timeline or status history for up/down states.
There's also the logs panel for Loki or log data, the heatmap for distributions, and geomap for location data. You pick the type to match the question: a trend wants a time series, a single current value wants a stat or gauge, and detailed rows want a table.
Key point: Naming a few types and matching each to a use case (trend to graph, single value to stat) reads better than listing every panel Grafana has.
Prometheus is an open-source metrics database and monitoring system. It scrapes metrics from your applications on an interval and stores them as time series on disk, and it has its own query language, PromQL. Prometheus owns the collection and storage half of monitoring, which is exactly the part Grafana doesn't do.
Grafana handles visualization. The two pair constantly: Prometheus stores the metrics, Grafana connects to it as a data source and draws dashboards and alerts on top. Prometheus does have a basic UI, but Grafana is the usual choice for real dashboards. They're complementary, not competitors.
Key point: Say clearly 'Prometheus stores and Grafana visualizes'. Confusing the two, or calling Grafana a database, is the most common junior mistake here.
Watch a deeper explanation
Video: Master Prometheus Monitoring: Step-by-Step Tutorial and Demo (Kaiwalya Koparkar, YouTube)
PromQL is Prometheus's query language. When Prometheus is your data source, the query you type into a Grafana panel or alert rule is PromQL, so you can't really build Prometheus-backed dashboards or alerts without knowing the basics. It comes up in almost every Grafana interview because the query and the visualization are two sides of the same panel.
The essentials the question expects: selecting a metric by name and labels, using rate() on counters, and aggregating with sum(), avg(), or max() plus by() to group. You don't need to be an expert, but writing a simple rate-and-sum query is table stakes.
# per-second request rate, summed by status code
sum(rate(http_requests_total[5m])) by (status)
# 95th percentile latency from a histogram
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))Key point: Being able to write a rate() plus sum() by() query out loud shows you can actually build a panel, not just describe one.
A counter only ever goes up (or resets to zero on restart): total requests served, total errors. You almost never graph a counter raw; you wrap it in rate() to get a per-second rate, because the raw number just climbs forever.
A gauge goes up and down and represents a current value: memory in use, temperature, queue depth, active connections. You can graph a gauge directly because its value at any moment is meaningful on its own.
| Counter | Gauge | |
|---|---|---|
| Direction | Only increases (resets on restart) | Goes up and down |
| Example | Total requests, total errors | Memory used, queue depth |
| How you graph it | Usually with rate() or increase() | Directly, raw value |
Key point: Knowing you wrap a counter in rate() but graph a gauge directly is the practical test here. It shows you've built real Prometheus panels.
A template variable is a dropdown at the top of a dashboard that feeds a value into the panel queries. Instead of hardcoding a hostname, you define a $host variable, and the query uses $host, so switching the dropdown re-runs every panel for that host.
They make one dashboard reusable across many targets: one 'service overview' dashboard that works for any service, environment, or region by picking from a dropdown, rather than copying the dashboard a dozen times. Variables can be static lists, or query-driven so the options come from the data itself.
# a query-driven variable populates the $instance dropdown
label_values(node_cpu_seconds_total, instance)
# panels then reference it
rate(node_cpu_seconds_total{instance="$instance"}[5m])Key point: Give the concrete win: one reusable dashboard instead of copies per host. That framing is what the question is really checking.
The time range picker at the top right sets the window every panel queries over: last 15 minutes, last 24 hours, or an absolute range. It applies to the whole dashboard by default, so all panels stay in sync, which is what you want when investigating an incident.
It supports relative ranges like now-6h and 'quick ranges', plus zooming by dragging on a graph. Under the hood Grafana passes the from/to timestamps into each query, so the same dashboard shows this hour or last week depending on the picker.
Key point: Note that the range applies dashboard-wide so panels stay in sync. That's the operational detail that matters during an incident.
Every dashboard is really a JSON document that describes its panels, queries, variables, and layout. Grafana keeps that JSON in its internal database, but you can export it to a file and import it elsewhere, which is how dashboards get shared.
That JSON is also what makes dashboards version-controllable: you can commit the file to Git, review changes, and provision it into Grafana automatically. The public Grafana dashboard library works the same way, you import a community dashboard by its JSON or ID.
Key point: Mention that dashboards are JSON you can put in Git. It sets up the provisioning and version-control questions that often follow.
A dashboard lives inside a folder, and folders group related dashboards and are the main unit for permissions, you grant a team access to a folder rather than to each dashboard. This keeps a large Grafana tidy and controls who sees what.
An organization is a higher-level tenant: a fully separate space with its own users, data sources, and dashboards. Most setups use one organization and many folders; multiple organizations are for hard multi-tenant separation.
Key point: Knowing folders are the unit of permissions, not individual dashboards, shows you've managed a real Grafana with teams.
Grafana core (OSS) is free and open source, licensed under AGPLv3, and you can self-host it with no license cost. It covers dashboards, most data sources, unified alerting, users, and folders, which is what most people mean when they say 'Grafana'. You only pay when you want the managed cloud or enterprise-only features on top.
Grafana Enterprise (and Grafana Cloud) add paid features on top: enterprise data source plugins, reporting, fine-grained access control, SSO integrations, and support. So the honest answer is: the core is free, and you pay for enterprise features or a managed cloud offering.
Key point: Don't say Grafana is a paid product. Being precise that the OSS core is free and Enterprise adds features is the accurate answer.
Grafana's web server listens on port 3000 by default, so you reach the UI at http://host:3000 and log in with the admin user you set on first start. That port is a common thing interviewers check you actually know, and it's also the first thing you change or map when you put Grafana behind a reverse proxy or run it in a container alongside other services.
Configuration comes from the grafana.ini file, environment variables, or provisioning files. Common changes are the HTTP port, the root URL (when behind a reverse proxy), the database backend, and auth settings. Environment variables follow the GF_SECTION_KEY pattern, which is handy in containers.
# grafana.ini
[server]
http_port = 3000
root_url = https://grafana.example.com
# same setting via env var (containers)
# GF_SERVER_HTTP_PORT=3000An alert watches a query and notifies you when a condition is met. You define an alert rule, for example 'error rate above 5% for 5 minutes', Grafana evaluates it on a schedule, and when it breaches, the alert fires and a notification goes out.
The notification lands wherever you've configured: Slack, email, PagerDuty, and so on. So the loop is: a query condition, an evaluation interval, and a destination. The point of alerting is you don't have to stare at a dashboard, Grafana tells you when something's wrong.
Key point: Frame alerting as 'so you don't have to watch the dashboard'. That's the purpose, and it sets up the deeper alerting questions in the next tier.
Both visualize data, but Grafana is data-source agnostic: it connects to Prometheus, Loki, SQL databases, cloud services, and more, so it's the general dashboard tool. Kibana is part of the Elastic stack and is built tightly around Elasticsearch, where it's strong for log search and exploration.
So the choice usually follows the data: if your world is Elasticsearch logs, Kibana is natural; if you have metrics in Prometheus and want one place for many backends, Grafana fits. Grafana can even query Elasticsearch as a data source, but Kibana can't easily reach non-Elastic sources.
| Grafana | Kibana | |
|---|---|---|
| Data sources | Many (Prometheus, Loki, SQL, cloud) | Elasticsearch / OpenSearch |
| Strongest at | Metrics dashboards across backends | Log search and exploration |
| Can query the other's data? | Yes, Elasticsearch is a data source | Not really, tied to Elastic |
Key point: Anchor the difference in 'Grafana is source-agnostic, Kibana is Elastic-centric'. That's the distinction the question is testing.
The refresh interval sets how often Grafana re-runs the panel queries and redraws the dashboard: every 10 seconds, every minute, or off entirely. It's how a dashboard on a wall stays live without anyone reloading the page, and it applies across every panel on that dashboard rather than per panel, so the whole view updates together on the same beat.
It's a trade-off: a short interval keeps data fresh but adds query load on your data source, which matters if many people watch heavy dashboards. For most cases a 30-second to 1-minute refresh is plenty, and you can leave it off for historical or investigation dashboards.
Key point: the query-load trade-off of a fast refresh matters.
In the panel options you set the unit (bytes, seconds, percent, requests/sec) so Grafana formats axes and tooltips correctly, 1500000000 becomes 1.5 GB instead of a wall of digits. Getting the unit right is a small thing that makes dashboards actually usable.
The legend controls how series are labeled and can show summary values (last, max, mean). You can also use field overrides to style specific series, and set thresholds so a value turns red past a limit. These touches are what separate a rough panel from one people trust at a glance.
Key point: Bringing up units and thresholds in practice signals you've built dashboards people actually read, not just working queries.
Grafana hosts thousands of shared dashboards, each with a numeric ID. In the UI you go to import, paste the dashboard ID (or its JSON), pick the data source it should query, and Grafana builds it for you. It's the fastest way to get a solid starting dashboard for a common tool like a Node exporter or a database.
The catch is the data source mapping: an imported dashboard assumes certain metric names, so it works cleanly only if your metrics match what it expects, usually true for standard exporters. Treat imports as a starting point you then adjust, not a finished dashboard.
Key point: Saying 'imports are a starting point, then you adapt them to your metrics' shows you've actually used the library rather than assuming it's plug-and-play.
For candidates with working experience: alerting internals, provisioning, transformations, mixed data sources, and the judgment questions that separate dashboard clickers from operators.
Unified alerting has a few moving parts. An alert rule holds a query and a condition plus an evaluation interval; Grafana runs it on schedule and produces alert instances (one per series). Those instances carry labels, which is how routing decides where they go.
Firing alerts flow into notification policies, a tree that matches on labels and routes to contact points (Slack, email, PagerDuty, webhook). Along the way you get grouping (bundle related alerts into one message), silences (mute during maintenance), and mute timings. So it's: rule evaluates, instance fires, policy routes by label, contact point delivers.
Key point: Naming the chain (rule to instance to policy to contact point) and the fact that routing is label-based proves you've configured real alerting, not just toggled a threshold.
A rule sits in Normal when the condition isn't met. When the query breaches the condition it goes to Pending, and it only becomes Firing after it's stayed breached for the 'for' duration, which is what stops a single spiky scrape from paging you.
There are also NoData and Error states for when the query returns nothing or fails, and you choose how to treat those (alerting, keeping the last state, or OK). Understanding Pending versus Firing is the practical bit: the 'for' duration is your main lever against flappy alerts.
Key point: Explaining that the 'for' duration is what turns Pending into Firing, and why that reduces flapping, is the exact insight the question needs here.
the rule: raise the 'for' duration so a brief blip doesn't page, and check the threshold is tied to real user impact rather than a raw resource number comes first. Alerting on symptoms (error rate, latency) instead of causes (CPU) removes a lot of false pages on its own.
Then use the routing tools: group related alerts so one incident isn't ten messages, add silences during known maintenance, and use 'keep last state' for NoData if a flaky scrape is the cause. If it's genuinely borderline, consider alerting on a burn rate over a longer window rather than an instantaneous value.
Key point: Offering both the rule fix (for duration, symptom-based threshold) and the routing fix (grouping, silences) shows you've actually tamed a noisy on-call, which is what this scenario screens for.
Provisioning configures Grafana from files (or the API) instead of by hand in the UI. You put YAML files that declare data sources, and dashboard JSON, in a directory Grafana reads on startup, and it creates or updates them automatically.
The payoff is reproducibility and version control: your data sources, dashboards, and alert rules live in Git, get code-reviewed, and deploy the same way to every environment. Hand-clicking works for one dashboard, but at scale it drifts and can't be recreated. Provisioning is how you treat monitoring config as code.
# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: trueKey point: provisioning maps to 'monitoring config as code' and reproducible environments. That framing is what lifts the answer above 'it loads files'.
Transformations reshape query results inside Grafana before they're drawn, without changing the underlying query. Common ones: join results from multiple queries by a field, filter or rename fields, group by, add a calculated field, or convert a time series to a table.
You reach for them when the data source can't easily produce the exact shape you want, or when you're combining results from different queries or data sources in one panel. The caution: heavy transformation of large result sets happens in Grafana and can be slow, so push work into the query when the data source can do it.
Key point: Adding the caveat 'push work to the query when you can, transform in Grafana when you can't' shows judgment, not just feature knowledge.
You select the special 'Mixed' data source on the panel, which lets each query target a different data source. So one graph can plot a Prometheus metric alongside a value pulled from a SQL database, useful for overlaying business data on operational data.
To actually combine the numbers (not just overlay two series), you add a transformation like an outer join on time or a shared field. Mixed data sources plus transformations are how you answer 'show orders per minute next to error rate', where the two live in different systems.
Key point: Knowing the panel-level 'Mixed' data source plus a join transformation is the concrete mechanism. Hand-waving 'you just add both' misses how it works.
Loki is Grafana's log aggregation system. Its trick is that it indexes only labels (like app, namespace, level), not the full log text, which makes it far cheaper to run than a full-text engine. You query it with LogQL from a Grafana logs or time series panel.
LogQL looks like PromQL by design: you select streams by label with {app="web"}, then filter lines with |= or |~, and you can even turn matching log lines into metrics with rate() over a range. The mental bridge: PromQL queries numbers, LogQL queries log streams but shares the label-selector and rate() ideas.
# select a stream and filter for errors
{app="web", env="prod"} |= "error"
# turn matching lines into a per-second error rate
sum(rate({app="web"} |= "error" [5m]))Key point: Explaining that Loki indexes labels not full text, and that LogQL mirrors PromQL, shows real familiarity beyond 'it's for logs'.
Watch a deeper explanation
Video: Meet Grafana Loki, a Log Aggregation System for Everything (Techno Tim, YouTube)
Annotations are markers overlaid on a graph at a point or range in time, usually with a note. They let you line up 'what changed' with 'what the metrics did', for example a vertical line where a deploy happened, so a latency spike right after it is obvious.
You can add them manually by drawing on a graph, or automatically from a query (deploys from a table, or Prometheus alerts). Annotation queries pull events from a data source and stamp them on panels, which is how teams correlate releases, incidents, and config changes with metric movements.
Key point: Giving the 'deploy marker next to a latency spike' example shows you use annotations to correlate change with impact, which is their real value.
Grafana OSS has basic roles: Viewer, Editor, and Admin, assigned per organization, plus folder and dashboard level permissions so a team can edit its own folder but only view others. That covers most needs: control who sees and edits what by folder.
Grafana Enterprise adds fine-grained role-based access control (RBAC) with custom roles and per-resource permissions, and data source permissions so users can query only certain sources. Auth itself usually comes from an external provider (LDAP, OAuth, SAML) that maps into these roles.
Key point: Distinguishing the OSS Viewer/Editor/Admin plus folder permissions from Enterprise fine-grained RBAC shows you know where the free line sits, a frequent follow-up.
First find where the time goes: usually it's the queries. Too many panels, each firing a heavy query over a long range, hammers the data source. Reduce the panel count, widen the query step/interval so you fetch fewer points, and avoid high-cardinality queries that return thousands of series.
Then structural fixes: use recording rules in Prometheus to pre-compute expensive aggregations so the panel reads a cheap pre-made series, cache where possible, and lazy-load panels below the fold. Also check the refresh interval isn't so aggressive that the dashboard re-queries constantly. The theme: less work per query, fewer queries, cheaper data to query.
Key point: Naming recording rules and high cardinality as culprits is what proves you've debugged a real slow dashboard, not just guessed 'add more CPU'.
Recording rules live in Prometheus, not Grafana. They pre-compute an expensive query on a schedule and save the result as a new metric. So instead of Grafana recalculating a heavy aggregation every refresh, it reads the already-computed series, which is fast and cheap.
They matter for dashboards that aggregate a lot, a percentile across hundreds of instances, or a fleet-wide sum, where the raw query is slow. You define the rule once in Prometheus, point the panel at the recorded metric, and the dashboard loads quickly. It's the standard fix for a slow, aggregation-heavy panel.
# prometheus recording rule
groups:
- name: api
rules:
- record: job:http_request_rate:sum
expr: sum(rate(http_requests_total[5m])) by (job)Key point: Knowing recording rules live in Prometheus (not Grafana) and why they speed dashboards up is a clean signal you understand the whole stack, not just the UI.
Grafana doesn't collect data; the backends do. Prometheus scrapes metrics endpoints your apps expose (a /metrics page in the Prometheus text format), pulling on an interval. For logs, an agent like Promtail or Grafana Alloy ships log lines with labels into Loki.
So the full pipeline is: your app exposes metrics and writes logs, collectors scrape or ship them into Prometheus and Loki, and Grafana queries those stores to draw dashboards and evaluate alerts. Knowing this end-to-end flow matters, because a lot of 'why is my panel empty' problems are collection problems, not Grafana problems.
Key point: Being able to trace the whole pipeline (app to collector to store to Grafana) is what lets you debug an empty panel, which is a favorite scenario follow-up.
You use data links and dashboard links. A data link on a panel turns a series or point into a clickable link, often to another dashboard, passing the current variables and time range along, so clicking a spiky service jumps you to that service's detailed dashboard, already scoped.
Dashboard links (in settings) add a menu of related dashboards at the top. Together they build a navigation flow: an overview dashboard where clicking a problem drills into detail, carrying context. It turns a pile of separate dashboards into a coherent investigation path.
Key point: data links carry variables and the time range into the target dashboard.
Explore is an ad-hoc query view: one data source, a query editor, and results, with no panels to build or save. It's made for investigation, you're chasing a problem and want to iterate on queries quickly, not build something permanent.
Dashboards are for the queries you want to keep and watch repeatedly; Explore is for the one-off 'what's happening right now' digging. Explore also makes it easy to jump from a metric to related logs (metrics to logs correlation) when the data sources are linked, which speeds up incident debugging.
Key point: Framing Explore as 'ad-hoc investigation' versus dashboards as 'saved and repeated' is the distinction, and the metrics-to-logs jump is a nice detail to add.
Latency is usually exposed as a Prometheus histogram, a set of buckets like http_request_duration_seconds_bucket that count requests falling under each latency threshold. To get a percentile you feed the rate of those buckets into histogram_quantile(), which gives you an approximate p95 or p99 you can graph in a panel. It's approximate because it interpolates within whichever bucket the percentile lands in.
The reason you use percentiles rather than an average is that averages hide the tail: a p99 tells you what your slowest 1% of users experience, which is where real pain lives. Interviewers like this because it mixes a real PromQL pattern with the judgment of why percentiles beat averages for latency.
# approximate p95 request latency
histogram_quantile(
0.95,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)Key point: Explaining why p99 beats an average (averages hide the tail) alongside the histogram_quantile query is the combination that works well.
Chained variables let one dropdown filter the options of the next. You define a $namespace variable, then a $pod variable whose query references $namespace, so picking a namespace narrows the pod list to just that namespace's pods. Grafana re-runs the dependent variable's query whenever its parent changes.
This keeps dashboards usable at scale: without chaining, a $pod dropdown would list every pod in the cluster, which is unworkable. The trade-off is more query load, since changing the top variable cascades re-queries down the chain, so you keep the chain shallow and the variable queries cheap.
# parent variable
label_values(kube_pod_info, namespace)
# dependent variable: filtered by the parent
label_values(kube_pod_info{namespace="$namespace"}, pod)Key point: Knowing that a child variable's query references the parent, and that this cascades re-queries, shows you've built large real dashboards, not toy ones.
advanced rounds probe architecture, scaling, security, and production scars. Expect every answer here to draw a follow-up about trade-offs and failure modes.
Run multiple stateless Grafana instances behind a load balancer, all pointing at a shared external database (MySQL or PostgreSQL) for dashboards, users, and settings. The default embedded SQLite can't back an HA setup because each instance would have its own file, so the external DB is the key move.
Beyond that, you make sessions and alerting HA-safe: alerting in unified mode coordinates so the same alert isn't sent by every replica, and you front it with a proper reverse proxy for TLS and the root URL. Grafana itself scales horizontally easily because it's a read layer; the harder scaling questions are usually about the data sources behind it.
Key point: Leading with 'external database instead of SQLite' is the answer that separates people who've operated HA Grafana from people who've only run the single-container demo.
The naive risk is duplicate notifications: if every replica evaluates the same rule and sends independently, you get one page per replica. Grafana's unified alerting solves this with a coordination layer (built on an Alertmanager-style component) so evaluation and notification are deduplicated across the cluster.
You configure the instances to form a cluster (a shared HA setting and peer addresses) so they share alert state, silences, and notification log. Without that, silences set on one node wouldn't apply on another either. The senior point: HA alerting isn't automatic just because you added replicas, you have to cluster the alerting layer.
Key point: Knowing that duplicate alerts are the failure and that the alerting layer must be explicitly clustered is a strong signal of production Grafana experience.
Cardinality is the number of unique label combinations for a metric. Every distinct set of label values is a separate time series, so putting a high-variety value in a label (user ID, request ID, full URL) explodes the series count, which blows up Prometheus memory and storage and makes Grafana queries slow.
The fix is discipline at the source: labels should be bounded and meaningful (status code, method, service), never unbounded identifiers. Push high-cardinality detail into logs or traces where it belongs. When it's already happened, you find the offending metric, drop or relabel the bad label at scrape time, and educate whoever added it. This is one of the most common real-world monitoring failures.
Key point: Being able to name unbounded labels as the cause and 'put it in logs, not labels' as the fix marks you as someone who's felt a cardinality blowup, not just read about it.
At scale you stop hand-building dashboards and generate them from code: Jsonnet with Grafonnet, or the Terraform Grafana provider, or committed JSON applied via provisioning. This gives version control, code review, reuse (a library panel used across dozens of dashboards), and identical dashboards in every environment.
The trade-off is a steeper authoring path and the discipline that UI edits are throwaway, since the next deploy overwrites them, so people have to change the code, not the UI. Teams often keep a sandbox folder for exploration and lock the provisioned folders. The win is consistency and reproducibility; the cost is that quick tweaks now go through a pipeline.
Dashboards-as-code delivery flow
Once dashboards are provisioned from code, hand-edits in the UI are temporary: the next deploy reverts them, which is the point, config stays in Git.
Key point: Acknowledging the downside (UI edits get overwritten, slower quick tweaks) alongside the wins is what makes this The production-ready answer rather than a tool pitch.
auth and access: disable anonymous access unless you truly want a public dashboard, integrate SSO (OAuth, SAML, LDAP) so accounts map to real identities, enforce least-privilege roles, and put TLS in front comes first. Rotate the admin password off the default and store data source credentials as secrets, not in plaintext provisioning files.
Then reduce blast radius: scope data source permissions so users query only what they should, keep Grafana behind a reverse proxy rather than exposed directly, patch it (it's had CVEs, including path-traversal issues, so version currency matters), and audit who can edit provisioning. The mindset is that Grafana can read a lot of sensitive operational data, so treat it as a system that needs the same care as production.
Key point: Grafana has had real CVEs and that patching plus SSO plus scoped data source access matter.
Prometheus is single-node and keeps limited local retention, so at scale you add a long-term, horizontally scalable metrics store: Grafana Mimir, Thanos, or Cortex. They give you long retention, high availability, and a global query view across many Prometheus servers, all still speaking PromQL so Grafana connects the same way.
The pattern is Prometheus servers scrape and remote-write into the scalable backend, and Grafana queries that backend instead of individual Prometheis. Mimir and Thanos differ in architecture (Thanos leans on object storage sidecars, Mimir is a more integrated microservices system), but both solve the same problem: Prometheus doesn't scale out or retain long-term on its own.
Key point: Naming Mimir, Thanos, or Cortex and the remote-write pattern shows you understand where Grafana's data layer goes when a single Prometheus runs out, a common senior probe.
The three pillars are metrics, logs, and traces, and Grafana is the single pane over all of them: Prometheus (or Mimir) for metrics, Loki for logs, and Tempo for traces, each a data source in one Grafana. That's the point of the Grafana stack, one UI to correlate across pillars.
The value is the jumps between them: from a metric spike in a graph, click to the logs for that service at that time, then from a log line's trace ID open the trace in Tempo to see exactly which span was slow. Wiring those correlations (exemplars from metrics to traces, derived fields from logs to traces) is what turns three separate tools into actual observability.
Key point: Describing the correlation jumps (metric to log to trace) rather than just listing Prometheus, Loki, and Tempo is what shows you understand observability, not just the product names.
Instead of alerting on a static number like 'CPU over 80%', you define an SLO (say 99.9% of requests succeed) and alert on the error budget burn rate: how fast you're consuming the allowed failure. A fast burn (budget gone in an hour) pages immediately; a slow burn (budget will last days but is trending bad) opens a ticket.
In Grafana this means multi-window, multi-burn-rate alert rules: a short window catches sudden outages, a long window catches slow degradations, and both must breach to fire, which cuts false pages. The result is alerts that map to user impact and to how urgently a human needs to act, which is far less noisy than raw resource thresholds.
Key point: Bringing up multi-window burn-rate alerting tied to an error budget is the senior SLO answer. Reciting 'alert when it's over X' misses what the question is really testing.
Work from the panel back toward the source. First check the time range and any variables, an empty result is often just the wrong window or a variable with no match. Then open the query in Explore against that data source: does the raw query return anything? If not, the problem is upstream of the panel.
If the query is empty, verify the metric or log stream actually exists (label names, typos) and that the data source connection is healthy. Then go to collection: is Prometheus scraping the target (check its targets page), is the exporter up, is the app exposing /metrics? 'No data' is usually a collection or query problem, rarely a Grafana bug, so the skill is knowing where along the pipeline to look.
Key point: A structured walk from panel to query to data source to collection is the answer. Guessing one cause without a method is what weaker candidates do here.
The lightweight approach uses folders and teams within one organization: each tenant gets folders they can access, and data source permissions (Enterprise) scope which sources they can query. This works when tenants are teams inside the same company who mostly trust each other.
For harder isolation, Grafana organizations give fully separate spaces (own users, data sources, dashboards), and at the data layer, backends like Mimir and Loki support real multi-tenancy with per-tenant isolation via a tenant header. The senior point: dashboard-level separation in Grafana is different from data-level isolation in the backend, and true multi-tenancy usually needs both.
Key point: Separating 'folder/org isolation in Grafana' from 'tenant isolation in Mimir/Loki' shows you understand that real multi-tenancy is a data-layer concern, not just a UI one.
Grafana is extended through plugins, and there are three main kinds. Data source plugins add a new backend to query (a new database or API). Panel plugins add new visualization types beyond the built-ins. App plugins bundle dashboards, data sources, and custom pages into an integrated experience.
Data source plugins can have a backend component (running server-side, needed for alerting and for auth Grafana must do securely) plus a frontend query editor. Some plugins are signed and official, some are community, and Enterprise data source plugins are paid. Knowing the three types and that data sources can be backend-plus-frontend is the depth interviewers look for.
Key point: Naming the three plugin types and that data source plugins can have a server-side backend (required for alerting) signals you've gone past using Grafana into extending it.
Frame it as build-and-operate versus buy. The Grafana stack (Grafana plus Prometheus, Loki, Tempo, or Grafana Cloud) is open source and avoids per-host billing, but you run and scale the backends yourself, which is real operational work. A vendor like Datadog bundles collection, storage, and dashboards with less setup, at a cost that grows with hosts and data volume and brings lock-in.
The decision follows the team: a platform team that can operate the stack often prefers Grafana for cost control and flexibility at scale; a small team without ops bandwidth may prefer paying a vendor to run it. Grafana Cloud sits in the middle, the Grafana experience as a managed service. The mature answer weighs operational capacity and cost trajectory, not just feature checklists.
Key point: Framing it as 'operational capacity and cost trajectory', and naming Grafana Cloud as the middle path, beats a feature-by-feature list. This question screens for real trade-off thinking.
Almost always the data source, not Grafana. Grafana is a thin read layer; when dashboards crawl under load it's because the backend is answering many expensive queries: high-cardinality series, long ranges, and heavy aggregations recomputed on every refresh across many concurrent viewers.
So the fixes live in the data layer: recording rules to precompute aggregations, a scalable backend (Mimir, Thanos) with query sharding and caching, tighter query intervals, and lower cardinality. On the Grafana side you trim panel counts and refresh rates, but if you only scale Grafana replicas and ignore the backend, the dashboards stay slow. Diagnosing that the data source is the constraint is the senior instinct.
Key point: Saying 'scaling Grafana replicas won't fix a slow data source' is the insight. Candidates who reach for more Grafana CPU instead of fixing queries miss the architecture.
Watch a deeper explanation
Video: Monitoring Made Easy with Grafana and Prometheus (Better Stack, YouTube)
Validate the query first in Explore against real data so you know the condition matches what you intend, then use the rule preview to see what it would fire on for recent data. Set a sensible 'for' duration and route it to a test contact point (a dev Slack channel) before it ever reaches PagerDuty.
Beyond one rule, treat alerting as code: keep rules in Git, review them, and where possible run them against recorded metric data in CI to catch obvious mistakes. Also plan the NoData and Error behavior deliberately, an alert that goes silent when the query breaks is worse than a noisy one. The goal is that the first time a rule pages, it's already been proven to fire correctly.
Key point: rule preview, a test contact point, and deliberate NoData handling matters.
Tempo is Grafana's distributed tracing backend, the third data source alongside Prometheus for metrics and Loki for logs. It stores traces cheaply by keeping only trace IDs indexed and putting the trace data itself in object storage, so you retrieve a full trace by its ID rather than searching heavily indexed fields.
It matters because a trace shows one request's path across every service, which metrics and logs alone can't. The real payoff in Grafana is the links: exemplars jump from a latency metric to an example trace, and derived fields jump from a log line's trace ID into Tempo. That metric-to-log-to-trace flow inside one Grafana is what turns three tools into actual debugging.
Key point: Naming Tempo's ID-based, object-storage design and the exemplar/derived-field links to metrics and logs shows you understand the full Grafana observability stack, not just dashboards.
Grafana's strength is being data-source agnostic: one tool that visualizes Prometheus, Loki, SQL, and cloud metrics side by side. Kibana is tied tightly to Elasticsearch and shines at log search and exploration inside that stack. Chronograf is the visualization piece of the InfluxDB (TICK) stack and centers on InfluxDB. Vendor dashboards (Datadog, New Relic, CloudWatch) bundle collection, storage, and visualization into one paid product with less setup but more lock-in. Knowing where each fits, and saying the trade-off out loud, is itself an interview signal.
| Tool | Best at | Data source model | Watch out for |
|---|---|---|---|
| Grafana | Unified dashboards across many backends | Agnostic: plugins for Prometheus, Loki, SQL, cloud | You still run and manage the data sources yourself |
| Kibana | Log search and exploration in the Elastic stack | Elasticsearch/OpenSearch centric | Weak outside Elastic; not a general dashboard tool |
| Chronograf | Visualizing InfluxDB time-series | InfluxDB (TICK stack) centric | Narrow ecosystem; largely folded into InfluxDB UI |
| Vendor SaaS (Datadog, New Relic) | All-in-one collection plus dashboards, fast setup | Their own agent and storage | Cost at scale and vendor lock-in |
Prepare in layers, and trade-offs out loud is the explanation path. Most monitoring rounds move from concept questions to a hands-on task (build a panel, write a query, fix an alert) to a scenario discussion, so rehearse each stage rather than only reading answers.
The typical Grafana interview flow
Earlier rounds increasingly run as AI-driven technical interviews. The concepts on this page are what gets probed at every stage.
10 questions, about 6 minutes. Score 70% or higher to earn a shareable certificate.
Hyring builds the AI Coding Interviewer that runs live technical rounds in 20+ languages for 5,000+ hiring teams. These Grafana questions and scoring notes reflect what actually gets asked and evaluated inside the interviews we host.
See how the AI Coding Interviewer works