Hands-on Labs · Hands-on lab

Build an Enterprise Prometheus and Grafana Observability Platform

Deploy and operate Prometheus and Grafana across data-center VMs, Kubernetes, Azure, AWS, and Google Cloud, with practical collection, storage, identity, dashboard, alerting, and recovery guidance.

Build a disposable metrics platform with Prometheus, Grafana Mimir, and Grafana, then use the deployment profiles to plan and deploy the same collection model on data-center VMs, Kubernetes, Azure, AWS, and Google Cloud. Collect a host signal, test remote write, create dashboards and alerts, and verify the security and operating limits of each profile.

Document class: Guided implementation lab. Owner: Cloud Center of Excellence. Evidence: Release manifest, collection configuration, PromQL queries, dashboard, alert test, failure exercise, and cleanup record. Validation boundary: The lab instructions and index are machine-validated. Learners must record actual container, integration, alert, and recovery results. This lab does not certify a production deployment.

Lab overview

Scenario and learning objectives

A platform team needs one operating view for workloads in a data center and three cloud providers. The lab builds a small, isolated environment that demonstrates the portable Prometheus metrics path and Grafana operations. It uses Grafana Mimir's official Docker Compose tutorial stack as the disposable backend, including its local object-storage emulator and three Mimir instances. This is a learning setup, not a production topology.

By completing this lab, you will:

  1. Deploy and inspect a multi-instance Mimir tutorial stack and its Grafana data source.
  2. Send Prometheus metrics to Mimir using authenticated-tenant routing in the lab configuration.
  3. Add a data-center-style exporter target and attach bounded location and environment labels.
  4. Build a Grafana dashboard and an actionable alert from the collected series.
  5. Observe collection gaps and recover from a single Mimir instance failure.
  6. Record which additional steps are required to connect provider-native service metrics safely.
  7. Select and deploy a self-managed VM or Kubernetes profile, or a provider-managed metrics profile.
  8. Configure identity, retention, dashboards, alerting, and recovery controls for the selected environment.

The local exporter is a containerized stand-in. It does not represent a real server, hypervisor, network device, managed cloud resource, or provider API. The optional cloud extension requires real provider accounts and must use read-only, short-lived identities.

Alignment with the engineering standard

Controls in SBP-10Lab implementationEvidence
Common names, labels, and ownershipAdd controlled environment, cloud, site, and service labelsPrometheus target labels and Grafana query result
Metric and alert qualityQuery up, build a service view, and test an outage conditionDashboard panel and recorded alert state
Collection resilienceStop and recover one backend instance and one exporterup series, Mimir query, and recovery notes
Least-privilege data accessKeep this local lab credential-free; specify read-only identities for cloud extensionsAccess plan and no secrets in the lab files
Retention, privacy, and costInspect demo storage and document production requirementsLab boundary and production-readiness checklist

These exercises cover selected controls. They do not verify every provider integration, a production identity design, regional recovery, or a complete telemetry standard.

Prerequisites

Workstation and access

ResourceRequirement
WorkstationLinux or WSL2 with Bash; Docker Engine and Docker Compose v2; Git; curl; and a browser
Capacity4 CPU cores, 8 GiB RAM, and 10 GiB free disk for images and disposable data
NetworkOutbound access to GitHub and container registries; host ports 9000 and 9009 available
Cloud accountsNot required for the core lab; optional Azure, AWS, and Google Cloud sandbox access for native data-source extensions
PermissionsPermission to run Docker containers and remove the lab's containers, network, and volumes

Do not run the lab on a production Docker host. Keep the ports bound to a trusted workstation. The tutorial's local HTTP endpoints, demo tenant, MinIO storage, and sample credentials are intentionally for local learning only.

Version and release record

Use the Grafana Mimir mimir-3.2.2 release source as the tutorial baseline and prom/node-exporter:v1.12.1 for the sample exporter. The Compose file from the pinned Mimir source defines the companion Prometheus, Grafana, MinIO, and load-balancer images. Record the Git commit and resolved image digests before starting:

git clone --depth 1 --branch mimir-3.2.2 https://github.com/grafana/mimir.git
MIMIR_ROOT="$(pwd)/mimir"
cd "$MIMIR_ROOT/docs/sources/mimir/get-started/play-with-grafana-mimir"
git -C "$MIMIR_ROOT" rev-parse HEAD
docker compose config --images

The last Git command uses the saved repository path to resolve the checked-out Mimir source commit. Save its output and the image list with the lab evidence. Resolve and record image digests with docker image inspect after pulling. If a tag or directory has changed upstream, stop and select a supported release deliberately; do not silently switch to main or latest.

Target architecture

flowchart LR subgraph workstation[Disposable workstation lab] EXP[Node exporter<br/>synthetic host view] PROM[Prometheus<br/>scrape and remote write] subgraph backend[Compose Mimir tutorial stack] LB[NGINX load balancer] M1[Mimir 1] M2[Mimir 2] M3[Mimir 3] OBJ[(MinIO demo storage)] LB --> M1 LB --> M2 LB --> M3 M1 --> OBJ M2 --> OBJ M3 --> OBJ end GRAF[Grafana<br/>Prometheus data source] EXP -->|HTTP scrape| PROM PROM -->|remote write, tenant demo| LB GRAF -->|PromQL query| LB end

The containers share a private Docker Compose network. Only the tutorial's Grafana and Mimir gateway ports are published to the workstation. The demo environment does not include production TLS, identity, private networking, or durable object storage.

Lab modules

Module 1: Start the pinned Mimir tutorial stack

From the tutorial directory, inspect the Compose configuration and start the stack:

docker compose config --services
docker compose up -d
docker compose ps

Wait until the Grafana, Mimir, MinIO, Prometheus, and load-balancer containers are healthy or running. Open http://localhost:9000 and sign in using the tutorial credentials documented in that pinned release's configuration. Change any default local password if the UI prompts you. Do not reuse lab credentials elsewhere.

Open Explore and choose the preconfigured Mimir data source. Query up and confirm the Mimir targets return samples. Open the included Mimir write and read dashboards; empty panels for components the tutorial does not deploy are expected. Record the query time range, sample labels, and any startup delay.

Module 2: Add a data-center-style scrape target

Start a node exporter on the tutorial's Compose network. It reports its own container environment; this lets the lab exercise the scrape path without mounting or exposing the learner's host filesystem.

PROMETHEUS_CONTAINER="$(docker compose ps -q prometheus)"
COMPOSE_NETWORK="$(docker inspect --format '{{range $name, $network := .NetworkSettings.Networks}}{{$name}}{{end}}' "$PROMETHEUS_CONTAINER")"
test -n "$COMPOSE_NETWORK"
docker run -d --name lab-node-exporter --network "$COMPOSE_NETWORK" prom/node-exporter:v1.12.1

Edit config/prometheus.yaml in the tutorial directory and add this job under scrape_configs:

  - job_name: lab/datacenter-node
    static_configs:
      - targets: ["lab-node-exporter:9100"]
        labels:
          environment: lab
          cloud: onprem
          site: dc-lab
          service: node-exporter

Restart the tutorial Prometheus service so it reads the updated configuration:

docker compose restart prometheus

Open the Prometheus targets page at http://localhost:9090/targets if that port is published in the pinned Compose file. Otherwise inspect the Prometheus container logs and query the target from Grafana. In Explore, run:

up{job="lab/datacenter-node"}

The result should be 1 and include the four lab labels. Also query node_memory_MemAvailable_bytes{site="dc-lab"}. Treat these values as exporter-container metrics, not as evidence about the physical workstation or a production data center.

Module 3: Build a service dashboard

In Grafana, create a folder named Lab - Hybrid Observability and a dashboard named Service and Collection Health. Add these panels using the Mimir data source:

PanelPromQLPurpose
Scrape statusup{job="lab/datacenter-node"}Shows whether the sample target can be scraped
Available memorynode_memory_MemAvailable_bytes{site="dc-lab"}Shows one host-level metric family
Healthy targets by jobsum by (job, cloud, site) (up)Groups collection health by source and job
Scrape failuressum by (job, site) (up == 0)Surfaces unreachable scrape targets

Set a five-minute time range, use a clear unit for the memory panel, and add a description explaining the synthetic target. Save the dashboard as JSON or record a screenshot and note its owner. In production, provision dashboard definitions and data sources through source control rather than leaving the UI as the only copy.

Module 4: Test an alert and recover

Create a Grafana-managed alert rule against the Mimir data source:

  • Query: up{job="lab/datacenter-node"}.
  • Condition: below 1 for one minute.
  • Labels: environment=lab, cloud=onprem, site=dc-lab, and an assigned lab owner.
  • Annotations: explain that the data-center-style exporter cannot be scraped and link to the lab runbook.
  • Contact point: use a local test contact point or Grafana's preview/test capability; do not send real notifications.

Stop the sample exporter and wait for the rule's pending period:

docker stop lab-node-exporter

Confirm that the target's up value becomes 0 and the alert becomes pending or firing according to the configured evaluation interval. Start the exporter and confirm that the alert resolves:

docker start lab-node-exporter

Next, stop one Mimir instance with docker compose stop mimir-1. Query up and record whether ingestion and reads continue through the tutorial load balancer. Start it again with docker compose start mimir-1, wait for recovery, and confirm that samples resume. This is a small failure exercise, not a regional disaster-recovery test.

Module 5: Map real provider integrations

Do this module only if approved sandbox accounts are available. Keep the local stack as the Prometheus-compatible workload data source. For each provider, use a separate read-only Grafana data source identity and record the exact account, subscription, or project scope:

ProviderGrafana data sourceMinimum lab scope
AzureAzure Monitor; optionally Azure Monitor managed service for PrometheusOne disposable resource group and an Azure identity with metrics read access
AWSCloudWatch; optionally Amazon Managed Service for PrometheusOne sandbox account and a role with only the required metrics query permissions
Google CloudGoogle Cloud Monitoring; optionally Google Cloud Managed Service for PrometheusOne sandbox project and a dedicated monitoring viewer identity

Use provider documentation for the current authentication and data-source setup; requirements vary by Grafana version and hosting location. Do not add access keys to Compose, Prometheus YAML, dashboards, or Git. Query one representative VM or managed service and compare its native metric names, dimensions, freshness, and units with the portable workload metrics. The extension is complete when access is revoked successfully and the data source stops returning protected metrics.

This local lab does not claim that a native provider data source, cloud identity, exporter, API quota, or production network path was tested. Record the provider, service, signal, collection path, identity, dashboard, alert owner, and known blind spots in the service inventory before any rollout.

Deployment guide for different environments

The core Compose exercise above is the smallest reproducible path. Use the profiles below to extend it to the platform that owns your workloads. You do not have to install a Prometheus server in every environment: managed collection and storage reduce operations work, while self-managed Prometheus provides local query and retention control. Keep one collection contract and select a backend per security, locality, retention, and cost requirements.

Choose the collection and storage model

EnvironmentCollector and storeGrafana optionTypical use
Data-center VMs and appliancesPrometheus plus exporters; remote-write selected metrics to Mimir or another compatible long-term storeSelf-managed Grafana on a private VM or Kubernetes serviceLocal operations, restricted WAN links, or systems without cloud-native agents
Kubernetes in any cloud or on-premiseskube-prometheus-stack for self-managed operation, or the provider's managed Prometheus collectorGrafana in-cluster, centrally hosted, or provider-managedCluster, node, pod, and application metrics
AzureAzure Monitor managed service for Prometheus for AKS and remote-write sources; Azure Monitor for native Azure resource metricsAzure Managed Grafana or self-managed GrafanaAzure-first operations with Entra identity and Azure Monitor integration
AWSAmazon Managed Service for Prometheus (AMP), using managed scrapers or SigV4 remote write; CloudWatch for native AWS metricsAmazon Managed Grafana (AMG) or self-managed GrafanaAWS-first operations and EKS monitoring
Google CloudGoogle Cloud Managed Service for Prometheus (GMP), using managed collection, the Ops Agent, or self-deployed collection; Cloud Monitoring for native metricsSelf-managed Grafana or a Grafana service with Google Cloud Monitoring data sourceGKE, Compute Engine, and cross-cloud Prometheus collection
Hybrid and multicloudEdge collectors scrape near workloads and send approved common metrics to a shared Mimir-compatible backend; each cloud's managed store remains available for native metricsA centrally governed Grafana with one data source per backendOne service view without pretending every provider metric has the same schema

Choose one authoritative source for each metric family. A Kubernetes node metric can flow through a Prometheus-compatible backend, while a managed database's provider metric can stay in Azure Monitor, CloudWatch, or Cloud Monitoring. Adding both routes for the same signal creates duplicate series, alert noise, and extra cost.

Establish a common scrape and label contract

Keep scrape intervals, job names, ownership, and location labels consistent. Use stable, bounded labels such as environment, cloud, region, site, cluster, namespace, service, and team. Do not put user IDs, request IDs, email addresses, full URLs, or other high-cardinality or personal values in labels. A label value creates a new time series for each combination of labels.

Use this portable starting point in prometheus.yml or the equivalent chart values. Replace example targets with internal DNS names reachable from the collector:

global:
  scrape_interval: 30s
  evaluation_interval: 30s
  external_labels:
    environment: lab
    site: dc-lab
    cluster: observability-lab

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]
        labels:
          service: prometheus

  - job_name: linux-node
    static_configs:
      - targets: ["node-exporter-01.internal.example:9100"]
        labels:
          cloud: onprem
          service: linux
          team: platform

  - job_name: application
    metrics_path: /metrics
    static_configs:
      - targets: ["orders-api.internal.example:8080"]
        labels:
          cloud: onprem
          service: orders-api
          team: commerce

The exporter and application endpoints should be reachable only from authorized collectors. For Kubernetes, prefer ServiceMonitor, PodMonitor, or the provider's scrape custom resource instead of maintaining a large static target list. Add remote_write only after you have the target backend URL, TLS requirements, and a supported workload identity or secret-provider design. Keep credentials out of the configuration repository.

Profile A: data-center Linux VMs and appliances

For a small site or a restricted network zone, run Prometheus and Grafana on dedicated Linux VMs. Install signed, pinned Prometheus and Grafana releases from their official sources, verify checksums, create separate non-login service accounts, and place the Prometheus TSDB on local block storage. Keep exporters on the hosts they measure and allow inbound exporter ports only from the Prometheus collector addresses.

  1. Reserve private DNS names and static addresses for the collector and Grafana. Decide if one collector per site is needed to survive WAN loss.
  2. Install a reviewed Prometheus release and promtool; store /etc/prometheus/prometheus.yml and alert rules under configuration management.
  3. Create /var/lib/prometheus owned only by the Prometheus service account. Do not use NFS or a shared network filesystem for the local TSDB.
  4. Create a systemd unit with --config.file=/etc/prometheus/prometheus.yml, --storage.tsdb.path=/var/lib/prometheus, a deliberate retention time and size, and --web.listen-address=127.0.0.1:9090 when Grafana runs on the same host. If Grafana is remote, use a private interface, TLS, authenticated reverse proxy, and a firewall allowlist.
  5. Install Grafana from its official package repository on a separate VM or approved shared service. Configure the Prometheus data source with the private Prometheus URL; keep Grafana's listener behind the enterprise identity-aware ingress.
  6. Run promtool check config /etc/prometheus/prometheus.yml and promtool check rules /etc/prometheus/rules/*.yml before restart. Enable both services at boot and verify their service health after a host reboot.
  7. For long retention or a global view, remote-write selected metrics to Mimir or a provider backend. Keep local retention sufficient for site operations during WAN or backend outages.

Example systemd unit for the local Prometheus service (adapt retention and binding to your capacity and network design):

[Unit]
Description=Prometheus monitoring server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=40GB \
  --web.listen-address=127.0.0.1:9090
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

After installing the verified prometheus and promtool binaries and creating the configuration and data directories, save this as /etc/systemd/system/prometheus.service, then run sudo systemctl daemon-reload and sudo systemctl enable --now prometheus. If Grafana is on another host, change the listener to an approved private address and enforce TLS/authentication at the network or proxy boundary.

Linux exporter example targets include node exporter, Blackbox Exporter for HTTP/DNS/TCP checks, and SNMP Exporter for network devices. For Windows use windows_exporter. Database and middleware exporters should use dedicated read-only accounts and expose only required collectors. Confirm each exporter supports TLS/authentication or put it behind a restricted network path. Do not scrape a management endpoint from the public internet.

Profile B: Kubernetes on-premises, AKS, EKS, or GKE

For a self-managed Kubernetes stack, kube-prometheus-stack deploys Prometheus Operator, Prometheus, Alertmanager, exporters, kube-state-metrics, rules, and Grafana. It works across Kubernetes distributions; the cloud is not the control plane for this installation.

  1. Create a dedicated monitoring namespace and an approved StorageClass backed by durable block storage. Use a private cluster endpoint or an internal ingress with TLS and organizational SSO.
  2. Review the chart's version, values, CRDs, container images, and change notes. Pin the Helm chart version and every production image. Do not install main or latest in a controlled environment.
  3. Create a values.yaml file with persistent volumes and retention bounds. Example values to adapt:
grafana:
  persistence:
    enabled: true
    size: 10Gi
  service:
    type: ClusterIP

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 40GB
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: <approved-block-storage-class>
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 50Gi
  1. Inspect the exact chart package and install the pinned version:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm show chart prometheus-community/kube-prometheus-stack --version "$CHART_VERSION"
helm upgrade --install observability prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --version "$CHART_VERSION" --values values.yaml
kubectl -n monitoring get pods,pvc,services

Set CHART_VERSION to the version reviewed by your platform team. Use kubectl port-forward only for a short local validation; production access should go through the approved private ingress and SSO. Configure application scraping with ServiceMonitor or PodMonitor, reviewed namespace selectors, and a dedicated metrics port. Apply Kubernetes network policies so only the monitoring namespace can reach exporter endpoints.

For a managed Kubernetes monitoring service, choose the cloud's managed collector rather than installing overlapping Prometheus stacks by default. Keep kube-prometheus-stack only if you require local rule evaluation, local dashboards, offline site operation, or a specific Prometheus feature. If managed storage is the destination, configure remote write with its provider-specific identity and track any local queue/backpressure behavior.

Profile C: Azure Monitor managed Prometheus and Azure Managed Grafana

For AKS, create or select an Azure Monitor workspace, then enable managed Prometheus collection and link the cluster to that workspace. Azure CLI example for an existing cluster:

az aks update \
  --name "$AKS_NAME" \
  --resource-group "$AKS_RESOURCE_GROUP" \
  --enable-azure-monitor-metrics \
  --azure-monitor-workspace-resource-id "$AZURE_MONITOR_WORKSPACE_ID" \
  --grafana-resource-id "$AZURE_MANAGED_GRAFANA_ID"

The managed add-on supplies collection for supported cluster targets. Use its documented ConfigMap or scrape custom resources to add application metrics, and enable control-plane metrics separately when required. Add data sources to Azure Managed Grafana and grant its managed identity the least required monitoring-reader permissions at the workspace/resource scope.

For a data-center VM, Arc-enabled server, or another cloud where the self-managed Prometheus collector must remain, configure Prometheus remote_write to the Azure Monitor workspace ingestion endpoint. The workspace's data collection rule provides the endpoint. Assign the Monitoring Metrics Publisher role to the chosen identity on the applicable data collection rule. Use managed identity for supported Azure-hosted collectors, workload identity for supported Kubernetes workloads, or Microsoft Entra authentication for external/on-premises sources. The shape below is illustrative; use the authentication block that matches your host and Prometheus version:

remote_write:
  - url: "<metrics-ingestion-endpoint-from-the-Azure-Monitor-workspace>"
    azuread:
      cloud: AzurePublic
      managed_identity:
        client_id: "<user-assigned-managed-identity-client-id>"

Never paste a client secret into this file. Confirm ingestion by querying a known series in Grafana. Keep Azure resource metrics and logs on their native Azure Monitor paths; they are not interchangeable with Prometheus metrics.

Profile D: Amazon Managed Service for Prometheus and Amazon Managed Grafana

Create an AMP workspace in the Region that will own metric storage. For EKS, an AMP managed scraper can discover and scrape cluster targets inside the permitted VPC scope. For a data-center collector, EC2, or another Kubernetes distribution, run Prometheus or an agent close to the targets and use SigV4 remote write with an IAM role rather than long-lived access keys.

aws amp create-workspace --alias "$AMP_ALIAS" --region "$AWS_REGION"

Record the workspace ID and remote-write endpoint shown by the service. A self-managed Prometheus configuration uses the AMP remote-write URL and SigV4 signing; this is a structural example, not a complete credential policy:

remote_write:
  - url: "https://aps-workspaces.<region>.amazonaws.com/workspaces/<workspace-id>/api/v1/remote_write"
    sigv4:
      region: <region>

Attach an IAM role to the collector's compute identity with only the required AMP remote-write permission. Restrict its network path to the AMP endpoint. In Amazon Managed Grafana, enable access to AMP and CloudWatch data sources through service-managed or customer-managed permissions, add the target AMP workspace from the Grafana data-source workflow, then use Save & test. For multi-account access, scope role assumption to specific accounts and workspaces. Use CloudWatch for AWS service metrics such as EC2, RDS, and load balancers; AMP stores Prometheus-compatible workload and exporter metrics.

Profile E: Google Cloud Managed Service for Prometheus

For GKE, verify whether Managed Service for Prometheus collection is already enabled; it is enabled by default on supported GKE Autopilot and Standard versions. For a cluster that needs it enabled explicitly, use the regional or zonal command that matches the cluster:

gcloud config set project "$GCP_PROJECT_ID"
gcloud container clusters update "$GKE_CLUSTER" \
  --enable-managed-prometheus --region "$GKE_REGION"

Managed collection uses Kubernetes resources such as PodMonitoring and ClusterPodMonitoring to define the scrape target; it does not install application exporters for you.

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: orders-api
  namespace: applications
spec:
  selector:
    matchLabels:
      app: orders-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Apply the resource to the same namespace as the selected application Pods and verify it with kubectl get podmonitoring -A. For VMs running on Compute Engine, configure the Ops Agent Prometheus receiver. For on-premises or other-cloud Kubernetes, install the managed collector according to the current Google documentation, pin the published manifest version, and configure the credentials method supported by that environment. Prefer workload identity federation where supported; if a service-account key is unavoidable, store it as a Kubernetes Secret or external secret and never commit it to Git. Record the Google Cloud project, location, and cluster hierarchy used for metric labels.

In Grafana, configure the Google Cloud Monitoring data source using a workload identity, GCE service account, or approved federation. Grant only Monitoring Viewer for query access and use the data source's PromQL query type for Prometheus-style metrics. Do not distribute service-account key JSON files to users. Google Cloud service metrics remain queryable in Cloud Monitoring; GMP stores Prometheus-compatible workload metrics.

Common dashboard and alert implementation

Keep dashboards and alert rules under version control. Provision the same dashboard definition in each Grafana deployment where practical, and use template variables for environment, cloud, region, cluster, namespace, and service. Keep each query against the intended data source: provider data sources have different metric names, units, dimensions, and refresh periods.

Create an alert rule file for local Prometheus environments and adapt the same intent to managed alerting for provider paths:

groups:
  - name: service-availability
    rules:
      - alert: ServiceTargetMissing
        expr: up{service!="prometheus"} == 0
        for: 5m
        labels:
          severity: warning
          owner: platform
        annotations:
          summary: "A monitored service target is unavailable"
          runbook_url: "https://runbooks.example.invalid/observability/target-down"

Add a separate absent() rule if a missing target should alert; up == 0 alone cannot detect a target that has disappeared from service discovery. Route alerts to an approved contact point, test notification delivery, and name an on-call owner. Do not use the lab contact point for production messages.

For a Grafana installation managed by your team, provision a Prometheus data source like this and supply the endpoint through the deployment's configuration system:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc:9090
    isDefault: true
    editable: false

For managed backends, use their dedicated data-source plugin and identity integration instead of copying a raw bearer token into provisioning files. Provision dashboard definitions with GitOps or your release pipeline, and review dashboard changes like application code.

Security, capacity, and operating controls

AreaRequired design check
NetworkScrape traffic flows from collectors to targets; remote write flows outbound to an approved endpoint; dashboards and query APIs are private or identity-gated. Never expose Prometheus, exporters, or unauthenticated Grafana directly to the internet.
IdentityUse managed/workload identities, IAM roles, or short-lived federation. Separate query, write, and administration permissions. Store any unavoidable credentials in a secret manager and rotate them.
TLSVerify certificates for scrape, remote-write, and query connections. Do not use insecure_skip_verify as a production fix.
DataDo not place secrets, personal information, or unbounded identifiers in metric names or labels. Define tenant, account, subscription, project, and workspace boundaries.
Local storageUse reliable local block storage for Prometheus TSDB. Set both time and size retention; reserve headroom for WAL and compaction. Upstream guidance recommends leaving roughly 15–20% free relative to the retention size limit.
ScaleMeasure active series, samples per second, scrape duration, remote-write queue, query concurrency, and dashboard refresh. Reduce unused labels and metrics before vertically scaling. Shard collection only when measured limits require it.
AvailabilityDefine what continues locally during WAN/backend failure, how long queues can absorb an outage, and what gaps are acceptable. A single Prometheus server is not an HA storage cluster. Use a compatible distributed backend or managed service for shared long retention and multi-tenant operation.
GovernancePin software/chart versions, scan images, review upgrades, own recording/alert rules, and test backup and restore. Track provider API limits, retention, egress, and ingestion costs.

Prometheus local storage is designed for local filesystem access and is not a shared multi-writer database. Do not put its TSDB on NFS or a shared volume. A Prometheus replica pair can improve collector availability, but duplicated series need replica labels and deduplication in the query backend if both replicas remote-write the same samples.

Troubleshooting by symptom

SymptomCheck firstCorrective action
Target is DOWNStatus > Targets, DNS, route/firewall, target port, TLS and auth errorsAllow only the collector source, correct the service port/path, and validate the target certificate and credentials
up exists but application metrics do notEndpoint path, exporter collectors, scrape response, metric relabel rulesCheck /metrics from the collector network and confirm the application exports the expected metric names
Prometheus works but remote backend is emptyRemote-write queue/failures, endpoint path, identity policy, tenant/workspace, provider ingestion limitsCheck collector logs and provider ingestion diagnostics; validate the exact writer identity and backend endpoint
Grafana Save & test failsData-source URL from the Grafana server's network, DNS, TLS, identity roles, plugin typeTest from the Grafana runtime and use the provider-specific data source for managed services
Dashboard has gaps or slow panelsQuery time range, step/interval, series count, backend query limits, refresh intervalNarrow selectors, lower cardinality, use recording rules, and avoid excessive refresh frequency
Disk fills or Prometheus restartsTSDB size, WAL growth, compaction, cardinality, retention settingsReduce unneeded series, set a safe retention size/time, grow local disk, and plan remote storage for longer retention
Alert never firesRule loaded, expression returns a series, evaluation interval, for period, notification policyRun the expression in Explore, validate rules with promtool, and test the route/contact point independently

Rollout from lab to production

  1. Select the deployment profile and name each authoritative source for infrastructure, platform, application, and managed-service metrics.
  2. Inventory targets, exporter owners, metric families, labels, network paths, scrape intervals, and expected cardinality before onboarding.
  3. Establish authentication, TLS, RBAC, tenancy, retention, region, backup, and cost controls before production ingestion.
  4. Pilot one data-center host, one Kubernetes namespace, one application, and one representative native cloud service. Validate dashboards, alerts, missing-data behavior, and recovery.
  5. Load-test query and ingest rates, measure remote-write backlog during an outage, and verify storage and restore assumptions.
  6. Roll out by environment and owner. Record coverage and exceptions; remove duplicate collectors and obsolete dashboards after cutover.
  7. Review alert quality and metric costs after a full operating cycle. Assign owners to every production dashboard and alert.

Evidence and release record

Keep a small evidence bundle containing:

  • Mimir Git commit, container image tags and digests, Docker and Compose versions, and lab date.
  • docker compose ps output before and after the failure exercise.
  • The Prometheus scrape job, successful up and memory queries, and the Grafana dashboard definition or screenshot.
  • Alert condition, owner labels, pending/firing/resolved timestamps, and exporter recovery result.
  • Optional provider data-source scopes and access-revocation result; never include secret material.
  • Cleanup record and any unresolved collection or dashboard gaps.

Cleanup

Remove the extra exporter, then stop and delete the lab containers and disposable volumes from the tutorial directory:

docker rm -f lab-node-exporter
docker compose down -v

Confirm no tutorial containers or network remain with docker compose ps -a and docker network ls. The -v option deletes the tutorial's local demo storage. Preserve only the evidence listed above. If the exporter was not created or has already been removed, docker rm -f may report that it does not exist; verify the final container list before closing the lab.

Validation and acceptance

  • The pinned Compose stack starts and Grafana can query up through the configured Mimir data source.
  • The node exporter target is reachable only on the Compose network and returns up=1 with the expected bounded labels.
  • Dashboard panels show collection health and data freshness with appropriate units and descriptions.
  • The exporter outage produces the expected alert state, and recovery resolves it without changing unrelated services.
  • One Mimir instance can be stopped and restored, with observed read/write behavior recorded.
  • Optional cloud data sources use scoped identities; revoking access prevents further metric queries.
  • No secrets were stored in the lab repository, and all disposable containers, networks, and volumes were removed.

The validation proves the local tutorial workflow only. Before production, complete the architecture acceptance checks in ES-11, including fleet coverage, secure remote write, provider API limits, production retention, scale, tenant isolation, and regional recovery.

Operational boundaries and acceptance criteria

Before adopting this guidance, record the accountable service owner, supported environments, critical dependencies, data classification, and the operational tier. State which decisions belong to the platform team and which remain with the workload team; document any exception with a risk owner and expiry date. This prevents a reference pattern from being treated as a universal fit.

Define observable acceptance checks before rollout. Include a positive path, a denied or invalid request, a dependency failure, and recovery after the failure clears. Capture the configuration or command output that proves each check, along with the software and policy versions used. Make checks repeatable in a representative non-production environment, and ensure they fail closed when required evidence is missing.

For production changes, identify the safe rollback or forward-recovery action, the person authorized to invoke it, and the condition that triggers it. Confirm that monitoring, support ownership, capacity limits, and escalation contacts are in place before declaring the change complete. Review these assumptions after a material platform, provider, or regulatory change.

Extension exercises and evidence quality

After the required path works, add one controlled failure exercise: revoke a test identity, block a dependency, or introduce an invalid configuration in an isolated environment. Predict the expected signal before running the exercise, capture the observed result, and explain any difference. Do not use production credentials or real customer data. Keep a record of image, tool, and provider versions so another learner can reproduce the outcome.

For a useful lab hand-off, include the expected state after each module, common cleanup failures, and a verification command that proves resources are gone. Separate optional extensions from required completion criteria, and note which steps incur billable usage or need an explicit quota increase. Restore modified local configuration and remove temporary credentials at the end; cleanup is complete only after checking the cloud inventory and billing view.

References