Enterprise Solutions · Architecture

Enterprise Hybrid and Multi-Cloud Observability with Prometheus and Grafana

Defines a governed Prometheus and Grafana solution for data centers and Azure, AWS, and GCP, combining portable workload metrics with provider-native service telemetry.

Document class: Enterprise architecture solution Normative terms: MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are requirements levels. Applicability: Infrastructure, platform, application, and service metrics across enterprise data centers and Azure, AWS, and Google Cloud. Exception process: Deviations require a documented risk assessment, compensating controls, named risk owner, and expiry date.

Control fieldValue
Document IDES-11
OwnerCloud Center of Excellence
Version1.0
Review cycleAt least annually and after material Prometheus, Grafana, provider-monitoring, identity, retention, or operating-model changes
EvidenceSource inventory, collector configuration, data-source permissions, metric contract, dashboards, alert tests, capacity model, and recovery exercise

Decision in brief: Use Prometheus-compatible instrumentation and regional Grafana Alloy or Prometheus collectors for portable workload metrics. Send the common metrics plane to a highly available Prometheus-compatible backend such as Grafana Mimir, then use Grafana for dashboards and alerting. Query each cloud provider's native monitoring service for managed-service metrics and provider-specific context.

Purpose

Establish a consistent monitoring experience for services that run in corporate data centers and in Azure, AWS, and Google Cloud. Teams need shared service dashboards, actionable alerts, ownership labels, and common PromQL for metrics that the organization controls, while cloud operations still need the full detail exposed by each provider.

This architecture covers metrics collection, storage, querying, visualization, and metric-based alerting. It does not replace cloud audit logs, security-event pipelines, distributed tracing, synthetic monitoring, or incident-management systems. Connect those systems to the wider observability program and correlate them with metrics as described in How to Build Centralized Multi-Cloud Observability.

Context and decision drivers

Prometheus is well suited to scraping applications and infrastructure that expose Prometheus metrics. Grafana can query Prometheus-compatible backends and provider data sources. Neither product makes every managed cloud service expose the same metric names, dimensions, retention, or query behavior.

The solution therefore uses two complementary data paths:

  1. Portable metrics path: Applications, hosts, Kubernetes, network appliances, and selected exporters are collected near their workloads and sent to the enterprise Prometheus-compatible metrics backend.
  2. Provider-native path: Azure Monitor, Amazon CloudWatch, and Google Cloud Monitoring remain authoritative for their managed services. Grafana queries those services through their supported data sources and workload identities or roles.

This is one operational view with multiple metric backends. It is not one universal schema or one PromQL endpoint for every metric. A service catalog records which path supplies each signal and which dashboard is authoritative.

Options considered

OptionBenefitsTradeoffsDecision
One self-managed Prometheus server for the enterpriseFamiliar Prometheus and PromQL; low initial complexityA single server is a capacity and failure boundary; remote sites require a reliable scrape path; long retention and multi-team isolation need additional designUse only for development, small environments, or local site monitoring
Federated local Prometheus serversSites can query local data and selectively aggregate metrics; useful when links are constrainedFederation is selective and query-oriented; it does not by itself provide a durable, scalable global metrics storeUse where sites need independent local queries or operate during WAN isolation
Central Mimir-compatible metrics plane with edge collectionPrometheus remote write, horizontal scaling, object storage, and tenant separation are available; collection stays close to workloadsThe enterprise owns backend sizing, upgrades, security, and recovery if self-hosted; managed offerings create service and egress dependenciesRecommended for the shared portable metrics plane when the enterprise can operate or procure it
Separate provider-managed metrics workspaces onlyProvider-native metrics, identity, retention, and scale are managed by each cloudData, query semantics, retention, and alert policies remain split across providers; on-premises exporters need a destinationKeep these workspaces for native service telemetry and use as Grafana data sources

Federation and remote write solve different problems. Use remote write to deliver selected local samples to a compatible long-term backend. Use federation when a central Prometheus server needs a selected view from other Prometheus servers. Do not create both flows by default; duplicate paths increase cost and can produce duplicate series or alerts.

Reference architecture

flowchart TB subgraph Sites[Enterprise sites and cloud regions] DC[Data center systems<br/>Linux and Windows, virtualisation, network, storage, databases] AZ[Azure workloads<br/>VMs, AKS, applications, platform services] AWS[AWS workloads<br/>EC2, EKS, applications, platform services] GCP[Google Cloud workloads<br/>Compute Engine, GKE, applications, platform services] COL[Regional Grafana Alloy or Prometheus collectors] NAT[Native cloud metric services<br/>Azure Monitor, CloudWatch, Cloud Monitoring] DC --> COL AZ --> COL AWS --> COL GCP --> COL end COL -->|TLS, workload identity, remote write| ING[Regional authenticated ingestion] ING --> STORE[Highly available Prometheus-compatible backend<br/>Grafana Mimir or managed equivalent] NAT -->|Provider data sources and read-only identities| GRAF[Grafana dashboards and alerting] STORE --> GRAF GRAF -->|Owned alert rules and contact points| ONCALL[On-call and incident system] CMDB[Service catalog and ownership metadata] --> COL CMDB --> GRAF

Collectors scrape local endpoints and provider APIs, attach stable ownership and location labels, and forward approved metrics. Network boundaries permit only the required exporter scrape traffic and authenticated outbound writes. Grafana queries the common backend and the provider-native stores with separate read identities. The service catalog supplies authoritative names for service, environment, site, account, subscription, project, cluster, and owner.

Collection and metric families

Environment or serviceRecommended collectionExamples of signals
Data-center Linux and UnixGrafana Alloy's Unix exporter or node_exporter, scraped locallyCPU, memory, filesystem, disk I/O, network interface, system load, and time synchronization
Data-center Windowswindows_exporter or an approved Alloy integrationCPU, memory, logical disks, services, IIS, process and OS health
Virtualisation, storage, network, and facilitiesVendor APIs, SNMP through a controlled SNMP exporter, or a supported Prometheus exporterHypervisor capacity, datastore latency, switch and firewall health, interface errors, storage controllers, UPS and environmental alarms
Databases and middlewareSupported product exporters plus vendor-native monitoring where requiredAvailability, connection pools, query latency, locks, replication, queue depth, broker throughput, cache health, and backup state
Kubernetes in any environmentExisting Prometheus Operator stack or Alloy collection, with one approved scrape owner per targetAPI and node health where exposed, scheduling, pod readiness, restarts, resource requests and limits, ingress, storage, and workload metrics
Applications and APIsPrometheus client libraries or OpenTelemetry metrics, with consistent service and deployment attributesRequest rate, errors, latency histograms, saturation, dependency outcomes, and business-level indicators without sensitive labels
Azure managed servicesAzure Monitor Metrics and Azure Monitor managed service for Prometheus; scrape custom endpoints with Alloy where appropriateAzure resource metrics, AKS and custom Prometheus metrics, service health, and resource dimensions
AWS managed servicesCloudWatch for AWS service metrics; Amazon Managed Service for Prometheus for Prometheus workloads where selectedEC2, EKS, ELB, RDS, DynamoDB, S3, Lambda, queues, and other CloudWatch-supported service metrics
Google Cloud managed servicesCloud Monitoring for Google Cloud service metrics; Google Cloud Managed Service for Prometheus for Prometheus workloads where selectedCompute Engine, GKE, Cloud Run, Cloud SQL, storage, load balancing, Pub/Sub, and other Cloud Monitoring-supported metrics

The service catalog MUST list every production service, expected signals, collection path, dashboard, alert owner, retention class, and known blind spots. Cloud-native services often publish metrics through provider APIs rather than scrape endpoints. Do not install exporters for services that the native API already covers unless a specific normalization need justifies the extra credentials, API calls, and maintenance.

Cloud integration pattern

ProviderNative service metricsPrometheus-compatible workload metricsGrafana access
AzureAzure Monitor Metrics, platform metrics, and Azure resource dimensionsAzure Monitor managed service for Prometheus or remote write to the enterprise backend for selected self-managed collectorsAzure Monitor data source for native metrics; Azure Monitor managed Prometheus data source for Azure-hosted Prometheus metrics; Prometheus data source for the enterprise backend
AWSCloudWatch metrics, dimensions, and alarmsAmazon Managed Service for Prometheus, AWS Distro for OpenTelemetry, or remote write to the enterprise backend for approved collectorsCloudWatch data source for native service metrics; Prometheus-compatible data source for AMP or the enterprise backend
Google CloudCloud Monitoring metrics and monitored-resource dimensionsGoogle Cloud Managed Service for Prometheus for managed or self-deployed collectionGoogle Cloud Monitoring data source for native services; Prometheus-compatible data source for Managed Service for Prometheus or the enterprise backend

Select the backend for each signal based on query requirements, data residency, retention, provider limits, cost, and team ownership. Azure Monitor supports remote write from self-managed Prometheus for hybrid collection; AWS AMP accepts remote write from customer-managed collectors; Google Cloud Managed Service for Prometheus supports Prometheus and OpenTelemetry collection from on-premises and other clouds. Provider support and limits differ, so the platform team MUST validate authentication, ingestion quotas, retention, and query compatibility for each region and workload before rollout.

Backend and Grafana placement

For a self-managed core, deploy Grafana Mimir in a supported production architecture with redundant components, replicated ingestion, durable cloud object storage, and a tested recovery plan. Do not use a single-process demo, local filesystem storage, or a single virtual machine as the production backend. Place ingestion endpoints near major collection regions and use private connectivity where available. Deploy Grafana with redundancy and configuration-as-code; make dashboards, data sources, folders, alert rules, and contact points reviewable in version control.

An enterprise MAY use Grafana Cloud, an appropriate provider-managed Prometheus service, or another managed Prometheus-compatible backend instead of operating Mimir. Record the resulting data locations, supported service levels, authentication boundaries, retention, query limits, and exit plan. Do not deploy a central shared platform and all three cloud-managed Prometheus stores for the same metric stream unless a documented availability, residency, or portability requirement needs that duplication.

Data contract and dashboard model

Every shared time series SHOULD carry bounded labels for environment, site, cloud, region, account, subscription or project, cluster, service, and owner where applicable. Use stable identifiers and controlled values. Keep provider-specific resource IDs available where useful, but do not overload portable service labels with provider syntax.

Metric names MUST identify one quantity and unit. Histograms MUST have documented bucket and aggregation behavior. Teams MUST NOT put request IDs, user IDs, email addresses, raw URLs, exception messages, or other unbounded or sensitive values into metric labels. Every new metric family SHOULD have an owner, unit, meaning, cardinality estimate, scrape interval, and retention class. Prometheus recommends clear names and warns that each unique label set creates a separate time series; see metric and label naming.

Dashboard tiers:

  1. Service overview: SLOs, availability, latency, traffic, errors, saturation, and links to service runbooks.
  2. Platform and dependency: Kubernetes, host, database, queue, storage, identity, DNS, network, and load balancer health.
  3. Provider operations: Native Azure, AWS, or Google Cloud resource metrics and provider-specific dimensions.
  4. Fleet and executive: Service coverage, alert health, capacity, cloud/account/site filters, and telemetry pipeline status.

Use a shared dashboard variable model where the backend supports the same labels. Provider-native dashboards SHOULD retain provider-specific labels and queries rather than hiding differences through fragile transformations. A single Grafana dashboard can combine supported data sources, but cross-data-source expressions must be tested for query semantics, time alignment, permissions, and failure behavior.

Identity, security, and tenancy

  • Collectors MUST run with least privilege. Use managed identities, workload identity, IAM roles, or short-lived federated credentials where supported. Avoid long-lived access keys and shared administrator credentials.
  • Exporters MUST be reachable only from approved collectors. Do not expose unauthenticated /metrics endpoints to the public internet or broad user networks.
  • Remote-write and query traffic MUST use TLS. Use private endpoints, private service connectivity, VPN, or controlled egress where the network design supports them; authenticate ingestion independently of network location.
  • Separate writer, platform administrator, dashboard editor, and read-only query roles. Scope provider-native Grafana data-source identities to the required accounts, subscriptions, projects, and metrics APIs.
  • Use Mimir tenant isolation or separate backends where teams, legal entities, or data residency boundaries require query isolation. Grafana folder permissions alone are not a data-store security boundary.
  • Treat metrics as potentially sensitive operational data. Restrict labels and metadata, define retention and deletion, and review dashboards and query access for leakage of infrastructure topology or customer context.
  • Pin collector and exporter releases, review plugin sources, and manage Grafana and Mimir configuration through approved change control.

Grafana data-source permissions can restrict query access, but provider credentials are used server-side and grant access to the underlying metric APIs. Grant only read permissions required for the selected data source and scope.

Resilience, capacity, and cost

Design for WAN interruptions, cloud API throttling, collector restarts, backend outages, and query surges. Alloy and Prometheus remote-write queues can buffer a bounded amount of data on local disk; configure queue capacity, WAL storage, retry behavior, and alerts against the expected outage window. Buffering is finite. Document which signals can be lost after that window and preserve local Prometheus query capability at sites that need to operate while disconnected.

The platform team MUST size ingestion and storage from measured active series, samples per second, scrape interval, label cardinality, retention, replication, and query concurrency. Forecast object storage, compute, inter-region transfer, cloud API calls, managed-service ingestion, and log/trace costs separately. Enforce per-tenant or per-team budgets, retention tiers, metric allowlists, and cardinality alerts. Do not use longer retention as a substitute for deciding which metrics have operational value.

Deploy the Grafana and metrics backend across failure domains appropriate to the service objectives. Back up configuration, dashboards, rules, tenant mappings, secrets references, and object-store metadata as required. Test recovery with realistic data volume and verify that agents resume delivery without generating duplicate or out-of-order series. Define a degraded mode for provider API failure: native cloud panels may be unavailable even when the common Prometheus backend remains healthy.

Alerting and operations

Assign a single evaluation and notification owner for every alert rule. Use Prometheus-compatible rules and Alertmanager or Grafana-managed alert rules according to an explicit ownership model; do not evaluate and page the same condition in multiple places without deduplication. Every page MUST include service, environment, owner, severity, impact, runbook, and a stable deduplication key. Route lower-severity capacity and trend signals to dashboards or tickets instead of paging.

Monitor the monitoring platform itself:

  • scrape target health, exporter health, collection delay, and missing expected series;
  • collector WAL growth, queue lag, retries, remote-write errors, and local disk headroom;
  • ingestion rates, rejected samples, active series, tenant quotas, and backend availability;
  • object-store, compactor, ruler, query, and Grafana health;
  • provider API errors, throttling, credential expiry, and missing account or project coverage;
  • alert delivery success, notification latency, stale dashboards, and unowned rules.

Use service-level indicators and error budgets to drive customer-impact alerts. Schedule maintenance behavior, suppression, escalation, and incident integration with the enterprise on-call process. Review alert quality after incidents and remove signals that have no action or owner.

Implementation sequence

  1. Build a service and resource inventory across data centers, Azure subscriptions, AWS accounts, and Google Cloud projects. Classify criticality, residency, owner, dependencies, and collection constraints.
  2. Agree on the metric and label contract, service taxonomy, retention classes, tenancy model, target data sources, and query access model.
  3. Deploy a non-production backend and Grafana configuration pipeline. Prove cloud identities, on-premises network paths, TLS, private routing, and data-source read permissions.
  4. Pilot one representative Linux service, Windows service, Kubernetes cluster, database, network device, and managed resource in each cloud. Compare native metrics with exporter data and resolve gaps or duplication.
  5. Publish base dashboards, recording rules, alert ownership, runbooks, and telemetry pipeline health panels. Test notifications end to end.
  6. Roll out by environment and service tier with ingestion budgets, cardinality limits, rollback procedures, and change records.
  7. Exercise data-center WAN isolation, cloud API throttling, collector disk exhaustion, backend failover, credential expiry, and Grafana recovery before production acceptance.

Validation and acceptance

  • The service catalog has an owner, metric source, dashboard, alert route, retention class, and blind-spot record for each production service.
  • Linux, Windows, Kubernetes, database, network, application, and cloud-managed service pilots deliver the expected metrics without duplicate collection.
  • Azure, AWS, and Google Cloud data sources authenticate with scoped short-lived identities or roles and fail closed when access is revoked.
  • Remote-write paths use TLS, authenticate writers, recover from transient outages, and alert before local buffers fill.
  • Common labels, metric units, histogram behavior, and cardinality budgets pass review; sensitive and unbounded label values are rejected.
  • Dashboards distinguish common Prometheus metrics from provider-native data and show source health and data freshness.
  • Alert tests prove correct owner routing, deduplication, maintenance handling, runbook access, and incident creation.
  • Retention, residency, ingestion quotas, query capacity, backend recovery, and monthly cost assumptions have documented evidence.
  • A site or region isolation exercise proves that critical teams retain the required local visibility and understand remote data-loss bounds.

Operational boundaries and acceptance criteria

Before adopting this guidance, record the accountable service owner, supported environments, critical dependencies, data classification, and the operational tier. State which decisions belong to the platform team and which remain with the workload team; document any exception with a risk owner and expiry date. This prevents a reference pattern from being treated as a universal fit.

Define observable acceptance checks before rollout. Include a positive path, a denied or invalid request, a dependency failure, and recovery after the failure clears. Capture the configuration or command output that proves each check, along with the software and policy versions used. Make checks repeatable in a representative non-production environment, and ensure they fail closed when required evidence is missing.

For production changes, identify the safe rollback or forward-recovery action, the person authorized to invoke it, and the condition that triggers it. Confirm that monitoring, support ownership, capacity limits, and escalation contacts are in place before declaring the change complete. Review these assumptions after a material platform, provider, or regulatory change.

Delivery semantics, quotas, and replay

For every integration boundary, state whether delivery is at-most-once, at-least-once, or effectively-once through deduplication. Consumers should be idempotent, acknowledge only after durable processing, and route poison messages to a dead-letter or quarantine path with enough metadata for diagnosis. Retries need bounded exponential backoff and jitter; synchronized retries can amplify an outage and exhaust provider quotas.

Version event or API contracts independently from deployment, define compatibility rules, and test old and new consumers during rollout. Measure queue age, rejected requests, throttling, retry volume, and dead-letter growth. Set explicit capacity and payload limits, and rehearse replay after correcting a consumer defect. A replay should be scoped, rate-limited, auditable, and safe to stop without losing the original message.

References