Production architecture case narrative

Deploying Cloud Infrastructure Through Infrastructure as Code

Contoso production infrastructure delivery: Terraform and Bicep module boundaries, isolated remote state, federated identities, policy gates, environment promotion, drift handling, and operational assurance.

Platform Engineering Cloud Foundation

Scenario context

This technical case narrative describes a production-oriented design. It does not verify that a client engagement was completed or that project-specific results were achieved.

Starting conditions

The source describes delivery conditions the proposed infrastructure workflow is intended to address. It does not provide a verified baseline from a deployed client environment.

Executive Summary

Business context — Contoso: Contoso operates multiple Azure application environments provisioned through a combination of portal changes, individual scripts, and inconsistent Terraform projects. Changes are difficult to trace across environments, production permissions are too broad, and release teams lack a single audited delivery path.

Manual infrastructure deployment works at small scale, but it becomes increasingly difficult to control as cloud environments grow. Configuration drift appears between environments, change history becomes incomplete, security controls depend on individual knowledge, and application teams wait for infrastructure specialists to provision resources.

This Contoso architecture case study documents how Tongtech converts cloud infrastructure delivery into a version-controlled engineering workflow using Infrastructure as Code, reusable modules, automated validation, secure deployment identities, and CI/CD.

The objective is to make infrastructure delivery repeatable without turning the IaC platform into an unnecessarily complex abstraction layer.

Client confidentiality: Contoso is the client alias used in this technical case study. Resource names, address spaces, workload volumes, and configuration examples are reference design parameters, not disclosed customer records. The design does not establish that a customer deployment occurred or that quantitative results were measured.

The Challenge

Contoso’s existing delivery model presents a mixture of:

  • resources created manually in the cloud portal
  • deployment scripts maintained by individual engineers
  • inconsistent naming and tagging
  • duplicated Terraform code
  • environment-specific configuration
  • long-lived service-principal secrets
  • infrastructure changes performed outside change control
  • incomplete documentation
  • no reliable drift-management process
  • production deployments that depend on a small number of specialists

Contoso needs a deployment model in which infrastructure is treated as software.

Objectives

The IaC platform is designed to:

  1. Make infrastructure definitions version-controlled and reviewable.
  2. Standardize recurring infrastructure patterns.
  3. Reduce configuration differences between environments.
  4. Eliminate unnecessary manual portal deployment.
  5. Integrate infrastructure changes with CI/CD.
  6. Add automated security and policy checks before deployment.
  7. Use short-lived or federated authentication where possible.
  8. Create clear promotion and approval processes.
  9. Keep the code understandable to internal engineering teams.
  10. Allow application teams to consume platform patterns without bypassing governance.

Target Delivery Model

flowchart TB CHANGE["Engineer changes infrastructure as code"] --> PR["Pull request"] PR --> VALIDATE["Format, lint, and validate"] VALIDATE --> SCAN["Security and policy checks"] SCAN --> PLAN["Terraform plan or Bicep What-If"] PLAN --> REVIEW["Peer review"] REVIEW --> APPROVE["Approval"] APPROVE --> MERGE["Merge"] MERGE --> DEPLOY["Environment deployment"] DEPLOY --> VERIFY["Post-deployment validation"]
Infrastructure as code pull request and deployment flow

The cloud portal becomes an operational interface rather than the primary deployment mechanism.

Repository and Code Structure

A maintainable structure separates reusable components from environment-specific configuration.

Example
infra/
  modules/
    network/
    private-endpoint/
    key-vault/
    storage/
    app-service/
    aks/
    monitoring/

  environments/
    dev/
    test/
    staging/
    production/

  platform/
    identity/
    connectivity/
    policy/
    monitoring/

  policies/
  pipelines/
  scripts/
  docs/

Key principles:

  • reusable logic belongs in modules
  • environment configuration remains explicit
  • secrets are not stored in code
  • provider and module versions are controlled
  • state ownership is clearly defined
  • changes are applied through repeatable workflows

Reusable Terraform and Bicep Modules

Reusable modules are created around real platform boundaries rather than around every individual resource.

Useful module boundaries can represent:

  • a private App Service pattern
  • an application network
  • a Key Vault with standard security controls
  • a storage account with private connectivity
  • AKS platform components
  • diagnostic settings
  • role assignments
  • a Private Endpoint plus DNS integration

The approach is to use abstraction only where it improves consistency, security, maintainability, or developer experience.

State Management

For Terraform, remote state is configured using a controlled backend.

The design addresses:

  • state isolation
  • environment separation
  • access control
  • locking
  • backup and recovery
  • sensitive values
  • state ownership

State boundaries follow deployment and ownership boundaries rather than placing the entire estate into one oversized state file.

Identity and Authentication

CI/CD pipelines should not depend on personal accounts or long-lived cloud secrets.

Where supported, deployment pipelines use workload identity federation.

For Azure deployments, this commonly means:

  • Microsoft Entra application or managed identity
  • federated credential
  • narrowly scoped Azure RBAC
  • GitHub OIDC or Azure DevOps workload identity federation

Permissions are separated between validation, non-production deployment, production deployment, state access, and administrative platform operations.

CI/CD Validation

Every infrastructure change should be validated before it reaches production.

Code Quality

  • terraform fmt
  • terraform validate
  • Bicep validation
  • linting
  • dependency/version checks

Security

Depending on tooling:

  • Checkov
  • tfsec
  • Trivy
  • Microsoft security tooling
  • custom static checks

Policy as Code

OPA/Conftest or equivalent policy checks can enforce engineering standards before cloud deployment.

Examples:

  • prohibit public storage access
  • require approved regions
  • require diagnostic configuration
  • prevent overly permissive network rules
  • restrict specific SKU choices
  • validate tags or ownership metadata

Azure Policy provides runtime governance, while pipeline policy catches issues earlier in delivery.

Pull Request Workflow

Infrastructure changes are reviewed using the same engineering discipline as application code.

A pull request can include:

  • change description
  • architecture impact
  • Terraform plan
  • policy results
  • security scan results
  • deployment risk
  • rollback considerations

Reviewers see the proposed infrastructure changes before they are applied.

Environment Promotion

Development, test, staging, and production use the same underlying infrastructure patterns where practical.

Differences should be configuration-driven rather than implemented as unrelated copies of code.

Promotion does not necessarily mean deploying the same resource sizes everywhere. It means applying the same architectural pattern through controlled configuration.

Secrets and Sensitive Configuration

Secrets are stored outside the code repository.

Common approaches include:

  • Azure Key Vault
  • managed identities
  • federated identities
  • secured CI/CD variables for unavoidable secrets

Where possible, workloads authenticate directly using managed identity so application-level secrets can also be reduced.

Drift Management

The operating model includes:

  • scheduled plan or drift detection
  • review of portal changes
  • clear break-glass procedures
  • remediation through code
  • restrictions on unnecessary write access

The declared IaC configuration remains the authoritative state of the platform.

Platform Consumption

Reusable infrastructure patterns can be exposed to application teams through:

  • documented Terraform modules
  • reusable pipeline templates
  • repository templates
  • service catalog entries
  • internal developer platform workflows
  • controlled self-service provisioning

This allows application teams to consume standardized cloud infrastructure without administrator access to every underlying service.

Implementation Sequence

1. Assess Existing Deployment Practices

Review current IaC, portal-created infrastructure, scripts, pipelines, identity model, state management, module duplication, environment differences, and operational pain points.

2. Define IaC Standards

Decide on Terraform, Bicep, or a justified combination; repository structure; module boundaries; naming conventions; state boundaries; version strategy; deployment identity model; and code review process.

3. Build Core Modules

Prioritize recurring patterns rather than attempting to modularize the entire cloud estate immediately.

4. Build CI/CD

Implement validation, plan/preview, security checks, policy checks, approvals, deployment, and post-deployment validation.

5. Migrate Existing Infrastructure

Existing resources can be imported or progressively brought under code rather than recreated blindly.

6. Operationalize

Document the contribution process, release process, break-glass procedure, module ownership, upgrade strategy, drift handling, and troubleshooting model.

Production Architecture and Low-Level Design

1. Scope, Ownership, and Deployment Boundaries

The target delivery system covers shared platform services and application landing-zone infrastructure in separate Azure subscriptions. Platform engineering owns baseline modules, pipeline templates, remote-state administration, policy guardrails, and shared connectivity. Application teams own their workload declarations, deployment requests, and post-release verification. Security approves policy exceptions and privileged role boundaries; operations owns alert routing, recovery documentation, and break-glass procedures.

Boundary Reference ownership Deployed through Separation rule
Management groups and policy assignments Cloud platform Platform pipeline Tenant-scope identity; separate state
Connectivity and DNS Network platform Connectivity pipeline Independent change window and state
Shared observability Operations platform Management pipeline Dedicated subscription and state
Application infrastructure Workload team Workload pipeline One environment/subscription per state boundary where appropriate
Application code Application team Release pipeline No direct modification of platform state

A production implementation maps these boundaries to Entra groups, Azure roles, Git repository ownership, branch-protection rules, state containers, deployment environments, and support escalation. No single pipeline identity is granted tenant-wide Contributor by default.

2. Reference Subscription and Resource Organization

The following is an implementation naming scheme, not a record of Contoso's actual inventory.

Example
mg-contoso
├── mg-platform
│   ├── sub-contoso-connectivity-prod
│   ├── sub-contoso-management-prod
│   └── sub-contoso-identity-prod
└── mg-landingzones
    ├── sub-contoso-payments-dev
    ├── sub-contoso-payments-test
    └── sub-contoso-payments-prod

Workload resource groups are divided by lifecycle: rg-payments-app-prod, rg-payments-data-prod, and rg-payments-network-prod. Reusable names are generated from a constrained schema (organization, workload, environment, region, resource_type, instance) and validated for Azure service-specific limits and globally unique names where required.

3. Terraform State Architecture and Recovery

Terraform remote state is hosted in a dedicated Azure Storage account controlled by the platform team. Containers and state keys segregate platform components and workload environments. The design uses Microsoft Entra authorization with Azure Blob Storage; the identity needs narrowly scoped data-plane permissions on the relevant container plus required management-plane permissions for deployed resources. Azure Blob leases provide state locking during compatible Terraform operations.

flowchart TB PIPELINE["Protected pipeline"] --> FEDERATION["Federated identity"] FEDERATION --> RUN["Terraform run"] RUN --> BACKEND["Azure Blob state backend"] BACKEND --> LOCK["Lease-based state lock"] BACKEND --> KEYS["Separate platform and workload keys"] RUN --> RBAC["Scoped Azure RBAC"] RBAC --> TARGET["Target subscriptions"] RECOVERY["Separate recovery role"] --> VERSION["Versioned state blob"] VERSION --> REVIEW["Lock, inspect, and reconcile before restore"]
Terraform state access and recovery boundaries
Example
Storage account: stcontosotfstateprod01     # reference name
Containers:      tfstate-platform, tfstate-workloads
Blob keys:
  platform/policy/prod.tfstate
  platform/connectivity/prod.tfstate
  workloads/payments/dev.tfstate
  workloads/payments/test.tfstate
  workloads/payments/prod.tfstate

State storage requires encryption, public access restrictions, private connectivity where the runners support it, diagnostic logging, soft delete/versioning as appropriate, and a rehearsed recovery process. Blob version restoration must be coordinated with deployment locking; restoring a previous state blob without reconciliation can cause resource deletion or replacement. State access is limited to the relevant deployment identity and a separately controlled recovery role. Terraform state is treated as sensitive even when configuration values are marked sensitive, because such markings do not generally remove secrets from state.

Backend configuration is injected at initialization instead of committing authentication values to the repository. A minimal partial configuration is:

Terraform / HCL
terraform {
  backend "azurerm" {}
  required_providers {
    azurerm = { source = "hashicorp/azurerm", version = "~> 4.0" }
  }
}
Bash
terraform init \
  -backend-config="resource_group_name=rg-platform-tfstate" \
  -backend-config="storage_account_name=stcontosotfstateprod01" \
  -backend-config="container_name=tfstate-workloads" \
  -backend-config="key=workloads/payments/prod.tfstate"

Versions shown are reference constraints and must be pinned and tested against the approved provider/CLI matrix before use. Workload identity authentication is set by the runner and backend environment, not by hard-coded credentials.

4. Repository Layout, Modules, and Dependency Graph

Example
cloud-platform/
  modules/
    network-spoke/              # VNet, subnets, NSGs, routes
    app-service-private/        # service plan, app, integration, PE/DNS
    key-vault/                  # RBAC, firewall, diagnostics
    sql-private/                # logical server, database, PE/DNS
    monitor-baseline/           # diagnostics, alerts, action groups
  live/
    platform/connectivity/prod/
    platform/management/prod/
    payments/dev/
    payments/test/
    payments/prod/
  policy/
    terraform-plan/             # Conftest/Rego checks
    azure/                      # definitions, initiatives, assignments
  .github/workflows/
    validate.yml
    deploy-nonprod.yml
    deploy-prod.yml
  scripts/verify-network.sh
  docs/adr/ runbooks/ ownership/

Module interfaces specify supported versions, required inputs, validations, outputs, diagnostic defaults, private DNS ownership, and lifecycle behavior. A module must not silently create shared DNS zones or RBAC assignments owned by another state; instead it consumes approved resource IDs and creates only explicitly delegated components. Cross-state outputs are shared by approved configuration or resource identifiers, not by unrestricted remote-state read access.

The dependency sequence is: management hierarchy and policy baseline → connectivity/DNS → monitoring and identities → workload subnet and data services → application resources → smoke tests. Pipeline orchestration avoids running changes concurrently against shared parent resources.

5. Deployment Identity and Permission Matrix

Identity Trust boundary Permitted operations Explicit restriction
id-tf-validate Pull request Read-only plan and validation, where required No production write grants
id-tf-dev Protected development environment Write to development workload scope; access its state No production role
id-tf-prod Approved production environment Write to designated workload resources and state No tenant-level Owner
id-platform-policy Platform-maintained workflow Authorized policy assignments at approved scopes Not shared with workload pipeline
id-recovery Privileged emergency path Time-limited state recovery access Not used for routine deployment

GitHub Actions or Azure DevOps workload identity federation uses trust criteria scoped to the exact repository or pipeline and environment, with production protection rules and required reviewers. Federated authentication eliminates static client secrets in the deployment workflow; it does not eliminate the need to secure the runner, workflow definition, approvals, or cloud role assignments.

6. CI/CD Release Contract

A pull request runs formatting, Terraform validation, static checks, dependency scanning, and a plan in a controlled read-only context. A complete change record stores the commit SHA, module versions, plan artifact identifier, policy results, environment, requester, approver, and deployment trace. Do not expose sensitive plan files to untrusted pull-request contributors. Protected branch and CODEOWNERS rules require platform review for shared modules and policy files.

For production, a fresh plan is generated after merge against current state, then an approved, immutable plan artifact is applied by the corresponding identity. An expired or superseded plan is discarded and regenerated. CI jobs must not apply a plan generated for a different commit, state snapshot, workspace, or identity scope.

flowchart TB PR["Pull request"] --> CHECKS["Format, validation, tests, and scans"] CHECKS --> SPEC["Speculative plan"] SPEC --> REVIEW["Peer review"] REVIEW --> MERGE["Merge"] MERGE --> ENV["Protected environment"] ENV --> PLAN["Refresh and produce current plan"] PLAN --> APPROVE["Approval"] APPROVE --> APPLY["Apply the exact approved plan"] APPLY --> VERIFY["Health and policy verification"] VERIFY --> AUDIT["Audit record"]
Terraform plan review and controlled apply

Deployment gating includes a manual decision for destructive replacements, changes to route tables/NSGs/private DNS, loss of encryption controls, or privilege expansion. Terraform plans identify expected resource operations, but cannot fully prove application compatibility or availability; smoke tests and service checks are mandatory after deployment.

7. Policy-as-Code and Runtime Governance

Pipeline controls check the planned change, while Azure Policy verifies resources at their effective scopes. Example gates include: disallow public storage access unless explicitly exempted, require approved regions, flag unrestricted inbound administrative ports, validate required resource ownership tags, and ensure diagnostics are provisioned for supported resource types. Not every property can be validated through a single Terraform plan rule, and some policy effects require existing resources to be remediated after assignment.

Policy deployments follow audit/reporting, impact analysis, exception review, and gradual deny/modify/deployIfNotExists activation as appropriate. Exemptions have owners, business justification, scope, and an expiry/review date. Every policy control maps to an Azure-native enforcement capability or a compensating operational check; broad "compliant" claims are avoided without evidence.

8. Existing Resource Adoption Without Re-creation

Before onboarding portal-managed production resources, engineers inventory resource IDs, dependencies, identity assignments, DNS, networking, SKU details, monitoring settings, and deletion protection. They define import blocks or use controlled terraform import, then align configuration to the existing resource shape until plan produces no unapproved replacement. Resource addresses and moved blocks are documented when refactoring state, and imported resources are validated against runtime behavior before the first write deployment.

High-impact resources (databases, Key Vault, private endpoints, firewall rules) are brought under management in separate change sets. A no-op import is not sufficient if unmanaged children, inherited permissions, or drift outside provider coverage remain. Backups and rollback are documented before modifying state or applying infrastructure changes.

9. Drift Detection, Break-Glass, and Rollback

Scheduled read-only plans detect drift against the desired configuration. Drift tickets capture resource IDs, expected state, actual state, change actor where observable, and disposition: restore declared configuration, formally accept and commit the change, or escalate for incident response. Continuous auto-apply is disabled for production drift remediation.

A break-glass workflow requires a ticket, elevated access with expiry, recorded portal/CLI changes, and post-incident reconciliation back into code. Rollback differs from application rollback: reverting a Git commit may produce destructive infrastructure changes. Operators review the reverse plan and use service-specific restoration for deleted or corrupted resources. The recovery plan includes Terraform state history and backups, application data backups, and re-provisioning dependencies.

10. Monitoring and Service Operations

Every deployment emits audit events into the pipeline platform and Azure activity/diagnostic logs. Shared monitoring covers failed deployments, service health, policy noncompliance, suspicious role changes, state-store access anomalies, connectivity changes, and workload telemetry. Action groups route to assigned responders; noise reduction is handled through severity, suppression windows, and runbooks. Cost tags identify owners but must be validated before using them for chargeback.

11. Production Acceptance and Handover Checklist

Acceptance test Evidence required Pass condition
Environment parity Parameter and module version comparison Intentional deviations documented and approved
Least-privilege deployment Role assignment review; test denied action Non-production pipeline cannot write production
State integrity Lock contention test; recovery rehearsal Concurrency controlled and state restorable
Private service connectivity Name resolution and TCP/application test from approved runner Correct private address and successful authenticated request
Policy controls Sample compliant and noncompliant deployment tests Expected audit/deny/remediation behavior
Release safety Plan approval and deployment audit trail Approved artifact tied to commit and environment
Drift process Controlled out-of-band change Detected, triaged, reconciled
Recovery Documented resource/state restore exercise Recovery procedures verified against stated objectives

Completion requires signed ownership, runbooks, escalation paths, IaC repository permissions, monitoring routes, and a prioritized backlog; pipeline success alone is not production acceptance.

Before and After Implementation: Operational Improvements

The following describes the architecture's expected operating-state changes, not independently measured Contoso results. Quantitative impact should be reported only after baseline and post-rollout measurement.

Dimension Before implementation After implementation (target state) Benefit / verification method
Provisioning Engineers create and update resources manually Version-controlled, approved Terraform/Bicep workflows Compare median ticket-to-ready time over equivalent workloads
Configuration consistency Differences across dev, test, and production Shared versioned modules plus controlled environment variables Compare unapproved parameter/resource differences
Change transparency Incomplete portal/script history PR, plan, approval, deployment and commit linkage Audit sample of production changes
Production access Broad standing permissions for deployment operators Scoped federated deployment identities and protected approvals Review standing privileges and role-assignment changes
Terraform state Fragmented local or inconsistently secured state Segregated remote state with locking, RBAC, retention and restore State-access audit and restore exercise
Security controls Manual checks near release Static checks, policy gates, and Azure Policy assignments Count prevented violations and time to remediation
Release quality Ad hoc operations and no standard readiness checks Test gates, review controls and post-apply validation Change failure rate and rollback incident count
Drift Portal changes remain unnoticed Scheduled read-only drift detection and ticketed correction Drift detection lead time and outstanding drift age
Workload onboarding Bespoke setup for each application Standard repository/module templates and owner assignment Request-to-first-approved-deploy time
Recovery Individual knowledge and manual restore Documented state recovery and service-specific runbooks Restore exercise results and RTO/RPO evidence
Cost ownership Inconsistent tagging and budgets Required ownership metadata and budget monitoring Percentage of spend mapped to accountable owner
Operational support Escalation depends on individuals Runbooks, action groups and ownership matrix Incident handoff completeness and time to acknowledge

Measurement and Claims Standard

Record baseline and after-period dates, workload scope, sample sizes, exclusions, data sources and metric owners. Suggested evidence includes CI execution records, Azure Activity Log, Azure Policy compliance exports, Terraform plan artifacts, incident tickets, and Azure Cost Management. Avoid statements such as "deployment time improved 80%" or "zero drift" without those measurements.

Project Deliverables

A typical engagement can include:

  • IaC architecture and standards
  • repository structure
  • reusable Terraform/Bicep modules
  • remote state design
  • environment configuration
  • GitHub Actions or Azure DevOps pipelines
  • workload identity federation
  • policy-as-code controls
  • security validation
  • documentation
  • deployment runbooks
  • migration plan for existing infrastructure
Evidence note: This page describes a production-oriented design. It is not evidence that the design was deployed for a client, and it reports no measured client results.

Expected Architectural Outcomes

The infrastructure platform is designed to become reproducible, reviewable, and easier to operate.

Instead of relying on manual cloud configuration, teams work from a declared desired state stored in source control.

Contoso’s target operating model provides:

  • consistent infrastructure patterns
  • clearer change history
  • stronger deployment controls
  • reduced configuration drift
  • reusable platform components
  • simpler onboarding for new workloads
  • better separation between platform and application responsibilities
  • a foundation for controlled self-service

Most importantly, infrastructure delivery becomes an engineering process rather than an accumulation of manual cloud changes.

Technologies

Terraform · Bicep · Microsoft Azure · GitHub Actions · Azure DevOps · Workload Identity Federation · Open Policy Agent · Conftest · Azure Policy · Key Vault · CI/CD

Next step

Discuss a similar Platform Engineering challenge.

Share your current architecture, constraints and desired next step. Tongtech can help determine whether a focused review or implementation engagement fits.

Book an Architecture Review