Scenario context
This technical case narrative describes a production-oriented design. It does not verify that a client engagement was completed or that project-specific results were achieved.
Starting conditions
The source describes delivery conditions the proposed infrastructure workflow is intended to address. It does not provide a verified baseline from a deployed client environment.
Executive Summary
Business context — Contoso: Contoso operates multiple Azure application environments provisioned through a combination of portal changes, individual scripts, and inconsistent Terraform projects. Changes are difficult to trace across environments, production permissions are too broad, and release teams lack a single audited delivery path.
Manual infrastructure deployment works at small scale, but it becomes increasingly difficult to control as cloud environments grow. Configuration drift appears between environments, change history becomes incomplete, security controls depend on individual knowledge, and application teams wait for infrastructure specialists to provision resources.
This Contoso architecture case study documents how Tongtech converts cloud infrastructure delivery into a version-controlled engineering workflow using Infrastructure as Code, reusable modules, automated validation, secure deployment identities, and CI/CD.
The objective is to make infrastructure delivery repeatable without turning the IaC platform into an unnecessarily complex abstraction layer.
Client confidentiality: Contoso is the client alias used in this technical case study. Resource names, address spaces, workload volumes, and configuration examples are reference design parameters, not disclosed customer records. The design does not establish that a customer deployment occurred or that quantitative results were measured.
The Challenge
Contoso’s existing delivery model presents a mixture of:
- resources created manually in the cloud portal
- deployment scripts maintained by individual engineers
- inconsistent naming and tagging
- duplicated Terraform code
- environment-specific configuration
- long-lived service-principal secrets
- infrastructure changes performed outside change control
- incomplete documentation
- no reliable drift-management process
- production deployments that depend on a small number of specialists
Contoso needs a deployment model in which infrastructure is treated as software.
Objectives
The IaC platform is designed to:
- Make infrastructure definitions version-controlled and reviewable.
- Standardize recurring infrastructure patterns.
- Reduce configuration differences between environments.
- Eliminate unnecessary manual portal deployment.
- Integrate infrastructure changes with CI/CD.
- Add automated security and policy checks before deployment.
- Use short-lived or federated authentication where possible.
- Create clear promotion and approval processes.
- Keep the code understandable to internal engineering teams.
- Allow application teams to consume platform patterns without bypassing governance.
Target Delivery Model
The cloud portal becomes an operational interface rather than the primary deployment mechanism.
Repository and Code Structure
A maintainable structure separates reusable components from environment-specific configuration.
infra/
modules/
network/
private-endpoint/
key-vault/
storage/
app-service/
aks/
monitoring/
environments/
dev/
test/
staging/
production/
platform/
identity/
connectivity/
policy/
monitoring/
policies/
pipelines/
scripts/
docs/Key principles:
- reusable logic belongs in modules
- environment configuration remains explicit
- secrets are not stored in code
- provider and module versions are controlled
- state ownership is clearly defined
- changes are applied through repeatable workflows
Reusable Terraform and Bicep Modules
Reusable modules are created around real platform boundaries rather than around every individual resource.
Useful module boundaries can represent:
- a private App Service pattern
- an application network
- a Key Vault with standard security controls
- a storage account with private connectivity
- AKS platform components
- diagnostic settings
- role assignments
- a Private Endpoint plus DNS integration
The approach is to use abstraction only where it improves consistency, security, maintainability, or developer experience.
State Management
For Terraform, remote state is configured using a controlled backend.
The design addresses:
- state isolation
- environment separation
- access control
- locking
- backup and recovery
- sensitive values
- state ownership
State boundaries follow deployment and ownership boundaries rather than placing the entire estate into one oversized state file.
Identity and Authentication
CI/CD pipelines should not depend on personal accounts or long-lived cloud secrets.
Where supported, deployment pipelines use workload identity federation.
For Azure deployments, this commonly means:
- Microsoft Entra application or managed identity
- federated credential
- narrowly scoped Azure RBAC
- GitHub OIDC or Azure DevOps workload identity federation
Permissions are separated between validation, non-production deployment, production deployment, state access, and administrative platform operations.
CI/CD Validation
Every infrastructure change should be validated before it reaches production.
Code Quality
terraform fmtterraform validate- Bicep validation
- linting
- dependency/version checks
Security
Depending on tooling:
- Checkov
- tfsec
- Trivy
- Microsoft security tooling
- custom static checks
Policy as Code
OPA/Conftest or equivalent policy checks can enforce engineering standards before cloud deployment.
Examples:
- prohibit public storage access
- require approved regions
- require diagnostic configuration
- prevent overly permissive network rules
- restrict specific SKU choices
- validate tags or ownership metadata
Azure Policy provides runtime governance, while pipeline policy catches issues earlier in delivery.
Pull Request Workflow
Infrastructure changes are reviewed using the same engineering discipline as application code.
A pull request can include:
- change description
- architecture impact
- Terraform plan
- policy results
- security scan results
- deployment risk
- rollback considerations
Reviewers see the proposed infrastructure changes before they are applied.
Environment Promotion
Development, test, staging, and production use the same underlying infrastructure patterns where practical.
Differences should be configuration-driven rather than implemented as unrelated copies of code.
Promotion does not necessarily mean deploying the same resource sizes everywhere. It means applying the same architectural pattern through controlled configuration.
Secrets and Sensitive Configuration
Secrets are stored outside the code repository.
Common approaches include:
- Azure Key Vault
- managed identities
- federated identities
- secured CI/CD variables for unavoidable secrets
Where possible, workloads authenticate directly using managed identity so application-level secrets can also be reduced.
Drift Management
The operating model includes:
- scheduled plan or drift detection
- review of portal changes
- clear break-glass procedures
- remediation through code
- restrictions on unnecessary write access
The declared IaC configuration remains the authoritative state of the platform.
Platform Consumption
Reusable infrastructure patterns can be exposed to application teams through:
- documented Terraform modules
- reusable pipeline templates
- repository templates
- service catalog entries
- internal developer platform workflows
- controlled self-service provisioning
This allows application teams to consume standardized cloud infrastructure without administrator access to every underlying service.
Implementation Sequence
1. Assess Existing Deployment Practices
Review current IaC, portal-created infrastructure, scripts, pipelines, identity model, state management, module duplication, environment differences, and operational pain points.
2. Define IaC Standards
Decide on Terraform, Bicep, or a justified combination; repository structure; module boundaries; naming conventions; state boundaries; version strategy; deployment identity model; and code review process.
3. Build Core Modules
Prioritize recurring patterns rather than attempting to modularize the entire cloud estate immediately.
4. Build CI/CD
Implement validation, plan/preview, security checks, policy checks, approvals, deployment, and post-deployment validation.
5. Migrate Existing Infrastructure
Existing resources can be imported or progressively brought under code rather than recreated blindly.
6. Operationalize
Document the contribution process, release process, break-glass procedure, module ownership, upgrade strategy, drift handling, and troubleshooting model.
Production Architecture and Low-Level Design
1. Scope, Ownership, and Deployment Boundaries
The target delivery system covers shared platform services and application landing-zone infrastructure in separate Azure subscriptions. Platform engineering owns baseline modules, pipeline templates, remote-state administration, policy guardrails, and shared connectivity. Application teams own their workload declarations, deployment requests, and post-release verification. Security approves policy exceptions and privileged role boundaries; operations owns alert routing, recovery documentation, and break-glass procedures.
| Boundary | Reference ownership | Deployed through | Separation rule |
|---|---|---|---|
| Management groups and policy assignments | Cloud platform | Platform pipeline | Tenant-scope identity; separate state |
| Connectivity and DNS | Network platform | Connectivity pipeline | Independent change window and state |
| Shared observability | Operations platform | Management pipeline | Dedicated subscription and state |
| Application infrastructure | Workload team | Workload pipeline | One environment/subscription per state boundary where appropriate |
| Application code | Application team | Release pipeline | No direct modification of platform state |
A production implementation maps these boundaries to Entra groups, Azure roles, Git repository ownership, branch-protection rules, state containers, deployment environments, and support escalation. No single pipeline identity is granted tenant-wide Contributor by default.
2. Reference Subscription and Resource Organization
The following is an implementation naming scheme, not a record of Contoso's actual inventory.
mg-contoso
├── mg-platform
│ ├── sub-contoso-connectivity-prod
│ ├── sub-contoso-management-prod
│ └── sub-contoso-identity-prod
└── mg-landingzones
├── sub-contoso-payments-dev
├── sub-contoso-payments-test
└── sub-contoso-payments-prodWorkload resource groups are divided by lifecycle: rg-payments-app-prod, rg-payments-data-prod, and rg-payments-network-prod. Reusable names are generated from a constrained schema (organization, workload, environment, region, resource_type, instance) and validated for Azure service-specific limits and globally unique names where required.
3. Terraform State Architecture and Recovery
Terraform remote state is hosted in a dedicated Azure Storage account controlled by the platform team. Containers and state keys segregate platform components and workload environments. The design uses Microsoft Entra authorization with Azure Blob Storage; the identity needs narrowly scoped data-plane permissions on the relevant container plus required management-plane permissions for deployed resources. Azure Blob leases provide state locking during compatible Terraform operations.
Storage account: stcontosotfstateprod01 # reference name
Containers: tfstate-platform, tfstate-workloads
Blob keys:
platform/policy/prod.tfstate
platform/connectivity/prod.tfstate
workloads/payments/dev.tfstate
workloads/payments/test.tfstate
workloads/payments/prod.tfstateState storage requires encryption, public access restrictions, private connectivity where the runners support it, diagnostic logging, soft delete/versioning as appropriate, and a rehearsed recovery process. Blob version restoration must be coordinated with deployment locking; restoring a previous state blob without reconciliation can cause resource deletion or replacement. State access is limited to the relevant deployment identity and a separately controlled recovery role. Terraform state is treated as sensitive even when configuration values are marked sensitive, because such markings do not generally remove secrets from state.
Backend configuration is injected at initialization instead of committing authentication values to the repository. A minimal partial configuration is:
terraform {
backend "azurerm" {}
required_providers {
azurerm = { source = "hashicorp/azurerm", version = "~> 4.0" }
}
}terraform init \
-backend-config="resource_group_name=rg-platform-tfstate" \
-backend-config="storage_account_name=stcontosotfstateprod01" \
-backend-config="container_name=tfstate-workloads" \
-backend-config="key=workloads/payments/prod.tfstate"Versions shown are reference constraints and must be pinned and tested against the approved provider/CLI matrix before use. Workload identity authentication is set by the runner and backend environment, not by hard-coded credentials.
4. Repository Layout, Modules, and Dependency Graph
cloud-platform/
modules/
network-spoke/ # VNet, subnets, NSGs, routes
app-service-private/ # service plan, app, integration, PE/DNS
key-vault/ # RBAC, firewall, diagnostics
sql-private/ # logical server, database, PE/DNS
monitor-baseline/ # diagnostics, alerts, action groups
live/
platform/connectivity/prod/
platform/management/prod/
payments/dev/
payments/test/
payments/prod/
policy/
terraform-plan/ # Conftest/Rego checks
azure/ # definitions, initiatives, assignments
.github/workflows/
validate.yml
deploy-nonprod.yml
deploy-prod.yml
scripts/verify-network.sh
docs/adr/ runbooks/ ownership/Module interfaces specify supported versions, required inputs, validations, outputs, diagnostic defaults, private DNS ownership, and lifecycle behavior. A module must not silently create shared DNS zones or RBAC assignments owned by another state; instead it consumes approved resource IDs and creates only explicitly delegated components. Cross-state outputs are shared by approved configuration or resource identifiers, not by unrestricted remote-state read access.
The dependency sequence is: management hierarchy and policy baseline → connectivity/DNS → monitoring and identities → workload subnet and data services → application resources → smoke tests. Pipeline orchestration avoids running changes concurrently against shared parent resources.
5. Deployment Identity and Permission Matrix
| Identity | Trust boundary | Permitted operations | Explicit restriction |
|---|---|---|---|
id-tf-validate |
Pull request | Read-only plan and validation, where required | No production write grants |
id-tf-dev |
Protected development environment | Write to development workload scope; access its state | No production role |
id-tf-prod |
Approved production environment | Write to designated workload resources and state | No tenant-level Owner |
id-platform-policy |
Platform-maintained workflow | Authorized policy assignments at approved scopes | Not shared with workload pipeline |
id-recovery |
Privileged emergency path | Time-limited state recovery access | Not used for routine deployment |
GitHub Actions or Azure DevOps workload identity federation uses trust criteria scoped to the exact repository or pipeline and environment, with production protection rules and required reviewers. Federated authentication eliminates static client secrets in the deployment workflow; it does not eliminate the need to secure the runner, workflow definition, approvals, or cloud role assignments.
6. CI/CD Release Contract
A pull request runs formatting, Terraform validation, static checks, dependency scanning, and a plan in a controlled read-only context. A complete change record stores the commit SHA, module versions, plan artifact identifier, policy results, environment, requester, approver, and deployment trace. Do not expose sensitive plan files to untrusted pull-request contributors. Protected branch and CODEOWNERS rules require platform review for shared modules and policy files.
For production, a fresh plan is generated after merge against current state, then an approved, immutable plan artifact is applied by the corresponding identity. An expired or superseded plan is discarded and regenerated. CI jobs must not apply a plan generated for a different commit, state snapshot, workspace, or identity scope.
Deployment gating includes a manual decision for destructive replacements, changes to route tables/NSGs/private DNS, loss of encryption controls, or privilege expansion. Terraform plans identify expected resource operations, but cannot fully prove application compatibility or availability; smoke tests and service checks are mandatory after deployment.
7. Policy-as-Code and Runtime Governance
Pipeline controls check the planned change, while Azure Policy verifies resources at their effective scopes. Example gates include: disallow public storage access unless explicitly exempted, require approved regions, flag unrestricted inbound administrative ports, validate required resource ownership tags, and ensure diagnostics are provisioned for supported resource types. Not every property can be validated through a single Terraform plan rule, and some policy effects require existing resources to be remediated after assignment.
Policy deployments follow audit/reporting, impact analysis, exception review, and gradual deny/modify/deployIfNotExists activation as appropriate. Exemptions have owners, business justification, scope, and an expiry/review date. Every policy control maps to an Azure-native enforcement capability or a compensating operational check; broad "compliant" claims are avoided without evidence.
8. Existing Resource Adoption Without Re-creation
Before onboarding portal-managed production resources, engineers inventory resource IDs, dependencies, identity assignments, DNS, networking, SKU details, monitoring settings, and deletion protection. They define import blocks or use controlled terraform import, then align configuration to the existing resource shape until plan produces no unapproved replacement. Resource addresses and moved blocks are documented when refactoring state, and imported resources are validated against runtime behavior before the first write deployment.
High-impact resources (databases, Key Vault, private endpoints, firewall rules) are brought under management in separate change sets. A no-op import is not sufficient if unmanaged children, inherited permissions, or drift outside provider coverage remain. Backups and rollback are documented before modifying state or applying infrastructure changes.
9. Drift Detection, Break-Glass, and Rollback
Scheduled read-only plans detect drift against the desired configuration. Drift tickets capture resource IDs, expected state, actual state, change actor where observable, and disposition: restore declared configuration, formally accept and commit the change, or escalate for incident response. Continuous auto-apply is disabled for production drift remediation.
A break-glass workflow requires a ticket, elevated access with expiry, recorded portal/CLI changes, and post-incident reconciliation back into code. Rollback differs from application rollback: reverting a Git commit may produce destructive infrastructure changes. Operators review the reverse plan and use service-specific restoration for deleted or corrupted resources. The recovery plan includes Terraform state history and backups, application data backups, and re-provisioning dependencies.
10. Monitoring and Service Operations
Every deployment emits audit events into the pipeline platform and Azure activity/diagnostic logs. Shared monitoring covers failed deployments, service health, policy noncompliance, suspicious role changes, state-store access anomalies, connectivity changes, and workload telemetry. Action groups route to assigned responders; noise reduction is handled through severity, suppression windows, and runbooks. Cost tags identify owners but must be validated before using them for chargeback.
11. Production Acceptance and Handover Checklist
| Acceptance test | Evidence required | Pass condition |
|---|---|---|
| Environment parity | Parameter and module version comparison | Intentional deviations documented and approved |
| Least-privilege deployment | Role assignment review; test denied action | Non-production pipeline cannot write production |
| State integrity | Lock contention test; recovery rehearsal | Concurrency controlled and state restorable |
| Private service connectivity | Name resolution and TCP/application test from approved runner | Correct private address and successful authenticated request |
| Policy controls | Sample compliant and noncompliant deployment tests | Expected audit/deny/remediation behavior |
| Release safety | Plan approval and deployment audit trail | Approved artifact tied to commit and environment |
| Drift process | Controlled out-of-band change | Detected, triaged, reconciled |
| Recovery | Documented resource/state restore exercise | Recovery procedures verified against stated objectives |
Completion requires signed ownership, runbooks, escalation paths, IaC repository permissions, monitoring routes, and a prioritized backlog; pipeline success alone is not production acceptance.
Before and After Implementation: Operational Improvements
The following describes the architecture's expected operating-state changes, not independently measured Contoso results. Quantitative impact should be reported only after baseline and post-rollout measurement.
| Dimension | Before implementation | After implementation (target state) | Benefit / verification method |
|---|---|---|---|
| Provisioning | Engineers create and update resources manually | Version-controlled, approved Terraform/Bicep workflows | Compare median ticket-to-ready time over equivalent workloads |
| Configuration consistency | Differences across dev, test, and production | Shared versioned modules plus controlled environment variables | Compare unapproved parameter/resource differences |
| Change transparency | Incomplete portal/script history | PR, plan, approval, deployment and commit linkage | Audit sample of production changes |
| Production access | Broad standing permissions for deployment operators | Scoped federated deployment identities and protected approvals | Review standing privileges and role-assignment changes |
| Terraform state | Fragmented local or inconsistently secured state | Segregated remote state with locking, RBAC, retention and restore | State-access audit and restore exercise |
| Security controls | Manual checks near release | Static checks, policy gates, and Azure Policy assignments | Count prevented violations and time to remediation |
| Release quality | Ad hoc operations and no standard readiness checks | Test gates, review controls and post-apply validation | Change failure rate and rollback incident count |
| Drift | Portal changes remain unnoticed | Scheduled read-only drift detection and ticketed correction | Drift detection lead time and outstanding drift age |
| Workload onboarding | Bespoke setup for each application | Standard repository/module templates and owner assignment | Request-to-first-approved-deploy time |
| Recovery | Individual knowledge and manual restore | Documented state recovery and service-specific runbooks | Restore exercise results and RTO/RPO evidence |
| Cost ownership | Inconsistent tagging and budgets | Required ownership metadata and budget monitoring | Percentage of spend mapped to accountable owner |
| Operational support | Escalation depends on individuals | Runbooks, action groups and ownership matrix | Incident handoff completeness and time to acknowledge |
Measurement and Claims Standard
Record baseline and after-period dates, workload scope, sample sizes, exclusions, data sources and metric owners. Suggested evidence includes CI execution records, Azure Activity Log, Azure Policy compliance exports, Terraform plan artifacts, incident tickets, and Azure Cost Management. Avoid statements such as "deployment time improved 80%" or "zero drift" without those measurements.
Project Deliverables
A typical engagement can include:
- IaC architecture and standards
- repository structure
- reusable Terraform/Bicep modules
- remote state design
- environment configuration
- GitHub Actions or Azure DevOps pipelines
- workload identity federation
- policy-as-code controls
- security validation
- documentation
- deployment runbooks
- migration plan for existing infrastructure
Expected Architectural Outcomes
The infrastructure platform is designed to become reproducible, reviewable, and easier to operate.
Instead of relying on manual cloud configuration, teams work from a declared desired state stored in source control.
Contoso’s target operating model provides:
- consistent infrastructure patterns
- clearer change history
- stronger deployment controls
- reduced configuration drift
- reusable platform components
- simpler onboarding for new workloads
- better separation between platform and application responsibilities
- a foundation for controlled self-service
Most importantly, infrastructure delivery becomes an engineering process rather than an accumulation of manual cloud changes.
Technologies
Terraform · Bicep · Microsoft Azure · GitHub Actions · Azure DevOps · Workload Identity Federation · Open Policy Agent · Conftest · Azure Policy · Key Vault · CI/CD
Related Patterns
Next step
Discuss a similar Platform Engineering challenge.
Share your current architecture, constraints and desired next step. Tongtech can help determine whether a focused review or implementation engagement fits.
Book an Architecture Review