Scenario context
This technical case narrative describes a production-oriented design. It does not verify that a client engagement was completed or that project-specific results were achieved.
Starting conditions
In the fictional engineering organization, infrastructure and application deployments use separate repositories, inconsistent checks, and long-lived credentials, with unclear promotion and recovery. This is an illustrative starting condition, not a verified client baseline.
Executive Summary
Contoso business context: Contoso maintains separate infrastructure and application repositories with inconsistent validation, deployment credentials and approval paths. A production release requires traceability from source revision through build artifacts, infrastructure changes, identity and post-deployment checks.
When infrastructure and application releases rely on separate manual processes, teams can struggle to understand what changed, reproduce an environment, or move a tested release safely into production. Pipeline automation helps when it connects source control, validation, security, deployment identity, approvals, and post-deployment checks into one traceable delivery path.
This case study specifies Tongtech’s production CI/CD architecture for Contoso, covering infrastructure and application delivery with reproducible artifacts, scoped deployment identities and governed releases.
The objective is to make frequent delivery repeatable and auditable while preserving a clear approval path for production and other high-impact changes.
Client naming: Contoso is a generic name used to protect business identity. The architecture and implementation details below form a realistic production delivery design; customer-specific deployment evidence and measured results are not included.
The Challenge
Cloud delivery processes often grow from project-specific scripts and manual steps:
- Infrastructure is changed through portal operations or scripts without a complete review trail.
- Validation steps vary between repositories and depend on individual engineers.
- Application artifacts are rebuilt separately for each environment, making provenance harder to establish.
- Deployment credentials are stored as long-lived secrets or shared across unrelated workflows.
- Security and policy checks happen late or inconsistently.
- Production approvals are either missing or applied as a blanket control that slows low-risk changes.
- Environment configuration and promotion rules differ between teams.
- Rollback expectations are unclear, especially when infrastructure or data schemas change.
- Operators have limited visibility into which source revision and artifact produced a deployment.
The organization needs a delivery workflow that applies consistent engineering controls without treating every infrastructure and application change as identical.
Objectives
The pipeline model is designed to:
- Make changes reviewable through protected repositories and pull-request workflows.
- Validate infrastructure and application code before deployment.
- Apply security and policy checks at useful points in the delivery path.
- Use short-lived federated identity and narrowly scoped Azure RBAC where supported.
- Build versioned artifacts once and promote them through controlled environments.
- Apply approvals where risk, business ownership, or compliance requires them.
- Verify deployment health and retain traceability for investigation and recovery.
- Keep infrastructure and application release responsibilities clear while coordinating dependencies.
Pipeline Architecture
Separate workflows can be used for infrastructure and application changes, with explicit dependencies when one release requires the other. Reusable templates establish common controls, while application-specific steps remain visible to the team responsible for the service.
Source Control and Environment Promotion
Repositories use protected branches, required review, and automated checks appropriate to the change. Shared workflows or pipeline templates provide consistent setup for validation, security scanning, artifact handling, and Azure authentication. Changes to shared templates receive the same review and version controls as application code.
Environment definitions make the promotion path explicit. Development and test deployments can run automatically after review, while production controls use environment protections, designated approvers, and deployment permissions scoped to the application or infrastructure boundary. The exact approval model reflects service criticality and change policy rather than requiring an extra manual gate for every low-risk action.
Application artifacts are immutable and versioned. A tested artifact is promoted to the next environment instead of rebuilt from a different code state. Deployment records connect source revision, build, artifact, target environment, approver where applicable, and final health result.
Secure Azure Authentication
Pipelines authenticate to Azure using OpenID Connect workload identity federation or the platform’s supported federated service-connection pattern. This avoids depending on client secrets that must be stored and rotated as long-lived credentials.
Azure RBAC grants only the permissions required for the pipeline’s job. Validation identities, non-production deployment identities, and production deployment identities are separated when that improves containment and review. State access for Terraform is limited to the required backend scope and is not implicitly granted to unrelated application pipelines.
Federation trust is scoped to the intended repository, branch, environment, or pipeline identity. Credentials and tokens are not printed into logs, and privileged access for pipeline administration remains distinct from routine workload deployment.
Infrastructure Delivery
Terraform or Bicep defines Azure infrastructure, with module and provider versions controlled in source. Pull-request validation formats and validates the code, runs security and policy checks, and produces a Terraform plan or Bicep What-If for review before applying changes.
Terraform state uses a controlled remote backend with appropriate access boundaries, locking, backup, and environment separation. Bicep deployments follow a similarly reviewable plan and deployment process. Environment-specific values are explicit and sensitive values are sourced from approved secret stores or identity-based access patterns rather than committed to the repository.
Policy-as-code checks can reject disallowed configurations before a cloud change is applied. Azure Policy remains a runtime and governance control; pipeline checks complement it by identifying issues earlier in the change lifecycle.
Application Delivery
The pipeline builds and packages the application using the delivery method for its target platform. This can include Azure App Service packages, Azure Container Apps images, AKS container images and deployment manifests, Azure Functions packages, or VM-based application configuration.
Builds produce identifiable artifacts, and container images are tagged with immutable versions or digests rather than relying on mutable labels alone. Secrets required by the application are retrieved through the workload’s supported identity and configuration model. Deployment steps include smoke tests or health checks that verify the application is ready after release.
Infrastructure and application releases coordinate API compatibility, database migrations, configuration changes, and shared-service dependencies. Schema changes are designed to support the required deployment overlap and recovery approach.
Security and Quality Gates
Security checks are selected for the code and artifact type and run early enough to provide useful feedback. Representative controls include:
- static application-code analysis and unit tests
- dependency and license review
- Infrastructure as Code security scanning
- policy-as-code checks with OPA/Conftest or equivalent tooling
- container image scanning and provenance checks
- secret detection and repository hygiene
- artifact signing or attestations where the delivery standard requires them
Findings have clear severity thresholds, ownership, and an exception process. Checks should block changes that violate required controls while allowing a documented and time-bound exception path for justified cases. Manual approval is reserved for decisions that need human ownership, such as production risk acceptance or a regulated release gate.
Release and Recovery
Release strategies are selected based on platform capability, service architecture, and the consequences of a failed deployment. App Service deployment slots, rolling updates, canary traffic, or blue-green patterns may be appropriate for different workloads. AKS releases can use versioned charts and a documented rollback procedure. Other platforms use their native deployment and health-validation mechanisms.
Application rollback is not always a safe response to infrastructure or data changes. Infrastructure plans are reviewed for destructive or replacement actions, state is protected, and recovery uses the supported IaC workflow rather than an assumed reverse operation. Database and data migration steps identify compatibility windows, backup points, and forward-fix or restore options before production execution.
After each release, health checks and telemetry inform whether the deployment is accepted, paused, or rolled back. Runbooks define who decides, what evidence is reviewed, and which actions remain safe under the current data state.
Implementation Sequence
1. Current-State and Delivery Assessment
Review repositories, build and deployment processes, infrastructure tools, Azure identities, environments, existing security checks, approval rules, artifact handling, and release ownership.
2. Target Workflow and Control Design
Define reusable pipeline components, branch and environment protection, identity boundaries, validation stages, artifact promotion, approval criteria, and release traceability.
3. Initial Production-Grade Repository and Azure Integration
Implement the workflow for a representative infrastructure or application repository. Configure federated Azure authentication, least-privilege permissions, environment boundaries, and artifact retention.
4. Validate Production Delivery and Recovery
Exercise pull-request checks, plan review, non-production deployment, production approval, health verification, and the documented recovery path. Resolve permissions and dependency issues before wider adoption.
5. Expand Reusable Patterns
Publish versioned workflow templates and onboarding guidance. Migrate additional repositories in manageable groups, adapting only where platform or application requirements differ.
6. Operate and Improve
Review failed deployments, security findings, approval delays, pipeline maintenance, and release evidence. Update templates and controls through normal code review and change management.
Delivery Artifacts
The production delivery scope includes:
- CI/CD target architecture and repository conventions
- reusable GitHub Actions workflows or Azure DevOps templates
- federated Azure authentication and scoped RBAC design
- Terraform or Bicep validation and deployment pipeline
- application build, artifact, and environment-promotion workflows
- security scanning and policy-as-code gates
- approval and production environment-protection model
- release traceability and deployment health checks
- rollback, recovery, and operational runbooks
- onboarding documentation and pipeline ownership model
Contoso Production Delivery Architecture
Contoso has separate application and cloud-infrastructure repositories with different owners and release cadences. The target design uses a common supply-chain and identity standard while retaining independent application and infrastructure workflows. A shared reusable pipeline does not grant every repository equivalent production permissions.
Release Domains and Trust Boundaries
| Boundary | Design decision | Production consequence |
|---|---|---|
| Source repositories | Branch protection, CODEOWNERS for sensitive paths, mandatory review and required status checks | A single developer cannot bypass agreed controls |
| Build | Ephemeral hosted or controlled self-hosted runner, pinned actions/tasks, restricted token permissions | Build identity is not a standing administrator |
| Artifact repository | Immutable container digest/package checksum with provenance metadata and retention | The same validated artifact moves between environments |
| Infrastructure deployment | Separate plan and apply stages, remote state, policy scans and scoped deployment identities | Production resource mutations require explicit review |
| Application deployment | Workload-scoped identity, service health tests, progressive rollout | Failed application deployment can be isolated from platform change |
| Production approvals | Protected environments/service connections and named approvers | Approval attached to a specific release/configuration |
| Operations | Central deployment timeline, alerts and incident/runbook linkage | Production releases can be reconstructed and investigated |
The organization selects GitHub Actions or Azure DevOps as the primary orchestration platform based on existing source-control governance and support arrangements. The following controls apply to either product; exact syntax differs and must be implemented for the chosen platform.
Low-Level Pipeline Implementation
Infrastructure Pipeline
Terraform plans contain potentially sensitive values; encrypt and scope access to plan artifacts, retain them briefly, and apply only when source revision, environment, provider configuration, state version and approval context match. Expired/stale plans are regenerated and reviewed. Plans generated against speculative PR code must not automatically be treated as approved production apply artifacts. For Bicep use template validation and What-If with the same review discipline, recognizing that What-If can have limits and cannot guarantee operational correctness.
Separate state for network foundation, security baseline and workload components as appropriate to lifecycle/ownership; protect Azure Storage backends with authenticated identities, controlled network paths, backend-locking semantics, recovery controls and restricted administrative access. Document bootstrap dependencies so deployment credentials and state storage can be recovered independently of a failed workload deployment.
Application Build and Artifact Promotion
For containers, pin deployment manifests to an immutable image digest; do not deploy mutable latest tags. For App Service, use a deployment slot when applicable, run warm-up and endpoint health checks, and explicitly identify whether slot-specific settings, database schema changes or network configuration could prevent safe swaps. For AKS, use Helm or GitOps with defined ownership of cluster-level and workload-level objects; require probes, resource limits, rollout status inspection and policy controls. For Functions and Container Apps, use deployment and traffic-management capabilities supported by the selected plan and runtime rather than assuming all Azure hosting choices have the same rollback semantics.
Workload Identity Federation and RBAC
Federated credentials bind trusted pipeline identity claims (repository/environment or project/service connection) to a specific Entra application or managed identity. Azure permissions are narrowly scoped to target resource groups/subscriptions and split production from non-production; Terraform backend Blob Data permissions are granted separately when needed. Scope GitHub Actions permissions minimally (for example id-token: write only on trusted Azure-authentication jobs), prevent untrusted PR workflows from obtaining production credentials, and restrict reusable workflow and environment configuration edits to reviewers with sufficient authority. For Azure DevOps, use workload identity federation service connections and environment checks with matching scope.
Do not persist Azure access tokens, storage keys or application secrets in logs or artifacts. Prefer workload managed identity to application credentials after deployment. Break-glass elevation follows privileged access review and is never hidden in a pipeline variable.
Quality and Security Gates
| Gate | Implementation and failure handling |
|---|---|
| Unit and contract tests | Required check before merge; retry only known infrastructure flakiness with captured evidence |
| SAST/dependency review | Block agreed critical-severity findings; document accepted risk with owner and expiry |
| Secret scanning | Reject exposed credentials and rotate any real leaked secret |
| Terraform/Policy-as-Code | Reject unapproved public exposure, location or identity patterns before apply |
| Artifact integrity | Require immutable digest/checksum and retention tied to release policy |
| Deployment validation | HTTP transactions, dependency health, telemetry, logs and synthetic checks |
| Production change risk | Differentiate routine application rollout from destructive state/data migration |
Release Recovery Runbook
- Identify release SHA, artifact digest, deployment environment, current app and database schema versions, and change approver.
- Inspect the failure scope and deployment health; pause progressive rollout and prevent conflicting new releases.
- Where rollback is safe, redeploy the last known-good immutable application artifact or revert the deployment slot/traffic weights.
- For Terraform infrastructure changes, identify resource replacements/destructive actions and use a reviewed corrective plan; never automatically revert state files or presume inverse plans are safe.
- For schema/data changes, use an approved forward-compatible migration or separately validated restore/reconciliation procedure. Determine the point beyond which returning to the old binary would corrupt data.
- Retain deployment records and incident timeline, verify service recovery, and open a follow-up action for the pipeline/control gap.
Operating Model and Measurable Acceptance
Every production release should have an identifiable commit, immutable artifact, successful required checks, controlled execution identity, approval if required, post-deployment verification and retention-compatible audit evidence. Test a rejected policy violation, a denied deployment identity, an unhealthy rollout and a recovery exercise before moving a critical service onto the delivery platform. Track DORA-style delivery indicators only after agreeing on definitions and data sources; the metrics describe real releases, not inferred gains from implementing pipelines.
Outcome
The result is a repeatable delivery path in which infrastructure plans and application artifacts move through review, validation, and environment promotion with clear identity and ownership boundaries. Teams gain a consistent place to see what was tested, approved, and deployed, while production controls remain aligned with service risk.
The operational outcomes above describe design capabilities; measured improvements must be substantiated by customer operating records.
Before and After Implementation: Delivery Performance and Control
This comparison sets out operational improvements enabled by the target architecture. The table is not a claim of observed Contoso delivery metrics.
| Dimension | Before implementation | After implementation / target-state benefit |
|---|---|---|
| Review path | Scripts and production changes can bypass peer review | Protected branches, CODEOWNERS and required checks |
| Build consistency | Build steps differ by team and environment | Versioned reusable workflows with explicit customization |
| Artifact provenance | Production rebuilt separately or deployed manually | Immutable artifact from reviewed commit promoted unchanged |
| Deployment identity | Long-lived or shared cloud credentials | Federated short-lived access with scoped Azure RBAC |
| Infrastructure releases | Manual updates and poorly captured impact | Versioned Terraform/Bicep plans and controlled apply |
| Security feedback | Checks late or optional | Required static, dependency, secret and policy checks |
| Environment promotion | Environment-dependent procedures | Automated, traceable staged progression and approval rules |
| Production deployments | Uncoordinated changes and variable verification | Documented release gates and automated health checks |
| Failure handling | Manual reconstruction of prior version | Known-good artifacts and tested service-specific rollback runbooks |
| Database changes | Rollback assumes schema is reversible | Compatibility windows and migration-specific recovery decisions |
| Auditability | Incomplete change-to-release linkage | Commit, artifact, identity, approver and outcome recorded |
| Engineering efficiency | Repeated custom pipeline setup | Maintained templates and repeatable onboarding |
Evidence to collect: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time, percentage of releases with artifact provenance, security-gate exception aging and manual deployment count. Define baselines and reporting windows before asserting improvements.
Why This Delivery Model Works
Source control provides a reviewable change history.
Automated checks catch issues before deployment.
Federated identity reduces reliance on long-lived credentials.
The same versioned artifact is promoted between environments.
Release and recovery decisions are visible and owned.
The result is a controlled delivery model that supports infrastructure and application change with consistent validation and production oversight.
Technologies
Microsoft Azure · GitHub Actions · Azure DevOps · Terraform · Bicep · Open Policy Agent · Conftest · Microsoft Entra ID · Workload Identity Federation · Azure App Service · Azure Container Apps · Azure Kubernetes Service · Azure Functions
Related Patterns
Next step
Discuss a similar Platform Engineering challenge.
Share your current architecture, constraints and desired next step. Tongtech can help determine whether a focused review or implementation engagement fits.
Book an Architecture Review