Building Secure AI Infrastructure and Production AI Applications on Azure

Executive Summary

Business context — Contoso: Contoso operates internal knowledge services across teams with distinct data ownership and access requirements. Its existing AI application architecture requires stronger permission enforcement, secure model connectivity, repeatable evaluation, and accountable production support.

Moving an AI prototype into production requires more than access to a model endpoint. Organizations need a secure application foundation, governed access to data and models, controls for retrieval and tools, a repeatable release process, and a way to evaluate quality and operations over time.

This architecture describes how Tongtech designs the Azure platform and application together. A governed AI foundation provides identity, network, policy, and monitoring controls. Application services integrate model access with authorized enterprise data, retrieval grounding, and bounded agent tools. Evaluation and operational telemetry are included in the delivery lifecycle.

The objective is to make an AI use case supportable in production while keeping data permissions, human decision points, model changes, and operating costs visible to the teams responsible for the service.

Client reference: Contoso is used as a generic, anonymized business name. The design documents a production implementation pattern; project completion and measured results must be substantiated separately.

The Challenge

A minimally governed AI implementation can use broad credentials and direct model calls. Production operation requires additional controls:

• Enterprise data has different owners, classifications, and user permissions.
• Model access must follow identity, network, capacity, and regional requirements.
• Retrieval must return relevant, current content without exposing documents a user cannot access.
• Model responses can be incomplete or incorrect even when retrieval succeeds.
• Agents that call tools can perform actions beyond a simple read unless their permissions and approvals are bounded.
• Prompts, model deployments, retrieval settings, and application code change on different schedules.
• Token consumption, latency, service errors, and retrieval quality need ongoing visibility.
• Production teams need a clear process for evaluation, release, rollback, and incident response.

The organization needs an architecture that treats identity, data governance, application behavior, and operations as part of one system.

Objectives

The target design is intended to:

1. Establish a governed Azure foundation for AI workloads, data access, and model endpoints.
2. Authenticate users and workloads through Microsoft Entra ID and scoped permissions.
3. Ground model responses in authorized enterprise content and return useful source references.
4. Limit agent tools to approved operations with appropriate human review for sensitive actions.
5. Manage prompts, model configuration, infrastructure, and application code through controlled promotion.
6. Evaluate retrieval, grounding, safety, latency, and cost before and after release.
7. Provide operational monitoring, incident ownership, and a path for ongoing improvement.

AI Landing Zone

AI workloads use subscriptions or resource groups that fit the organization’s landing-zone and ownership model. The foundation defines network paths, private connectivity where required, private DNS, policy, diagnostic settings, cost ownership, and access boundaries for application, model, and data services.

Microsoft Entra ID and managed identities provide identity-based access where supported. Azure Key Vault stores secrets that cannot be eliminated through managed identity. RBAC is scoped to the specific model, search index, storage location, API, or application resource required by each workload.

The design records which endpoints are public or private and how users, application services, model services, data sources, and operations teams reach them. Network isolation is validated end to end so that a private endpoint does not accidentally break a required service-to-service path.

Model and AI Platform

Microsoft Foundry provides a governed place to evaluate and work with model deployments. Azure OpenAI in Microsoft Foundry Models can be selected where its model capabilities, availability, and deployment options fit the use case. Model selection considers task quality, latency, context needs, regional availability, quota and capacity, content-safety requirements, and cost.

Applications access models through a controlled backend or API boundary rather than exposing model credentials to browsers or end users. Model names, deployment identifiers, parameters, safety settings, and prompt versions are configuration that moves through review and promotion. Capacity planning includes expected request volume, concurrency, and operational limits; availability and quota are confirmed for the target region and deployment at implementation time.

Azure AI Content Safety and application-level safeguards can help enforce content policies. These controls are tested against the use case and do not replace authorization checks, input validation, or human review for consequential actions.

Retrieval-Augmented Generation

For knowledge-grounded use cases, the retrieval path ingests approved source material, extracts and normalizes content, segments it into useful passages, and indexes text, vectors, and metadata in Azure AI Search. Metadata can include source, owner, classification, effective date, and permission information needed by the application.

At query time, the application authenticates the user, applies the relevant authorization context to retrieval, selects concise and relevant passages, and supplies those passages to the model as grounding material. The response can include source titles or links so users can inspect the material used. The application treats model output as generated content and does not represent a citation as proof that a claim is correct.

The ingestion pipeline manages updates, removals, duplicate content, and index refresh. Authorization metadata is maintained with the source system’s permission model and tested for both allowed and denied users. This avoids treating search-index access as a substitute for user-level document authorization.

Agent Architecture

An agent is introduced only where coordinating multiple steps or tools adds value beyond a direct model-and-retrieval flow. The application defines a bounded tool set, with explicit input schemas, authorization checks, logging, and service-side validation for every action.

Model Context Protocol (MCP) can provide a structured way for an agent to discover and invoke approved tools. Each tool connection uses an authentication method appropriate to its service and user-context requirements. Managed identity or agent identity is preferred where supported; credentials are not placed in prompts or exposed to the model.

Tools begin with the minimum access needed, and high-impact operations require an explicit user confirmation or human approval in the application workflow. The server verifies permissions independently of model instructions, records tool requests and results, and provides a safe failure path. Tool allowlists, rate limits, and data boundaries are reviewed as part of release governance.

Application Layer

The application typically separates a user-facing experience from a backend API that handles authentication, authorization, retrieval, model calls, and tool orchestration. Azure API Management can apply API authentication, quotas, and policies at an API boundary where appropriate.

The hosting platform is selected according to runtime, scaling, networking, and operational requirements. Azure App Service or Azure Container Apps may fit managed web and container workloads; AKS is selected where Kubernetes capabilities and its operating model are required; Azure Functions can handle bounded event-driven tasks. Conversation or session state is stored in an approved data service with retention, access, and deletion behavior defined for the use case.

Microsoft Entra authentication establishes the user identity, while the backend enforces authorization for each data source and operation. The model does not decide a user’s access rights.

Security and Safety

Security controls cover the full request path:

• Model, search, storage, and application access use scoped identities and RBAC.
• Private Link and private DNS are applied where required by the network design and service support.
• Secrets are held in Key Vault and retrieved through approved identity-based access.
• Retrieval enforces document permissions and avoids mixing content across security boundaries.
• Prompt injection and untrusted retrieved text are handled as data, with explicit tool boundaries and server-side authorization.
• Outputs are validated for format, safety, and application-specific constraints before use.
• Agent actions that affect business records or external systems require appropriate user confirmation or approval.
• User inputs, retrieved content, model responses, and tool traces are logged according to data classification and retention requirements.

No single prompt, filter, or model setting provides complete security. The architecture layers identity, network, data authorization, application validation, and operational oversight.

Evaluation and Observability

Evaluation covers both model output and the retrieval/application path. A representative evaluation set includes expected user intents, relevant source documents, authorization scenarios, and cases where the system should abstain or ask for clarification.

Teams review retrieval relevance, grounding support, citation quality, response format, safety behavior, and tool selection. Changes to prompts, models, chunking, ranking, or filters are compared against the same evaluation set before promotion. Human review remains part of quality management for high-impact or ambiguous outputs.

Azure Monitor and Application Insights capture application telemetry and distributed traces. Operational views include request latency, model errors, search failures, tool outcomes, throttling, and token consumption where available. Logs are designed to avoid unnecessary capture of sensitive prompts or retrieved documents. Cost ownership is assigned across the application, model, and search resources, with usage reviewed against expected demand.

CI/CD and Infrastructure as Code

Terraform or Bicep provisions the AI foundation and application dependencies. CI/CD validates infrastructure, application code, prompts, retrieval configuration, and policy controls through reviewed pipelines. Model and prompt configuration are promoted in a controlled way so that a tested application revision can be associated with the model and retrieval settings used during evaluation.

Development, test, and production environments separate data, identities, permissions, and endpoints according to the risk model. Automated checks cover application tests, retrieval and evaluation cases, secret scanning, dependency review, infrastructure validation, and security policies. Production release approvals are applied where required, with a documented path to revert application or configuration changes.

Implementation Sequence

1. Use-Case and Data Assessment

Define user needs, expected decisions, data sources, information classifications, permission model, model constraints, acceptable risk, and evaluation criteria. Confirm whether retrieval or agent tools are necessary for the use case.

2. AI Landing-Zone Design

Confirm subscription and ownership boundaries, identity, RBAC, network paths, private access requirements, Key Vault, diagnostics, policy, and cost allocation for the AI workload.

3. Model and Retrieval Integration

Select candidate model deployments, build a representative authorized retrieval path, define metadata and citations, and create evaluation cases that include relevant and denied content.

4. Application and Agent Controls

Implement the API and user experience, enforce user authorization, add only the required tools, and validate content, output, and human-approval controls.

5. Evaluation and Production Readiness

Run quality, security, performance, operational, and cost reviews. Confirm monitoring, alert ownership, incident procedures, data retention, and release rollback steps.

6. Controlled Release and Improvement

Promote the approved infrastructure, model configuration, prompts, and application through environments. Monitor quality and service behavior, then use observed feedback to prioritize changes through normal review.

Representative Deliverables

A typical engagement can produce:

• AI use-case and data-access assessment
• Azure AI landing-zone and network architecture
• model selection and capacity assumptions
• Azure AI Search ingestion, indexing, retrieval, and authorization design
• prompt, grounding, and citation patterns
• agent tool, identity, and approval boundaries where agents are needed
• application and API architecture
• Terraform or Bicep and CI/CD pipelines
• evaluation datasets, quality checks, and release gates
• observability, cost views, and operational runbooks
• security review and production-readiness documentation

Production Reference Architecture and Deployment Boundaries

Contoso organizes the AI service as a production workload rather than a standalone model endpoint. Platform resources have named owners, separate deployment identities, explicit routing, and independent recovery requirements. The Azure tenant retains central governance while the workload team owns application code, prompt versions, and operating thresholds.

Network topology and traffic paths

The production network design uses a hub-and-spoke connectivity model where required by the enterprise landing zone. A dedicated workload spoke hosts the application integration subnet, private endpoint subnet, and any runtime components that require VNet integration. Private DNS zones for applicable Search, Storage, Key Vault and model endpoints are linked to the resolving VNets, with DNS forwarding centrally managed. VNet integration is outbound connectivity; it does not by itself make the application's inbound endpoint private. Inbound exposure is controlled separately using private endpoints, access restrictions, or an approved ingress pattern.

The expected request sequence is: employee -> approved HTTPS ingress -> authenticated API -> authorization resolution -> filtered retrieval -> model inference -> output validation -> response with citations. The ingestion route is separate: approved source -> extractor -> normalization/chunking -> permission metadata -> staged index build -> validation -> index promotion. Model inference does not have read permissions to source repositories, and the browser has neither Search administrative access nor model secrets.

A production readiness test includes DNS resolution from the actual runtime subnet; private endpoint connectivity; outbound dependency reachability; deny-by-default behavior when directory resolution fails; and route/NSG review for each endpoint. Shared egress policy and firewall routing are validated rather than assumed from a diagram.

Representative application and IaC layout

contoso-ai-platform/
  platform/{network,dns,identity,policy,monitoring}/
  environments/{dev,test,prod}/
  modules/{app,search,model,storage,key-vault,private-endpoint}/
  services/{api,ingestion,evaluation}/
  prompts/{system,templates}/
  evaluations/{datasets,scorers,regression}/
  pipelines/{infra,app,ai-release}/
  runbooks/{incident,revoke-source,quota-failure,rollback}/

IaC state is isolated by environment and service ownership. Production applies use approved identities and protected environments; a plan is reviewed for destructive changes. Key Vault access uses data-plane roles, with user and workload permissions separated. Service tags, retention policies and Azure Policy assignments are enforced through the landing-zone hierarchy, while application-level authorization remains a code concern.

Retrieval and Agent Implementation Controls

A chunk record includes source_id, source_version, chunk_id, title, source_uri, classification, last_modified_utc, content_hash, and the authorization principals or equivalent source-specific security attributes. The query API resolves the caller's effective rights and constructs mandatory server-side filters. It denies retrieval if rights cannot be resolved, and tests cover direct search, reranking, caching, retries, index aliases and supplementary lookup endpoints. Search API keys must not be exposed to clients. Permission sync failures quarantine or exclude affected sources until reconciled.

The assistant returns a response only if the retrieved evidence passes application-defined relevance and authorization checks. Citations link to source documents that remain subject to their own access controls. Source text is treated as untrusted content; prompt injection cannot confer extra roles or turn on tools. The inference service bounds token budgets, timeouts, concurrent calls, retry limits and the maximum number of retrieved passages. It records deployment IDs and retrieval versions for troubleshooting, not the full documents in general-purpose logs.

Agent tools are enabled only for an approved business workflow. Each tool has an allowlisted operation, typed input validation, service-side authorization, rate limits and an auditable action result. Write actions require confirmation and, for consequential operations, a separate authorization or approval step. Failure to authorize returns a denial, not a fallback to the model's judgment.

Detailed Production Delivery and Verification

1. Foundation: Provision target subscriptions/resource groups, Azure Policy, private DNS, private endpoints, managed identities, data-plane role assignments, diagnostics and cost budgets through reviewed IaC.
2. Data contracts: Inventory source owners and classifications, specify deletion/revocation SLAs, approve chunk metadata schema and test index synchronization under group membership changes.
3. Runtime: Deploy backend with request authentication, authorization filters, model gateway, bounded retries, output validation, and health probes distinguishing readiness from liveness.
4. CI/CD: Run IaC validation, secret detection, dependency scanning, application tests, retrieval and negative-authorization regression tests, then promote the approved artifact and pinned AI configuration.
5. Operational rehearsal: Exercise model throttling, Search outage, stale ACLs, document withdrawal, expired credentials, DNS failures and rollback; retain test evidence and named action owners.
6. Go-live: Require security sign-off for retrieval boundaries, application-owner sign-off for task quality, and operations sign-off for alerts, capacity, backup and recovery paths.

Acceptance evidence and service objectives

Production acceptance evidence includes a successful forbidden-document retrieval test for every sensitive source class (success means no disclosure), verified source deletion and ACL-revocation propagation within the agreed service window, a recorded restore/rebuild procedure for the index, and reproducible deployment manifests. Define and baseline p95 end-to-end latency, error rate, citation validation, grounded-answer acceptance, abstention rate, token spend per successful task and support escalation. Targets are contractual or operational decisions, not assumed results. Alerting covers sustained 429/5xx, latency breaches, ingest backlog, stale index builds, and abnormal token consumption.

Before and After Implementation: Operational Improvements

The table documents the intended operational state; it does not assert that Contoso realized quantified savings or completed a verified client engagement.

Outcome

The result is a production-oriented AI application pattern in which cloud foundations, model access, data authorization, retrieval, tools, evaluation, and operations are designed together. Teams gain a controlled path from an AI use case to an application that can be reviewed, released, monitored, and improved.

Why This Architecture Works

The cloud foundation supports the application’s security and operating needs.

Model responses use authorized data and visible source references.

Agent tools are bounded by service-side permissions and approval rules.

Evaluation covers retrieval, grounding, safety, and operational behavior.

Infrastructure and AI configuration move through controlled releases.

The result is an AI application architecture that connects a useful user experience to the identity, data, model, security, and operational controls needed for production.

Technologies

Microsoft Azure · Microsoft Foundry · Azure OpenAI in Microsoft Foundry Models · Azure AI Search · Azure API Management · Azure App Service · Azure Container Apps · Azure Kubernetes Service · Azure Functions · Microsoft Entra ID · Managed Identity · Azure Key Vault · Private Link · Azure AI Content Safety · Azure Monitor · Application Insights · Terraform / Bicep

Related Patterns

• AI security and identity guide
• Extending landing zones for AI