Data, AI & Integration · Standard

Data Platform Resilience, Backup, and Disaster Recovery Standard

Defines tiered recovery objectives, protected backups, replication, reconstruction, failover, and evidence requirements for data and AI platforms.

disaster-recoverybackupdata-resilienceransomware-protectionbusiness-continuitydataintegrationplatform

Document class: Data, AI & Integration standard Normative terms: MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are requirements levels. Applicability: Data and AI platform protection from deletion, corruption, regional failure, ransomware, operator error, and dependency loss. Exception process: Deviations require a documented risk assessment, compensating controls, named risk owner, and expiry date.

Control fieldValue
Document IDDAI-14
OwnerCloud Center of Excellence
Review cycleAt least annually and after material platform, provider, data, security, or operating-model changes
EvidenceProtection inventory, backup policy, restore tests, recovery exercise, and operational readiness evidence

Decision in brief: Design availability, replication, backup, and disaster recovery as separate but coordinated capabilities, and prove recovery through exercises.

Purpose

This standard protects data platforms from deletion, corruption, regional failure, ransomware, operator error, and dependency loss. High availability, replication, backup, and disaster recovery solve different problems and MUST be designed together.

Recovery model

flowchart TB A[Authoritative data] --> R[Replication for availability] A --> B[Immutable isolated backup] C[Code, schemas, policy, IaC] --> X[Reconstruct platform] M[Catalog, lineage, identity, keys] --> B R --> F[Fast failover] B --> S[Point-in-time restore] X --> S S --> V[Reconcile, validate, resume consumers]

Tiering

Each product MUST declare maximum tolerable outage, RTO, RPO, recovery region, minimum retained restore points, and recovery owner. Dependencies must meet or exceed the product objective.

TierTypical useArchitecture expectation
CriticalSafety, regulated, revenue-criticalMulti-zone, regional recovery, isolated backup, frequent exercise
ImportantCore analytics and operationsAutomated restore, tested regional plan
StandardRebuildable productsProtected source and configuration, scheduled restore test
EphemeralDevelopment/cacheRecreate from source; no unsupported recovery promise

Protection scope

Protect raw and curated data, databases, object versions, stream retention where required, schemas, catalogs, lineage, orchestration state, checkpoints, model artifacts, feature definitions, semantic models, policies, configurations, IaC, keys or documented key recovery, and release evidence.

Replication MUST NOT be treated as backup because corruption and deletion can replicate. Keep at least one logically isolated, access-restricted, immutable copy for critical data. Separate backup administration from workload administration.

Provider mapping

CapabilityAzureAWSGCPOCI
Backup orchestrationAzure Backup/service-native backupAWS Backup/service-native backupBackup and DR/service-native backupOCI Backup/service-native backup
Immutable object protectionBlob immutability/versioningS3 Object Lock/versioningBucket Lock/versioningObject Storage retention/versioning
Regional replicationService-specific geo replicationCross-Region Replication/servicesDual/multi-region and service replicationCross-region replication/services
AutomationAzure DevOps/GitHub/IaCSystems Manager/GitHub/IaCCloud Build/GitHub/IaCOCI DevOps/GitHub/IaC

Recovery sequencing

Recover identity and keys, network and DNS, storage, catalog and policy, orchestration, compute, products, and consumers in dependency order. Reconciliation MUST compare recovered counts, checksums, control totals, quality results, model versions, and last processed offsets before traffic resumes.

Testing

Perform component restores regularly and full service recovery exercises according to tier. Include unavailable primary credentials, compromised administrator, corrupt recent backup, regional isolation, missing key, schema mismatch, and downstream replay. Record actual RTO/RPO, manual steps, data loss, cost, defects, and corrective owners.

Validation

Confirm every critical asset appears in the protection inventory; backup success is monitored; retention is enforced; workload administrators cannot delete protected copies; clean credentials can restore; and recovery works in an isolated environment. Track backup gaps, untested assets, restore success, recovery objective attainment, immutable coverage, and overdue corrective actions.

Operational considerations

Data-product owners define criticality and validate restored meaning. Platform teams automate protection and recovery. Security controls isolation and ransomware response. Finance approves sustained recovery capacity. Review provider region dependencies, egress time and cost, data sovereignty, key recovery, capacity reservations, and emergency access.

Failure-Domain and Dependency Matrix

Recovery design MUST identify dependencies that can fail independently. At minimum, assess identity, key management, DNS, private connectivity, control plane, object storage, database, stream, catalog, orchestration, model registry, artifact registry, and observability.

DependencyLoss scenarioRecovery requirement
Identity federationEnterprise sign-in unavailablecloud-local emergency access with audited use
Key servicekey or permission unavailableprotected key recovery and tested decrypt path
Cataloggrants and metadata unavailablereconstruct or restore before governed access resumes
Object storageregional or logical corruptionisolated version or replica plus integrity checks
Stream checkpointoffsets lost or corruptcontrolled replay from retained source
Orchestratorschedules and state unavailablerebuild from version control and reconcile active runs
Model or artifact registryreleased asset unavailableimmutable replicated artifact and provenance record

The declared RTO is valid only if the slowest critical dependency can meet it.

Backup Isolation and Ransomware Controls

Critical backups MUST use an administrative boundary separate from ordinary workload operators. Protection SHOULD include immutability or write-once retention, multi-party deletion controls where available, dedicated backup identities, restricted network paths, anomaly detection, and independent inventory.

Do not use the same broadly privileged automation identity for production mutation and backup deletion. Emergency restoration credentials SHOULD be stored and exercised independently from normal federation and CI/CD.

Recovery Exercise Acceptance Criteria

A recovery exercise is successful only when the recovered service is usable and trustworthy. The exercise record SHOULD include:

  • initiating scenario and assumed unavailable dependencies;
  • selected restore point and evidence that it predates corruption;
  • actual platform reconstruction time;
  • actual data restore and replay time;
  • data-loss interval and last confirmed transaction or offset;
  • record counts, checksums, control totals, schema versions, and quality results;
  • re-established identities, policies, alerts, and audit export;
  • consumer validation and business-owner acceptance;
  • failback or steady-state transition plan;
  • defects, owners, and remediation dates.

A test that restores a file but does not resume the data product does not validate the product RTO.

Recovery Capacity and Cost

Recovery regions and environments require enough quotas, network throughput, storage operations, compute, model capacity, and database limits to meet the objective. Document whether capacity is continuously provisioned, reserved, pre-approved, or obtained on demand.

Recovery cost forecasts SHOULD include retained replicas, immutable storage, transfer, temporary dual operation, high-priority compute, re-indexing, re-embedding, reconciliation, and business validation.

References