Backfill 6 months data: orchestrate without affecting production
Средний
Data Engineering
50 мин
Backfill orchestration
Ситуация: New computed metric needs backfill 6 months historic. Cluster shared с production. How orchestrate без breaking SLAs?
ETL Airflow. Backfill = re-process historic dates. Cluster: 200 cores shared. Production SLA-bound.
Доступные данные
airflow_runs: task_id, ts, status, runtime
cluster_metrics: ts, cpu_used, memory_used
sla_tracking: workflow, target_time, hit
Задачи
- Backfill strategy: parallel vs serial.
- Resource budget: production reservation.
- Monitoring: SLA impact.
- Rollback plan.
Все кейсы для подготовки →
← Все кейсы