Spark partition skew: 1 worker overloaded, others idle

Ситуация: Spark job: 1 worker doing 80% work, 200 others idle. Job duration 2 hours instead 30 min. Skew killing throughput.

Spark job: aggregation by user_id. Long-tail users (top 0.1%) huge volume. Skew = uneven partition distribution.

Доступные данные

• spark_metrics: job_id, stage, task_id, duration
• data_distribution: key, rows
• job_logs: ts, type, message

Задачи

• Skew diagnosis.
• Solutions: salt + re-aggregate, repartition, AQE.
• Cost-benefit.
• Long-term: data modeling.

Все кейсы для подготовки → ← Все кейсы