“For BDT, this efficiency gain translates to an annual saving of over 220,000 years of EC2 vCPU computing time.”
Amazon moved compaction for its biggest tables off Spark onto Ray and pushed over 1.5 EiB of Parquet through it in Q1 2024. The largest 1% of tables carried about 40% of the Spark compaction bill. Ray claims about 82% better cost efficiency per GiB with memory utilization averaging 54.6%. Find the 1% of jobs that eat 40% of the budget before rewriting anything else.