The Business Data Technologies (BDT) team at Amazon Retail is bravely migrating their massive business intelligence datasets from Apache Spark to Ray to increase data processing efficiency and reduce costs. They have contributed a critical component to Ray’s open-source project and have found that Ray outperforms Spark in terms of scalability and cost-effectiveness. By using Ray, they have successfully tackled challenges with compaction jobs and data quality insights. Despite the risks involved, BDT has achieved impressive results, with Ray compacting over exabytes of data efficiently and maintaining a high on-time delivery rate to table subscribers. The transition from Spark to Ray has proven to be a successful and rewarding journey for BDT.
https://aws.amazon.com/blogs/opensource/amazons-exabyte-scale-migration-from-apache-spark-to-ray-on-amazon-ec2/