Thursday, February 4, 2016

Reproducible Stream Processing Benchmark to compare Apache Spark and Apache Flink on Cloud

Stream processing is becoming a crucial requirement with the high volume of data generated and the need for real-time processing of those data. And the data processing platforms are trying to provide smart and efficient approaches for stream processing. 

Yahoo has recently published a stream processing benchmark and has published the resources here to run the experiment in a single node. Since benchmarking stream processing is an interesting and important task, we wanted to reproduce the experiment on a clustered setup. 

We created a reproducible experiment on Amazon EC2 to reproduce yahoo streaming benchmark on a cluster of Apache Spark and Apache Flink. You can find all the resources and in instruction here which will help you to reproduce our experiment. And more importantly, you can reproduce the experiment with different configurations conveniently by following the instruction in above-mentioned link.

Following are some of the application-level and system-level performance results that we obtained during the experiment.

Application-level performance:

System-level performance



The configurations and explanation of the results can be found in the stream processing evaluation section of full report.

We have completed a performance comparison of batch processing as well for Apache Spark and Apache Flink to reproduce DongWong’s performance comparison.

The full project report can be found here.


    Jim Dowling
    Kamal Hakinzaheh
    Shelan Perera


  1. Wonderful post !!! Genuinely loved this kind of post.
    - Lenny Face

  2. Video streaming is likewise represented by different conventions that bring under thought specialized execution, quality issues, dependability, cost elements, and lawful and social iptv service 2019

  3. Prescient information investigation utilizes this data to channel current just as verifiable information to recognize examples and figure occasions and conditions that may happen later on at a specific time, given gave parameters.
    Data Analytics Course in Bangalore

  4. nice article!! try to create an article on digital marketing and post. that would also help people in doing the course.and one institute is good in offering certified courses that is 360DigiTMG.if u want u can refer that institute.

  5. This comment has been removed by the author.

  6. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. gostream

  7. This is not a position of power or a title, but a role for somebody who has the right mindset, attitude and ambition to change things, and who wants to make a difference and to set new standards. Salesforce training in Hyderabad

  8. Really nice article. I appreciated your effort in this article. I bookmarked your website for further posts. Keep sharing!
    Data Science Course in Hyderabad

  9. wow, great, I was wondering how to cure acne naturally. I found your site on Google, learned a lot, and now I'm a bit clearer. I’ve bookmarked your site and also added rss. keep us updated.
    data science course in hyderabad