Thursday, February 4, 2016

Reproducible Stream Processing Benchmark to compare Apache Spark and Apache Flink on Cloud

Stream processing is becoming a crucial requirement with the high volume of data generated and the need for real-time processing of those data. And the data processing platforms are trying to provide smart and efficient approaches for stream processing. 

Yahoo has recently published a stream processing benchmark and has published the resources here to run the experiment in a single node. Since benchmarking stream processing is an interesting and important task, we wanted to reproduce the experiment on a clustered setup. 

We created a reproducible experiment on Amazon EC2 to reproduce yahoo streaming benchmark on a cluster of Apache Spark and Apache Flink. You can find all the resources and in instruction here which will help you to reproduce our experiment. And more importantly, you can reproduce the experiment with different configurations conveniently by following the instruction in above-mentioned link.

Following are some of the application-level and system-level performance results that we obtained during the experiment.

Application-level performance:

System-level performance



The configurations and explanation of the results can be found in the stream processing evaluation section of full report.

We have completed a performance comparison of batch processing as well for Apache Spark and Apache Flink to reproduce DongWong’s performance comparison.

The full project report can be found here.


    Jim Dowling
    Kamal Hakinzaheh
    Shelan Perera


  1. Wonderful post !!! Genuinely loved this kind of post.
    - Lenny Face

  2. Video streaming is likewise represented by different conventions that bring under thought specialized execution, quality issues, dependability, cost elements, and lawful and social iptv service 2019

  3. Prescient information investigation utilizes this data to channel current just as verifiable information to recognize examples and figure occasions and conditions that may happen later on at a specific time, given gave parameters.
    Data Analytics Course in Bangalore

  4. nice article!! try to create an article on digital marketing and post. that would also help people in doing the course.and one institute is good in offering certified courses that is 360DigiTMG.if u want u can refer that institute.

  5. Gone through this wonderful coures called Salesforce Certification Training in Dallas who are offering fully practical course, who parent is Salesforce Training in USA and they have students at Salesforce Training classes in Canada institutes.

  6. This comment has been removed by the author.

  7. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. gostream

  8. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

  9. This is not a position of power or a title, but a role for somebody who has the right mindset, attitude and ambition to change things, and who wants to make a difference and to set new standards. Salesforce training in Hyderabad

  10. I am happy to visit here. Thanks to my friend who told me about this webpage, this blog is really awesome
    Data Science Training in Hyderabad
    Data Science Course in Hyderabad

  11. This is a great inspiring article. I am pretty much pleased with your good work. You put really very helpful information.
    Python Training in Hyderabad
    Python Course in Hyderabad

  12. Really nice article. I appreciated your effort in this article. I bookmarked your website for further posts. Keep sharing!
    Data Science Course in Hyderabad

  13. Thanks for your sharing great article, I am very happy to read this article and I like it very much!I would also recommend it to my friends.
    sun news live
    sun news
    sun tv live
    news 7 live
    sun news live today
    sun tv news
    sun tv news live
    sun tv live news
    சன் நியூஸ்

  14. wow, great, I was wondering how to cure acne naturally. I found your site on Google, learned a lot, and now I'm a bit clearer. I’ve bookmarked your site and also added rss. keep us updated.
    data science course in hyderabad