Thursday, February 4, 2016

Reproducible Stream Processing Benchmark to compare Apache Spark and Apache Flink on Cloud

Stream processing is becoming a crucial requirement with the high volume of data generated and the need for real-time processing of those data. And the data processing platforms are trying to provide smart and efficient approaches for stream processing. 

Yahoo has recently published a stream processing benchmark and has published the resources here to run the experiment in a single node. Since benchmarking stream processing is an interesting and important task, we wanted to reproduce the experiment on a clustered setup. 

We created a reproducible experiment on Amazon EC2 to reproduce yahoo streaming benchmark on a cluster of Apache Spark and Apache Flink. You can find all the resources and in instruction here which will help you to reproduce our experiment. And more importantly, you can reproduce the experiment with different configurations conveniently by following the instruction in above-mentioned link.

Following are some of the application-level and system-level performance results that we obtained during the experiment.

Application-level performance:



System-level performance


Memory:






CPU:





The configurations and explanation of the results can be found in the stream processing evaluation section of full report.

We have completed a performance comparison of batch processing as well for Apache Spark and Apache Flink to reproduce DongWong’s performance comparison.

The full project report can be found here.

Acknowledgement


    Jim Dowling
    Kamal Hakinzaheh
    Shelan Perera





11 comments:

  1. Wonderful post !!! Genuinely loved this kind of post.
    - Lenny Face

    ReplyDelete
  2. Video streaming is likewise represented by different conventions that bring under thought specialized execution, quality issues, dependability, cost elements, and lawful and social issues.best iptv service 2019

    ReplyDelete
  3. Prescient information investigation utilizes this data to channel current just as verifiable information to recognize examples and figure occasions and conditions that may happen later on at a specific time, given gave parameters.
    Data Analytics Course in Bangalore

    ReplyDelete
  4. nice article!! try to create an article on digital marketing and post. that would also help people in doing the course.and one institute is good in offering certified courses that is 360DigiTMG.if u want u can refer that institute.

    ReplyDelete
  5. Gone through this wonderful coures called Salesforce Certification Training in Dallas who are offering fully practical course, who parent is Salesforce Training in USA and they have students at Salesforce Training classes in Canada institutes.

    ReplyDelete
  6. I have been searching for a useful post like this on salesforce course details, it is highly helpful for me and I have a great experience with this Salesforce Training who are providing certification and job assistance.
    Salesforce training in Hyderabad  

    ReplyDelete
  7. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. gostream

    ReplyDelete