Thursday, February 4, 2016

Reproducible Stream Processing Benchmark to compare Apache Spark and Apache Flink on Cloud

Stream processing is becoming a crucial requirement with the high volume of data generated and the need for real-time processing of those data. And the data processing platforms are trying to provide smart and efficient approaches for stream processing. 

Yahoo has recently published a stream processing benchmark and has published the resources here to run the experiment in a single node. Since benchmarking stream processing is an interesting and important task, we wanted to reproduce the experiment on a clustered setup. 

We created a reproducible experiment on Amazon EC2 to reproduce yahoo streaming benchmark on a cluster of Apache Spark and Apache Flink. You can find all the resources and in instruction here which will help you to reproduce our experiment. And more importantly, you can reproduce the experiment with different configurations conveniently by following the instruction in above-mentioned link.

Following are some of the application-level and system-level performance results that we obtained during the experiment.

Application-level performance:



System-level performance


Memory:






CPU:





The configurations and explanation of the results can be found in the stream processing evaluation section of full report.

We have completed a performance comparison of batch processing as well for Apache Spark and Apache Flink to reproduce DongWong’s performance comparison.

The full project report can be found here.

Acknowledgement


    Jim Dowling
    Kamal Hakinzaheh
    Shelan Perera





16 comments:

  1. Wonderful post !!! Genuinely loved this kind of post.
    - Lenny Face

    ReplyDelete
  2. Video streaming is likewise represented by different conventions that bring under thought specialized execution, quality issues, dependability, cost elements, and lawful and social issues.best iptv service 2019

    ReplyDelete
  3. Prescient information investigation utilizes this data to channel current just as verifiable information to recognize examples and figure occasions and conditions that may happen later on at a specific time, given gave parameters.
    Data Analytics Course in Bangalore

    ReplyDelete
  4. nice article!! try to create an article on digital marketing and post. that would also help people in doing the course.and one institute is good in offering certified courses that is 360DigiTMG.if u want u can refer that institute.

    ReplyDelete
  5. Gone through this wonderful coures called Salesforce Certification Training in Dallas who are offering fully practical course, who parent is Salesforce Training in USA and they have students at Salesforce Training classes in Canada institutes.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. gostream

    ReplyDelete
  8. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.

    data science interview questions

    ReplyDelete
  9. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  10. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  11. very well explained .I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Simple Linear Regression
    Correlation vs covariance
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  12. This is not a position of power or a title, but a role for somebody who has the right mindset, attitude and ambition to change things, and who wants to make a difference and to set new standards. Salesforce training in Hyderabad

    ReplyDelete