Sunday, January 25, 2015

Issue in put files to hadoop - 0 datanodes running

I setup hadoop as I have described in my previous posts. While I was working in some interesting projects on Map-Reduce, I faced an issue on sending some of my local files to the hdfs node. I have started dfs already, but exception says
14/10/25 17:46:20 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/ashansa/input._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1471) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2791) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:606) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:455) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
I faced this issue several times, so I think it will be helpful if I tell you how to overcome this issue. ( I believe this is a common issue we get while running hdfs, since I got the same issue several times and I found that many others also have faced the same issue )

What worked for me?


1. Stop the hadoop cluster with
             bin/stop-all.sh

2. Clean hadoop tmp directory
    You can find the path to your hadoop tmp directory in hdfs-site.xml.
    Check for the following tag in hdfs-site.xml file.
             <name>hadoop.tmp.dir</name>
             <value>root/data/hdfstmp</value>
    You can find the path to your hadoop temp directory in <value> tag.

3. Format node
             bin/hadoop namenode -format

4. Start hadoop cluster
             bin/start-all.sh

With these few simple steps, I was able to overcome this issue.
Hope this will help you too.

51 comments:

  1. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies. By the way you are running a great blog. Thanks for sharing this.

    ReplyDelete
  2. A table is the basic unit of data storage in an oracle database. The table of a database hold all of the user accesible data. Table data is stored in rows and columns. But what is all about the clusters and how to handle it using oracle database system? Expecting a right answer from you. By the way you are maintaining a great blog. Thanks for sharing this in here.
    Oracle Training in Chennai | Oracle Course in Chennai | Oracle Training Center in Chennai

    ReplyDelete
  3. There are lots of information about latest technology, like Hadoop cluster is a special type of computational cluster designed specifically for storing and analyzing huge amounts of unstructured data in a distributed computing environment. This information seems to be more unique and interesting. Thanks for sharing.
    Big Data Training in Chennai | Hadoop Course in Chennai | Big Data Training in Chennai

    ReplyDelete

  4. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence.
    HADOOP Training in Chennai

    ReplyDelete
  5. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    SAS Training in Chennai | SAS Course in Chennai

    ReplyDelete
  6. Great article. It helps to explain complicated things to people who are not competent and explicate why this is important.
    data room services

    ReplyDelete

  7. Thanks for sharing this information and keep updating us. This information is very useful to me.
    Ethical Hacking Course in Chennai | Hacking Course in Chennai | Ethical Hacking Course

    ReplyDelete
  8. hai admin.thanks for sharing the post.Any individual who finishes our
    Hadoop Training in Chennai will turn into an ace in Hadoop with hands-on exercises and activities.More course details........ Selenium Training in Chennai
    Dot Net Training in Chennai
    Android Training in Chennai

    ReplyDelete
  9. Thank you so much for sharing this worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.

    Hadoop Online Training
    Data Science Online Training

    ReplyDelete
  10. Updating with the recent skills and applying it is the only tactic to live in our vocation. You have done really a great job by sharing this blog in here. Keep writing blog like this.



    Hadoop Training in BTM Layout

    Hadoop Training in Marathahalli

    ReplyDelete
  11. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information. 

    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune
    Data science training in kalyan nagar

    ReplyDelete
  12. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    Devops training in Chennai
    Devops training in Bangalore
    Devops Online training
    Devops training in Pune

    ReplyDelete

  13. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.

    AWS Online Training | Online AWS Certification Course - Gangboard
    AWS Training in Chennai | AWS Training Institute in Chennai Velachery, Tambaram, OMR
    AWS Training in Bangalore |Best AWS Training Institute in BTM ,Marathahalli

    ReplyDelete
  14. Brilliant article shared with lots of knowledge for Hadoop Training in Bangalore

    ReplyDelete
  15. Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
    If you are looking for any Big data Hadoop Related information please visit our website Big Data Training In Bangalore page!

    ReplyDelete
  16. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.
    cloud computing training in chennai | Cloud computing training class in chennai

    ReplyDelete
  17. Clinical sas training in chennai | SAS Training course chennai
    I have to voice my passion for your kindness giving support to those people that should have guidance on this important matter.

    ReplyDelete
  18. Nice and useful information.Thanks for the effort and the results,impressed with your post.
    big data training in btm layout

    ReplyDelete
  19. Informative post about hadoop. the Post is really amazing. blog is informative and the contents are very clear.

    ReplyDelete
  20. I have Grasp a lots of information's here...Thanks for posting Such a Useful comments...Informative post about hadoop are really Good...Keep updating
    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  21. You have provided a nice article, Thank you very much for this one. And I hope this will be useful for many people. Salesforce Training India 

    ReplyDelete
  22. Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. getbestmowers

    ReplyDelete
  23. Keep up the great work, I read few blog posts on this site and I believe that your website is really interesting and has loads of good info.Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Data Science Training In Chennai

    Data Science Online Training In Chennai

    Data Science Training In Bangalore

    Data Science Training In Hyderabad

    Data Science Training In Coimbatore

    Data Science Training

    Data Science Online Training

    ReplyDelete
  24. It help to make good advancements in your business that helps you to feel the ultimate confidence knowing that your employees would give you the best performance. It is beneficial especially for new teams who want to do the job well. This gives them the inspiration to carry out the activities efficiently. Salesforce training in Hyderabad

    ReplyDelete
  25. Python is a dynamic and object-oriented programming language, widely used for web application development. 90% of people prefer Python over other technology because of its simplicity, reliability and easy interfacing. data science course in india

    ReplyDelete
  26. Communication is a two way process. If done properly, it gives excellent result. Thus opting for the best Integrated Marketing Communication Course on Talentedge is wise. To know more visit:

    ReplyDelete
  27. Visit Bharat Go Digital Academy to learn the digital marketing skills in India.

    ReplyDelete
  28. Online lottery, Thai lottery, Malaysian lottery, Vietnamese lottery is available for you to play all. End every need, good price, get a lot, less waste. There are more than 100 slot games, fun, สมัคร ufa, easy to play, beautiful images, beautiful graphics, fast bonus if you want to get money quickly Must play slots

    ReplyDelete
  29. สล็อตออนไลน์ (Slot Online) คือการนำเครื่องเล่นพนัน ตู้สล็อตแมชชีน ที่กล่าวมาข้างต้นนี้ นำมาทำเป็นเกมอิเล็กทรอนิกส์ที่เรียกกันว่า สล็อตออนไลน์ เนื่องด้วยยุคสมัยพัฒนาคนหันมานิยมเล่นพนันกันผ่านคอมพิวเตอร์ จะได้นำเกมสล็อตมาทำเป็นเกมพนันออนไลน์ ผ่านระบบเครื่อข่ายอินเตอร์เน็ต ซึ่งผู้เล่นสามารถเล่นแบบผ่าน โปรแกรมสล็อต ก็ได้หรือจะเป็นการเข้าเล่น Slots ผ่านหน้าเว็บไซต์ผู้ให้บริการ ซึ่งบริการเกม สล็อตออนไลน์ นั้นก็มีรูปแบบของกติกาการเล่น คล้ายกันกับการเล่นบนตู้สล็อตแมชชีน ทั้งภาพและเสียงสมจริงเร้าใจไม่แพ้กันกับไปนั่งเล่นในคาสิโนเลยทีเดียว.

    ReplyDelete
  30. ufabet Parent company, the most popular online gambling website, whether it is online football betting Online casinos Baccarat online All of them were not less popular than each other. Become a member of UEFA Bet Playing with the parent company Did not pass agent Bet on a variety of casino games Especially the gambler who likes to Online football betting Our website provides 4 football odds, football betting, a minimum of 10 baht , betting is easy

    ReplyDelete
  31. แทงบอล กับ i99CLUB เว็บแทงบอล ที่ตอบโจทย์ให้กับคุณได้อย่างดีเยี่ยมที่สุด เท่าที่จะเป็นไปได้แล้วก็บอกได้เลยว่าการทำเงินตรงจุดนี้ เป็นอีกหนึ่งทางเลือกที่ข้างๆ ที่จะการันตีได้อย่างชัดเจนแล้วว่า เป็นสิ่งที่ดีที่สุดและทรงประสิทธิภาพมากที่สุด เท่าที่จะเป็นไปได้และเชื่อได้เลยว่า มันจะเป็นอีกหนึ่งตัวเลือกที่พร้อมจะเปิดกว้างให้กับคุณได้ อย่างมั่นคงและทรงประสิทธิภาพมากที่สุด

    ReplyDelete
  32. I am always searching online for articles that can help me. There is obviously a lot to know about this. I think you made some good points in Features also. Keep working, great job
    data scientist course in hyderabad

    ReplyDelete
  33. Excellent and very cool idea and great content of different kinds of the valuable information's.

    Tableau Training in Pune
    Best Tableau Training Institute in Pune

    ReplyDelete
  34. It's a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that "The content of your post is awesome" Great work.
    data scientist course in hyderabad

    ReplyDelete
  35. Informative blog and knowledgeable content. If you want to become a data science training then follow the below link.
    Best Courses for Data Science in Hyderabad

    ReplyDelete
  36. This comment has been removed by the author.

    ReplyDelete
  37. This comment has been removed by the author.

    ReplyDelete
  38. It's so good that you have showered your experience and knowledge through this blog and it will help many beginners and also experience personnel's. Just check this out also if anyone searching related to devops.

    ReplyDelete