Sunday, January 25, 2015

Issue in put files to hadoop - 0 datanodes running

I setup hadoop as I have described in my previous posts. While I was working in some interesting projects on Map-Reduce, I faced an issue on sending some of my local files to the hdfs node. I have started dfs already, but exception says
14/10/25 17:46:20 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/ashansa/input._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1471) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2791) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:606) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:455) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
I faced this issue several times, so I think it will be helpful if I tell you how to overcome this issue. ( I believe this is a common issue we get while running hdfs, since I got the same issue several times and I found that many others also have faced the same issue )

What worked for me?


1. Stop the hadoop cluster with
             bin/stop-all.sh

2. Clean hadoop tmp directory
    You can find the path to your hadoop tmp directory in hdfs-site.xml.
    Check for the following tag in hdfs-site.xml file.
             <name>hadoop.tmp.dir</name>
             <value>root/data/hdfstmp</value>
    You can find the path to your hadoop temp directory in <value> tag.

3. Format node
             bin/hadoop namenode -format

4. Start hadoop cluster
             bin/start-all.sh

With these few simple steps, I was able to overcome this issue.
Hope this will help you too.

50 comments:

  1. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies. By the way you are running a great blog. Thanks for sharing this.

    ReplyDelete
  2. A table is the basic unit of data storage in an oracle database. The table of a database hold all of the user accesible data. Table data is stored in rows and columns. But what is all about the clusters and how to handle it using oracle database system? Expecting a right answer from you. By the way you are maintaining a great blog. Thanks for sharing this in here.
    Oracle Training in Chennai | Oracle Course in Chennai | Oracle Training Center in Chennai

    ReplyDelete
  3. There are lots of information about latest technology, like Hadoop cluster is a special type of computational cluster designed specifically for storing and analyzing huge amounts of unstructured data in a distributed computing environment. This information seems to be more unique and interesting. Thanks for sharing.
    Big Data Training in Chennai | Hadoop Course in Chennai | Big Data Training in Chennai

    ReplyDelete
  4. Your blog is really awesome and I get a lot of great information from this blog. Thanks for sharing such an informative post on blog. Keep posting. It’s really useful and interesting...
    Healthcare Business Analysis Online Training
    Tableau Online Training
    Qlikview Online Training
    ServiceNow Administrator Online Training
    Peoplesoft HRMS Functional Online Training

    ReplyDelete

  5. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence.
    HADOOP Training in Chennai

    ReplyDelete
  6. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    SAS Training in Chennai | SAS Course in Chennai

    ReplyDelete
  7. Great article. It helps to explain complicated things to people who are not competent and explicate why this is important.
    data room services

    ReplyDelete
  8. The strategy you posted was nice. The people who want to shift their career to the IT sector then it is the right option to go with the ethical hacking course.
    Ethical hacking course in Chennai | Ethical hacking training in chennai

    ReplyDelete
  9. we are offering best servicenow training in hyderabad with job assistance and high quality training facilities and well expert faculty .
    to Register you free demo please visit servicenow training in hyderabad

    ReplyDelete
  10. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

    ReplyDelete

  11. Thanks for sharing this information and keep updating us. This information is very useful to me.
    Ethical Hacking Course in Chennai | Hacking Course in Chennai | Ethical Hacking Course

    ReplyDelete
  12. hai admin.thanks for sharing the post.Any individual who finishes our
    Hadoop Training in Chennai will turn into an ace in Hadoop with hands-on exercises and activities.More course details........ Selenium Training in Chennai
    Dot Net Training in Chennai
    Android Training in Chennai

    ReplyDelete
  13. Thank you so much for sharing this worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.

    Hadoop Online Training
    Data Science Online Training

    ReplyDelete
  14. Well Said, you have provided the right info that will be beneficial to somebody at all time. Thanks for sharing your valuable Ideas to our vision.Big Data Hadoop Training in Bangalore | Data Science Training in Bangalore

    ReplyDelete
  15. Updating with the recent skills and applying it is the only tactic to live in our vocation. You have done really a great job by sharing this blog in here. Keep writing blog like this.



    Hadoop Training in BTM Layout

    Hadoop Training in Marathahalli

    ReplyDelete
  16. Informative post about hadoop, i am looking forward for realtime hadoop online training institute.

    ReplyDelete
  17. I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site.
    Best Hadoop Training Institute In chennai

    ReplyDelete
  18. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.

    Selenium Training in Chennai

    ReplyDelete
  19. Best institute for 3d Animation and Multimedia Course training Classes


    Best institute for 3d Animation and Multimedia

    Best institute for 3d Animation Course training Classes in Noida- webtrackker Is providing the 3d Animation and Multimedia training in noida with 100% placement supports. for more call - 8802820025.

    3D Animation Training in Noida

    Company Address:

    Webtrackker Technology

    C- 67, Sector- 63, Noida

    Phone: 01204330760, 8802820025

    Email: info@webtrackker.com

    Website: http://webtrackker.com/Best-institute-3dAnimation-Multimedia-Course-training-Classes-in-Noida.php

    ReplyDelete
  20. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information. 

    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune
    Data science training in kalyan nagar

    ReplyDelete
  21. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    Devops training in Chennai
    Devops training in Bangalore
    Devops Online training
    Devops training in Pune

    ReplyDelete

  22. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.

    AWS Online Training | Online AWS Certification Course - Gangboard
    AWS Training in Chennai | AWS Training Institute in Chennai Velachery, Tambaram, OMR
    AWS Training in Bangalore |Best AWS Training Institute in BTM ,Marathahalli

    ReplyDelete
  23. Read all the information that i've given in above article. It'll give u the whole idea about it.
    python online training
    python training in OMR
    python training course in chennai

    ReplyDelete
  24. This is very good content you share on this blog. it's very informative and provide me future related information.
    Online DevOps Certification Course - Gangboard
    Best Devops Training institute in Chennai

    ReplyDelete
  25. A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.

    rpa training in electronic-city | rpa training in btm | rpa training in marathahalli | rpa training in pune

    ReplyDelete
  26. Brilliant article shared with lots of knowledge for Hadoop Training in Bangalore

    ReplyDelete
  27. Sap fico training institute in Noida

    Sap fico training institute in Noida - Webtrackker Technology is IT Company which is providing the web designing, development, mobile application, and sap installation, digital marketing service in Noida, India and out of India. Webtrackker is also providing the sap fico training in Noida with working trainers.


    WEBTRACKKER TECHNOLOGY (P) LTD.
    C - 67, sector- 63, Noida, India.
    F -1 Sector 3 (Near Sector 16 metro station) Noida, India.

    +91 - 8802820025
    0120-433-0760
    0120-4204716
    EMAIL: info@webtrackker.com
    Website: www.webtrackker.com

    ReplyDelete
  28. Your info is really amazing with impressive content..Excellent blog with informative concept. Really I feel happy to see this useful blog, Thanks for sharing such a nice blog..
    If you are looking for any Big data Hadoop Related information please visit our website Big Data Training In Bangalore page!

    ReplyDelete
  29. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.
    cloud computing training in chennai | Cloud computing training class in chennai

    ReplyDelete
  30. Clinical sas training in chennai | SAS Training course chennai
    I have to voice my passion for your kindness giving support to those people that should have guidance on this important matter.

    ReplyDelete
  31. I am glad that I saw this post. It is informative blog for us and we need this type of blog thanks for share this blog, Keep posting such instructional blogs and I am looking forward for your future posts.
    Cyber Security Projects for Final Year

    JavaScript Training in Chennai

    Project Centers in Chennai

    JavaScript Training in Chennai

    ReplyDelete