I love Traveling...

Create your own visited countries map or check out the JavaScript Charts.

Sri Lanka (Home Country)
Thailand
Idaho (US)
Seattle (US)
Italy
India
Vatican City
Portugal
Spain
France
Belgium

Changing font style in Tumblr posts

Tumblr is a simple and nice microblogging site. I wanted to use it for very short posts which I think are important. But when I started using Tumblr for this I felt that the font size of the posts are bit bigger. ( This may not be the case if you create posts with a considerable content. But it my case I mostly wanted to post few lines of posts, so the title of the posts looked bit bigger than expected)

I could not find a simple setting to change the font size of the post titles, but I could find a simple way  to do that by editing the html code.

1. Go to the home page of your Tumblr account and click on 'Customize' menu

2. In the left side panel, select 'Edit HTML'

3. Search for 'block:Posts' section in the html code. (use ctrl + f to get the search box )

Then you can see the 'block:Title' section in that. You can add the font style that you need for the title of the posts in that section.

4. You can use 'Update Preview' button and see how the new font style looks like.

Make the changes you need and hit 'Save' and you have what you need.

Issue in put files to hadoop - 0 datanodes running

I setup hadoop as I have described in my previous posts. While I was working in some interesting projects on Map-Reduce, I faced an issue on sending some of my local files to the hdfs node. I have started dfs already, but exception says

14/10/25 17:46:20 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/ashansa/input._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1471)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2791)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:606)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:455)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

I faced this issue several times, so I think it will be helpful if I tell you how to overcome this issue. ( I believe this is a common issue we get while running hdfs, since I got the same issue several times and I found that many others also have faced the same issue )

What worked for me?

1. Stop the hadoop cluster with
             bin/stop-all.sh

2. Clean hadoop tmp directory
    You can find the path to your hadoop tmp directory in hdfs-site.xml.
    Check for the following tag in hdfs-site.xml file.
             <name>hadoop.tmp.dir</name>
             <value>root/data/hdfstmp</value>
    You can find the path to your hadoop temp directory in <value> tag.

3. Format node
             bin/hadoop namenode -format

4. Start hadoop cluster
             bin/start-all.sh

With these few simple steps, I was able to overcome this issue.
Hope this will help you too.

Specify download directory at the time of downloading - Safari

When I download something I used to save it to a relevant directory in my machine. When I moved to Mac and to Safari, one of the most inconvenient things I experienced was that Safari download everything to the download directory and I cannot choose the place at the time of downloading (Even though I can change the path from downloads directory to some other directory, it downloads everything to that and it was not the thing I was looking for)

After much trouble going through different sources finally found something reasonable. Since I saw that there are many more people out there with the same problem thought that it would be useful to share this simple tip (though the tip is simple, it was very useful to me + had to spend a considerable amount of time to find this out)

So simply if you need to specify the download directory at the time of downloading, without just clicking the link to download

Right click on link and choose ‘Download Linked File As…’

This will do the trick.
Hope this would make your life easier with Safari ☺

Run a jar file in command line

This is a very simple and short post on running a jar file in command line.
Simplest command that you can try is
java –jar [jarFileName].jar

But you may need to have the MANIFEST file in the jar to run it simply with the above command.

So how to bundle MANIFEST.MF to your jar?

You can add the following maven plugin to your pom.xml and get it done. Remember to add the main class name of your app here.

How to run the jars with external dependencies?

If you need to use the simple command java –jar [jarFileName].jar to run your application which has external dependencies, you can bundle the dependencies you need to the executable jar itself. Again use the below maven plugin to get it done.

The highlighted part is doing this for you and the part above is to bundle MANIFEST.MF file to that jar file too.

Hadoop - pseudo distributed mode setup

You can simply make your standalone hadoop setup to a pseudo distributed mode with following changes

• In HADOOP_HOME/etc/hadoop/core-site.xml, add

    
        fs.defaultFS
        hdfs://localhost:9000
    

• In HADOOP_HOME/etc/hadoop/hdfs-site.xml, add

    
        dfs.replication
        1
    

• Make sure that you can connect to localhost with ssh.

Start and test your hadoop setup

• Fist navigate to HADOOP_HOME

• Format the hadoop file system

        /bin/hdfs namenode –format

• Start Name node and Data node

        /sbin/start-dfs.sh

• Now you should be able to browse the hadoop web interface through

        http://localhost:50070

        And your hadoop file system under

        Utilities > browse the file system

• Add /user/ to hadoop file system

        hdfs dfs –mkdir /user

        hdfs dfs –mkdir /user/

        You will be able to see these directories when you browse the file system now. And you can list         the files with

        hdfs dfs –ls ( ie: hdfs dfs –ls / )

• Copy the input file to the hadoop file system

        hdfs dfs –put

        ie: hdfs dfs –put myinput input

and the file will be copied to /user//input

• Run the application with

      hadoop jar [local path to jar file] [path to main class] [input path in dfs]  [output location in dfs]

        ie: hadoop jar myapp.jar test.org.AppRunner input output

Result file: part-r-00000 should be saved in the output directory of dfs ( /user/[username]/output

Setup Hadoop in Mac

It is really simple to setup hadoop in Mac. I tried the latest available version at the moment. ( hadoop-2.5.1 ) You can setup hadoop in standalone mode or pseudo-distributed mode in your local machine. By following the below steps you will be able to setup hadoop in your machine in standalone mode.

( you need to install java and ssh beforehand to run hadoop )

1. Download the version you need to install from here 
2. Extract the downloaded pack
   The extracted directory will be your HADOOP_HOME ( ie: /Users/username/hadoopDir )
3. Add HADOOP_HOME to .bash_profile

   Export HADOOP_HOME=/Users/userName/hadoop-2.5.1 

   export PATH=$PATH:$HADOOP_HOME/bin

   
   
4. Source .bash_profile to affect the new changes

   source ~/.bash_profile

   
   
   Now you should be able to echo HADOOP_HOME in terminal ( echo $HADOOP_HOME )
5. Make sure that you can ssh to localhost

   ssh localhost

Now your stand alone hadoop setup is ready to use.

I will share a sample code I found on map reduce to test your setup.

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper
            extends Mapper{

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context
        ) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer
            extends Reducer {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable values,
                           Context context
        ) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Run sample

• Create a jar from the sample
• Create a text file of which you need to count words
• Run

             hadoop jar [path_to_jar] [path_to_main_class] [path_to_input] [path_to_output]              ie: hadoop jar wordCount.jar WordCount inputFile output

• On  a successful execution, you will have the output directory created at the path you specify. And your result will be in output/part-r-00000
• When you run the program again you need to remove the ‘output’ directory or give some other path for the output to be written.

You will see that this is really simple. 

You can find steps on setting up hadoop in pseudo-distributed mode in this post