Monday, March 28, 2016

Vagrant : An amazing tool to manage VMs

Vagrant is an amazing tool for managing VMs. Why it is so amazing is because of the convenience it adds in the aspect of managing VMs. It can work with VirtualBox which makes it possible to get the advantage of vagrant for free.
If you have installed Vagrant and VirtualBox starting a vagrant and spawning up a virtual machine is super simple. You just need to enter 2 commands below.

vagrant init hashicorp/precise64
vagrant up

First command creates a configuration file in your current directory with the name "Vagrantfile" and this file can be used to configure the VMs as required. The second command will create and start the VM with the name "default". With these two commands, you will have your VM up and running in VirtualBox.

But most of the time we may need to start several VMs and manage them. The Vagrantfile can be used to support that. Given below is a sample Vagrantfile (without comments) that was used to spawn up 3 VMs: vm1, vm2 and vm3 with the specified IPs. The memory of each VM, number of cpus, etc can be configured in the Vagrantfile as required.

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise64"
  config.vm.provider "virtualbox" do |vb|
  vb.memory = "3072"
  vb.cpus = 2
end

config.hostmanager.enabled = true
config.hostmanager.manage_host = false
config.hostmanager.ignore_private_ip = false
config.hostmanager.include_offline = true

  config.vm.define "vm1" do |vm1|
    vm1.vm.hostname = 'vm1-hostname'
    vm1.vm.network "private_network", ip: "172.28.128.5"
  end

  config.vm.define "vm2" do |vm2|
    vm2.vm.hostname = 'vm2-hostname'
    vm2.vm.network "private_network", ip: "172.28.128.6"
  end

  config.vm.define "vm3" do |vm3|
    vm3.vm.hostname = 'vm3-hostname'
    vm3.vm.network "private_network", ip: "172.28.128.7"
  end
end


Below are some useful commands that are be required to manage these VMs.
  • vagrant up vm1 (create vm1 if not created and start the vm1)
  • vagrant ssh vm1 (ssh into vm1)
  • vagrant halt vm1 (gracefully shutdown vm1)
  • vagrant destroy vm1 (destroy vm1)
  • vagrant status (shows status of the VMs created in current directory)
  • vagrant global-status (shows status of all the VMs in your host)

And one another useful feature is the easiness of copying the files to your VM. You can simply copy whatever the files you need to the same directory where the Vagrantfile is and those files will be available in /vagrant directory in your VM.
Give it a try when you need to work with VMs.

Friday, February 5, 2016

Do you want to deploy a cluster in a single click with single configuration file?

Do you want to create a hadoop cluster to test your map-reduce job? Are you postponing it thinking about the overhead of setting up the environment? Karamel provides you the easiest way to do it. Karamel is a platform which supports convenient deployment orchestration of distributed system deployments. You can set up your cluster in cloud with few simple steps.  

Download the latest version of Karamel here. You just need a configuration file describing how you need to deploy your cluster. The link provides all required information on setting up Karamel (you just need to extract the .tar file). Following video will help you to have a smooth floor in dealing with Karamel in doing your deployment and running experiments.









Thursday, February 4, 2016

Reproducible Stream Processing Benchmark to compare Apache Spark and Apache Flink on Cloud

Stream processing is becoming a crucial requirement with the high volume of data generated and the need for real-time processing of those data. And the data processing platforms are trying to provide smart and efficient approaches for stream processing. 

Yahoo has recently published a stream processing benchmark and has published the resources here to run the experiment in a single node. Since benchmarking stream processing is an interesting and important task, we wanted to reproduce the experiment on a clustered setup. 

We created a reproducible experiment on Amazon EC2 to reproduce yahoo streaming benchmark on a cluster of Apache Spark and Apache Flink. You can find all the resources and in instruction here which will help you to reproduce our experiment. And more importantly, you can reproduce the experiment with different configurations conveniently by following the instruction in above-mentioned link.

Following are some of the application-level and system-level performance results that we obtained during the experiment.

Application-level performance:



System-level performance


Memory:






CPU:





The configurations and explanation of the results can be found in the stream processing evaluation section of full report.

We have completed a performance comparison of batch processing as well for Apache Spark and Apache Flink to reproduce DongWong’s performance comparison.

The full project report can be found here.

Acknowledgement


    Jim Dowling
    Kamal Hakinzaheh
    Shelan Perera





Tuesday, December 1, 2015

Computer users, relax your tired eyes using 20-20-20 rule

We all know that using a computer for a longer time each day, makes your eyes tired. This is a simple rule which may help reduce your eyestrain and let your eyes relax for a while. It is recommended to stand up and stretch once in a while without sitting on your chair all the time. But sometimes it may not be possible to walk in every 30 minutes from your chair. But at least try following 20-20-20 rule to relax your eyes :)

20-20-20 rule:




And try to blink your eyes more so that it will help your eyes to avoid getting dry. 

Wednesday, November 4, 2015

Presentation tips and 10-20-30 Rule for Presentations

When it is required to do a presentation either it is business related or academic, we always want to make it attractive and smart. Following are some of the well known methods that I have been using.

  • Use less text
We know that the audience will get board when you have a lot of text in your slides. If they do not get board, they will start reading the slides without listening to your talk. So you miss the change to emphasis what you want. Either way you loose :(
So if you feel that should have text for a particular slide to explain it easier, try point format and be precise. 3, 4 bullet points with up to 6, 7 words per each bullet will be ideal.

  • Communicate your idea with pictures
We know that "A picture is worth a thousand words". It is always easy and clear to convey your idea through a picture than having a lot of text. (This is a very well known fact)

  • Use simple animations
If some complex concept or an algorithm should be presented, make some animated slides which can explain it easily than trying to struggle with all Xs and Ys. You may feel that it will be clear, but believe me it will not. Just try to memorize the very first time you got to know about that concept. Even though you have mastered it now, for the audience it may be the first time :)

And of course you should not have these complex explanations in your presentation if you do not think that you MUST have them.

  • Less slides and good formatting
Have some attractive formatting, so your audience will not get board seeing it. But of course I did not mean 'fancy' :)

  • 10-20-30 Rule
This was a concept introduced by Guy Kawasaki who is a Silicon Valley marketing executive.
In brief it says
               10 slides only
               20 minutes presentation time
               30 font size
Here is his explanation on how it will be effective. 


And I have some final tip for the audience. If your colleague is doing a presentation for academic purposes and you are supposed to give feedback, please provide the real feedback you have. Do not say that it was amazing just because you think that your friend will feel bad if you didn't do so. I specifically mentioned academic because I believe when it comes to business, it is most probable to have the real feedback since it is business. :)

I personally had this practice of being nice and trying to avoid giving bad feedback and suddenly I realized that I am taking away the change of my colleague to further improve his/her presentation skills. This can be something simple like 'You spoke too fast' but if he improve that, it will help a lot to attract more audience next time. You can of course be polite and convey it nicely, so that it will really help the presenter to improve himself/herself. And may be you can give written feedback too.

So good luck with your next presentation :)

Wednesday, October 7, 2015

Singleton Pattern

This is one of the simplest, yet very useful design pattern. Singleton design pattern can be used when it is required to create just one object of a class. As an example, if you have a ApplicationManager class in your application where all other components need to get help from, you need to have access to ApplicationManager object. But just think that you need to share some objects through your ApplicationManager. Then of course you do not want to have different object referring to different ApplicationManager objects. You need to have one common object shared with all of them. And singleton pattern is the way to achieve this.

I believe an example code snippet will help a lot in understanding this simply.


public class ApplicationManager {

    private static ApplicationManager applicationManager = new ApplicationManager();

    private ApplicationManager() { }

    public static ApplicationManager getInstance() {
        return applicationManager;
    }
    
    public void yourMethod() {
        //code
    }
}


So you see the constructor is private, so no one can initialize an object of ApplicationManager and if they want to use it, can call getInstance() method and use the already created static ApplicationManager object. So no more than one object is created and global access is provided.

Monday, October 5, 2015

Producer Consumer Pattern

Producer Consumer Design Pattern is one of the mainly used design patterns. And I am pretty sure that, many who does not know that this is an example of a design pattern too have used it. This design pattern is mainly used to introduce decoupling between consuming something and the relevant producing procedure. In other words it implements an efficient and smooth data sharing between the producer of the data and the data consumer. This decoupling is mainly done maintaining a queue for the data items.




By using a queue even the producer and consumer have different rates of producing and consuming data, the data sharing floor will be smooth through the queue. Producer do not have to wait until the consumer finishes data item 1 to deliver data item 2. And producer do not have any overhead of managing the produced items even though there are many consumers as far as all the consumers are consuming the items in that same queue.

So this design pattern can be easily used to perform data sharing between the producer parties and consumer parties in order to achieve good synchronization among them.



Importance of Design Patterns

Design Patterns is one of the most interesting topics in software development. They give well structured and standard solutions for solving common software implementation problems. To understand the design patterns, you should understand the problem that they solve. If you understand them, when you get some problem related to any of the design patterns you know, you can easily map the solution to your problem.

But if ‘Design Patterns’ is a new term for you even though you are doing good software implementation may be still you don’t have to worry. Because may be you are already using design patterns without knowing the correct terminology for those patterns. Still knowing them will help you a lot in improving your implementation as well as it will make it easy for others to quickly understand the functionalities of your implementation. And it will help a lot in smoothly communicating your solutions to the others. As an example if you say that you are using Factory method to solve problem X, others will just understand it without you explaining how you are doing it.

So Design patterns make your code handy, make it more readable and self explanative as well as will be helpful in smooth communication about your implementations.

Here are some commonly used design patterns

  • Producer Consumer    (Oh... You were using this right? That is why I added it first )
  • Singleton
  • Factory/Builder
  • Observer
  • Decorator
  • Command
  • Proxy
  • Strategy

Saturday, April 4, 2015

Avoid tracking your own page views

Are you interested in finding out the stats of your website? I am sure you do not need to count your own page views (if you like to see the true value). If you are using blogger, it gives a simple option for this. But unfortunately it did not work for me. It says " There was an error while fetching stats. Please reload page."
( you can try logging to your blogger account > Stats and click on "Don't track your own pageviews")



After some effort could find another solution that could fix my issue and here it is...

While been logged in to your account, open a new tab in the browser and enter

http://your_site.com/b/statsBlockingCookie?action=SET&callback=__gwt_jsonp__.P2.onSuccess

(make sure you replace the your_site.com with your site name)

You will get the following response
// API callback
__gwt_jsonp__.P2.onSuccess({"status":"yes"});

Restart your browser and you can see that your page views are not counted in stats :)

Wednesday, April 1, 2015

Want to map your keyboard with the keyboard for your VM?

Recently I had to work in VMs for few of my projects and the biggest burden is the changes in keyboard. I am using a Mac and wanted to use Windows on a VM for a project. It is really irritating when you have to work in both environments interchangeably. You will definitely press cmd + c to copy in your windows VM and wonder why cmd + v won't work :O

I could find a simple solution for this with the help of autohotkey.
So this is how I map the Mac keyboard to Windows VM. ( and I am using VirtualBox)


Map the hostkey of the VirtualBox to right command key.


Start VirtualBox
Select VirtualBox > Preferences from the top menu



Input > Virtual Machine
Search for 'host key combination' and set the right command key for the key combination



Setup the VM to have changes


Download autohotkey to VM
Create a file in VM with extension .ahk ( ie: script.ahk )
Following is a sample content for the file with some of the most useful key mappings

#SingleInstance force
#r::Send ^r ;reload
#z::Send ^z ; undo
#y::Send ^y ; redo
#f::Send ^f ; find inside apps
#c::Send ^c ; copy
#x::Send ^x ; cut
#v::Send ^v ; paste
#t::Send ^t ; new tab, Firefox
#s::Send ^s ; save inside apps
LWin & Tab::AltTab ; the motherlode, alt-tab!
#b::Send ^b ; bold / bookmarks in Firefox
#i::Send ^i ; italic
#u::Send ^u ; underline
#a::Send ^a ; select all

Save the content and save the file.
Right click on file and do 'run as administrator' and you are ready to go... :)


Add the script to the Startup Programs


Add your file to Startup folder and you will have no issue with VM restarts...
Start > Program Files > Startup

Hope you will find your life with VMs much more easier with this.


Saturday, March 21, 2015

Need to run many instances of your project in visual studio?


Solution is simple.

Right click on your solution
Set Startup Projects
Select ‘Multiple startup projects
And select ‘Start’ option for your projects in the list.

And now when you hit start button, all the project you selected will be started.

Well, you need several instances of the same project to be started?
Right click on the project
Debug > Start New Instance

(This option will be useful to interestingly try out the Chat Application we developed previously. Because with more chat clients, it will be more interesting )

Friday, March 20, 2015

Building .NET Remoting sample (Step by Step)


I am new to .Net and for one of my projects I had to use .NET Remoting. It was bit hard to find an end to end sample for a beginner even though there are code snippets describing each functionality. So I felt that it will be useful to share this with you.

I am going to give step by step details so that someone who is not familiar with visual studio can also follow and build the sample without any issue. (I am going to use visual studio which is most widely used IDE for .NET)

.NET Remoting is a mechanism that is used to access the remote services across network. (If you are familiar with WebServices the requirement which Remoting fulfills is similar to a WebService. But Remoting works with remote objects)

We are going to implement a simple chat service using .NET Remoting. We will need to implement 3 modules for this. 
  • Shared Library
  • Server side implementation
  • Client side implementation 


Implementing Shared Library

 
We will first create the shared library because both the client module and the server module is going to need it.
The shared library is used to define the remote objects that one party is going to expose to the other party. In our chat example we will need a remote object that server expose to client which allows sending messages. And another object that client expose to server which allows receiving messages. (All this will be very clear when you start building the project, so don’t worry  )
 
So let’s start creating the project.
 
File > New > Project
Select ‘Class Library’ and give the name as ‘ChatLib’ in the field below.





We need to have 2 Interfaces for the two remote objects exposed by the server and client. So let’s create two classes ClientLib and ServerLib in our Chat lib project.

In solution explorer, right click on ChatLib
Add > New Item > Interface
And give the name for the interface in the textbox below. (In this sample I am going to use IService for the interface name)






Now add the methods that we are going to expose
Register
SendMessage


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ChatLib
{
    public interface IService
    {
        void register(string userName, string url);
        void sendMessage(string userName, string msg);
    }
}


Create interface IClientService as same as above and add the method
receiveMessage


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace ChatLib
{
    public interface IClientService
    {
        void receiveMessage(string sender, string msg);
    }
}


Now you are done with the ChatLib component

Server Side Implementation

Now lets add the server side code. For this do right click on solution




Add > New Project > Console App
And give the name as ServerApp




The new project will be visible along with ChatLib under solution view.

Since we are going to have the implementation of the IService in this module we need to add ChatLib as a reference to our new project. To do that right click on ServerApp

Add Reference > Solution > Projects
Tick ChatLib project which is under Solutions > Projects

We need one more reference
Assemblies > Framework
Tick System.Runtime.Remoting




Here is the code for ServerApp (to make it easy I am going to add the whole code to one file here)


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using ChatLib;
using System.Runtime.Remoting;
using System.Runtime.Remoting.Channels;
using System.Runtime.Remoting.Channels.Tcp;

namespace ServerApp
{
    class Program
    {
        static void Main(string[] args)
        {
            TcpChannel channel = new TcpChannel(9443);
            ChannelServices.RegisterChannel(channel, true);
            RemotingConfiguration.RegisterWellKnownServiceType(typeof(Server),
                "ChatServerService", WellKnownObjectMode.Singleton);

            System.Console.WriteLine("Server...... press any key");
            System.Console.ReadLine();
        }
    }

    public class Server  : System.MarshalByRefObject, IService
    {
        public Dictionary users = new Dictionary();
        public TcpChannel channel;

        public Server()
        {
            channel = new TcpChannel();
        }
        

        public void register(string userName, string port)
        {
            users.Add(userName, port);
            Console.WriteLine("user added: " + userName);     
        }

        public void sendMessage(string sender, string msg)
        {
            Console.WriteLine("msg : " + msg + " received from " + sender);

            foreach(var item in users)
            {
                //send msg to all clients 
                IClientService clientService = (IClientService)Activator.GetObject(typeof(IClientService),
                            "tcp://localhost:" + item.Value + "/ChatClientService");
                if (item.Key != sender)
                    clientService.receiveMessage(sender, msg);
            }
        }
    }
}


Now as the same way you created the ServerApp, create another console application for the ClientApp too and add the same references (ChatLib and System.Runtime.Remoting )


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using ChatLib;
using System;
using System.Runtime.Remoting;
using System.Runtime.Remoting.Channels;
using System.Runtime.Remoting.Channels.Tcp;

namespace ClientApp
{
    class Program 
    {
        static string userName;

        static void Main(string[] args)
        {
            Console.WriteLine("Enter username:");
            userName = Console.ReadLine();
           
            Console.WriteLine("Enter port:");
            string port = Console.ReadLine();

            Program program = new Program();
            program.register(port);

            Console.WriteLine("Now you can send messages");
            program.sendMessage();
            Console.ReadLine();
        }

        public void register(string port)
        {
            TcpChannel clientChannel = new TcpChannel(Convert.ToInt32(port));
            ChannelServices.RegisterChannel(clientChannel, true);

            ClientService clientLib = new ClientService();
            RemotingServices.Marshal(clientLib, "ChatClientService", typeof(ClientService));

            IService service = (IService)Activator.GetObject(typeof(IService),
               "tcp://localhost:9443/ChatServerService");

            service.register(userName, port);
            
        }

        public void sendMessage()
        {
            Console.WriteLine("mmmmm");
            IService service = (IService)Activator.GetObject(typeof(IService),
               "tcp://localhost:9443/ChatServerService");
            
            while (true)
            {
                Console.WriteLine("me:");
                string msg = Console.ReadLine();
                service.sendMessage(userName, msg);
            }
        }
        
    }

    public class ClientService : MarshalByRefObject, IClientService
    {
        public void receiveMessage(string sender, string msg)
        {
            Console.WriteLine(sender + " : " + msg);
        }
    }
}

OK... You are done with your chat application with .NET Remoting. Now you can run the application and see.





Sunday, March 8, 2015

I love Traveling...






Sri Lanka (Home Country)
Thailand
Idaho (US)
Seattle (US)
Italy
India
Vatican City
Portugal
Spain
France
Belgium

Thursday, February 19, 2015

Changing font style in Tumblr posts

Tumblr is a simple and nice microblogging site. I wanted to use it for very short posts which I think are important. But when I started using Tumblr for this I felt that the font size of the posts are bit bigger. ( This may not be the case if you create posts with a considerable content. But it my case I mostly wanted to post few lines of posts, so the title of the posts looked bit bigger than expected)

I could not find a simple setting to change the font size of the post titles, but I could find a simple way  to do that by editing the html code.

1. Go to the home page of your Tumblr account and click on 'Customize' menu



2. In the left side panel, select 'Edit HTML'


3. Search for 'block:Posts' section in the html code. (use ctrl + f to get the search box )
Then you can see the 'block:Title' section in that. You can add the font style that you need for the title of the posts in that section.


4. You can use 'Update Preview' button and see how the new font style looks like.
Make the changes you need and hit 'Save' and you have what you need.

Sunday, January 25, 2015

Issue in put files to hadoop - 0 datanodes running

I setup hadoop as I have described in my previous posts. While I was working in some interesting projects on Map-Reduce, I faced an issue on sending some of my local files to the hdfs node. I have started dfs already, but exception says
14/10/25 17:46:20 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/ashansa/input._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1471) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2791) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:606) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:455) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
I faced this issue several times, so I think it will be helpful if I tell you how to overcome this issue. ( I believe this is a common issue we get while running hdfs, since I got the same issue several times and I found that many others also have faced the same issue )

What worked for me?


1. Stop the hadoop cluster with
             bin/stop-all.sh

2. Clean hadoop tmp directory
    You can find the path to your hadoop tmp directory in hdfs-site.xml.
    Check for the following tag in hdfs-site.xml file.
             <name>hadoop.tmp.dir</name>
             <value>root/data/hdfstmp</value>
    You can find the path to your hadoop temp directory in <value> tag.

3. Format node
             bin/hadoop namenode -format

4. Start hadoop cluster
             bin/start-all.sh

With these few simple steps, I was able to overcome this issue.
Hope this will help you too.

Friday, November 7, 2014

Specify download directory at the time of downloading - Safari

When I download something I used to save it to a relevant directory in my machine. When I moved to Mac and to Safari, one of the most inconvenient things I experienced was that Safari download everything to the download directory and I cannot choose the place at the time of downloading (Even though I can change the path from downloads directory to some other directory, it downloads everything to that and it was not the thing I was looking for)

After much trouble going through different sources finally found something reasonable. Since I saw that there are many more people out there with the same problem thought that it would be useful to share this simple tip (though the tip is simple, it was very useful to me + had to spend a considerable amount of time to find this out)

So simply if you need to specify the download directory at the time of downloading, without just clicking the link to download

Right click on link and choose ‘Download Linked File As…’

This will do the trick.
Hope this would make your life easier with Safari ☺

Saturday, November 1, 2014

Run a jar file in command line

This is a very simple and short post on running a jar file in command line.
Simplest command that you can try is
java –jar [jarFileName].jar

But you may need to have the MANIFEST file in the jar to run it simply with the above command.

So how to bundle MANIFEST.MF to your jar?

You can add the following maven plugin to your pom.xml and get it done. Remember to add the main class name of your app here.




How to run the jars with external dependencies?

If you need to use the simple command java –jar [jarFileName].jar to run your application which has external dependencies, you can bundle the dependencies you need to the executable jar itself. Again use the below maven plugin to get it done.




The highlighted part is doing this for you and the part above is to bundle MANIFEST.MF file to that jar file too.

Thursday, October 30, 2014

Hadoop - pseudo distributed mode setup


You can simply make your standalone hadoop setup to a pseudo distributed mode with following changes

  • In HADOOP_HOME/etc/hadoop/core-site.xml, add

    
        fs.defaultFS
        hdfs://localhost:9000
    

  • In HADOOP_HOME/etc/hadoop/hdfs-site.xml, add
    
        
            dfs.replication
            1
        
    
  • Make sure that you can connect to localhost with ssh.

Start and test your hadoop setup


  • Fist navigate to HADOOP_HOME
  • Format the hadoop file system
        /bin/hdfs namenode –format
  • Start Name node and Data node
        /sbin/start-dfs.sh

  • Now you should be able to browse the hadoop web interface through
        And your hadoop file system under
        Utilities > browse the file system

  • Add /user/ to hadoop file system
        hdfs dfs –mkdir /user
        hdfs dfs –mkdir /user/
        You will be able to see these directories when you browse the file system now. And you can list         the files with
        hdfs dfs –ls ( ie: hdfs dfs –ls / )

  • Copy the input file to the hadoop file system
        hdfs dfs –put
        ie: hdfs dfs –put myinput input
and the file will be copied to /user//input

  • Run the application with
      hadoop jar [local path to jar file] [path to main class] [input path in dfs]  [output location in dfs]
        ie: hadoop jar myapp.jar test.org.AppRunner input output

Result file: part-r-00000 should be saved in the output directory of dfs ( /user/[username]/output

Tuesday, October 28, 2014

Setup Hadoop in Mac

It is really simple to setup hadoop in Mac. I tried the latest available version at the moment. ( hadoop-2.5.1 ) You can setup hadoop in standalone mode or pseudo-distributed mode in your local machine. By following the below steps you will be able to setup hadoop in your machine in standalone mode.

( you need to install java and ssh beforehand to run hadoop )

  1. Download the version you need to install from here 
  2. Extract the downloaded pack
    The extracted directory will be your HADOOP_HOME ( ie: /Users/username/hadoopDir )
  3. Add HADOOP_HOME to .bash_profile
    Export HADOOP_HOME=/Users/userName/hadoop-2.5.1 
    export PATH=$PATH:$HADOOP_HOME/bin

  4. Source .bash_profile to affect the new changes
    source ~/.bash_profile

    Now you should be able to echo HADOOP_HOME in terminal ( echo $HADOOP_HOME )
  5. Make sure that you can ssh to localhost
    ssh localhost

Now your stand alone hadoop setup is ready to use.
I will share a sample code I found on map reduce to test your setup.

import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount { public static class TokenizerMapper extends Mapper{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }

Run sample

  • Create a jar from the sample
  • Create a text file of which you need to count words
  • Run
             hadoop jar [path_to_jar] [path_to_main_class] [path_to_input] [path_to_output]              ie: hadoop jar wordCount.jar WordCount inputFile output
  • On  a successful execution, you will have the output directory created at the path you specify. And your result will be in output/part-r-00000
  • When you run the program again you need to remove the ‘output’ directory or give some other path for the output to be written.
You will see that this is really simple. 
You can find steps on setting up hadoop in pseudo-distributed mode in this post