Setup Hadoop in Mac

It is really simple to setup hadoop in Mac. I tried the latest available version at the moment. ( hadoop-2.5.1 ) You can setup hadoop in standalone mode or pseudo-distributed mode in your local machine. By following the below steps you will be able to setup hadoop in your machine in standalone mode.

( you need to install java and ssh beforehand to run hadoop )

1. Download the version you need to install from here 
2. Extract the downloaded pack
   The extracted directory will be your HADOOP_HOME ( ie: /Users/username/hadoopDir )
3. Add HADOOP_HOME to .bash_profile

   Export HADOOP_HOME=/Users/userName/hadoop-2.5.1 

   export PATH=$PATH:$HADOOP_HOME/bin

   
   
4. Source .bash_profile to affect the new changes

   source ~/.bash_profile

   
   
   Now you should be able to echo HADOOP_HOME in terminal ( echo $HADOOP_HOME )
5. Make sure that you can ssh to localhost

   ssh localhost

Now your stand alone hadoop setup is ready to use.

I will share a sample code I found on map reduce to test your setup.

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper
            extends Mapper{

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context
        ) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer
            extends Reducer {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable values,
                           Context context
        ) throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

Run sample

• Create a jar from the sample
• Create a text file of which you need to count words
• Run

             hadoop jar [path_to_jar] [path_to_main_class] [path_to_input] [path_to_output]              ie: hadoop jar wordCount.jar WordCount inputFile output

• On  a successful execution, you will have the output directory created at the path you specify. And your result will be in output/part-r-00000
• When you run the program again you need to remove the ‘output’ directory or give some other path for the output to be written.

You will see that this is really simple. 

You can find steps on setting up hadoop in pseudo-distributed mode in this post

Simple overview on the main roles in App Factory how they involve in application process.

With the multi tenanted App Factory there are some changes in user model in App Factory. I am going to give you and idea on the default roles and the main actions that those roles are responsible of doing in the application space in App Factory.

Admin

• Creates a space for the organization in App Factory.
• Can add organization level users and assign them roles
  Default roles would be Developer, DevOps, QA, Application Owner, CXO

Application Owner

• Only the application owners can create applications.
• After creating an application he can assign people ( that has been already added to the organization by the organization admin ) to his application. And those people become the members of the application and can play relevant roles ( developer, QA... ) assigned for them ( by admin ) for the created applications.

Developer

• He will see all the applications of which he is a member of.
• He can do git clone, push, trigger build, etc ( the work related for developing the application )

QA

• Will see the applications that he has is a member of.
• Can perform the testing tasks. ( testing the deployed artifacts, report bugs... )

CXO

• Can view dashboards.
  
  
  
  
  

Configure SAML2 Single Sign-On on WSO2 servers with WSO2 Identity Server.

By following this post you will be able to find out how to configure WSO2 servers to have SAML2 SSO with WSO2 Identity Server (IS) as the identity provider. It is really simple to configure SAML2 SSO for carbon servers.
I am going to address the server that you need to have SSO configured as 'Carbon Server' and just by following the below 2 steps you can configure SSO in your carbon server with WSO2 IS.

1. Configure your carbon server to enable SSO

All the required configuration to have SSO in your carbon server are in Carbon server/repository/conf/security/authenticators.xml

• Enable SSOAuthenticator in authenticators.xml

( 1 ) Set disabled="false"

( 2 ) This should be unique to your carbon server. You will need this value when configuring IS too.

• Start your carbon server with an offset ( offset can be configured in carbon.xml)

2. Register a service provider in IS side

• Start IS in default port ( 9443 ) and log in 
• Follow Main > Manage > SAML SSO > Register New Service Provider
• Add the unique identifier ( 2 ) as the Issuer
• Provide Assertion Consumer URL with your carbon server info as https://[host name]:[port]/acs
• Tick on Enable Response Signing and Enable Assertion Signing
• Click on "Register"

Now you are done. You can simply try to log into your carbon server with SSO.
To verify
    - Try to access https://[host name]:[port]/carbon
    - This will direct you to the authentication endpoint of IdentityProviderSSOServiceURL specified in authenticators.xml
      ( here https://localhost:9443/authenticationendpoint )
    - Give the credentials and hit Sign in
    - You will be logged in to your carbon server

Write a simple JDBC PIP attribute finder module for WSO2 Identity Server

With this post I am going to discuss on how you can implement a simple JDBC PIP attribute finder module for WSO2 IS. I am using the latest released IS version ( WSO2 IS version 4.5.0 ) which you can download from here.

To have your own customized PIP module, the main task is to implement an attribute finder. It is not that hard since we already have the modeling interfaces. You can simply extend the AbstractPIPAttributeFinder ( abstract class) or implement PIPAttributeFinder ( interface ) to create your attribute finder.

I will provide step by step guide on how to

• Create your own attribute finder
• Register your PIP module in WSO2 IS
• Test your attribute finder

Create your own attribute finder

I am going to create a JDBC attribute finder where the attributes required are stored in a database. I am going to use mysql for this sample.

I will add a sample code for our attribute finder and I'm going to address it as JDBCAttributeFinder.

package org.wso2.identity.samples.entitlement.pip.jdbc;

import org.apache.commons.dbcp.BasicDataSource;
import org.wso2.carbon.identity.entitlement.pip.AbstractPIPAttributeFinder;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.HashSet;
import java.util.Properties;
import java.util.Set;

/**
 * This is sample implementation of PIPAttributeFinder in Wso2 Entitlement Engine Here we are
 * calling to a external user base to find given attribute Assume that user store is reside on mysql
 * database
 */
public class JDBCAttributeFinder extends AbstractPIPAttributeFinder {

    /**
     * DBCP connection pool is used to create connection to database
     */
    private BasicDataSource dataSource;

    /**
     * List of attribute finders supported by the this PIP attribute finder
     */
    private Set supportedAttributes = new HashSet();

    /**
     * initializes the Attribute finder module. creates a connection with JDBC database and the
     * retrieve attribute names from following sample table
     * +--------------+----------------+-----------------+---------+
     * | ATTRIBUTE_ID | ATTRIBUTE_NAME | ATTRIBUTE_VALUE | USER_ID |
     * +--------------+----------------+-----------------+---------+
     * | 1 | EmailOfUser | asela@gmail.com | 1 |
     * | 2 | EmailOfUser | bob@gmail.com | 2 |
     * | 3 | EmailOfUser | peter@gmail.com | 3 |
     * | 4 | CountryOfUser | SL | 1 |
     * | 5 | CountryOfUser | USA | 2 |
     * | 6 | CountryOfUser | UK | 3 |
     * | 7 | AgeOfUser | 23 | 1 |
     * | 8 | AgeOfUser | 19 | 2 |
     * | 9 | AgeOfUser | 31 | 3 |
     * +--------------+----------------+-----------------+---------+
     *
     *
     * @throws Exception throws when initialization is failed
     */
    public void init(Properties properties) throws Exception {
        /**
         * JDBC connection parameters
         */
        String dbUrl = properties.getProperty("databaseUrl");
        String driver = properties.getProperty("driverName");
        String userName = properties.getProperty("userName");
        String password = properties.getProperty("password");
/**
 * SQL statement to retrieve all attributes from database
 */
        String sqlStmt = "SELECT * FROM UM_USER_ATTRIBUTE";

        Connection connection = null;
        PreparedStatement prepStmt = null;
        ResultSet resultSet = null;

        dataSource = new BasicDataSource();
        dataSource.setUrl(dbUrl);
        dataSource.setDriverClassName(driver);
        dataSource.setUsername(userName);
        dataSource.setPassword(password);

        try {
            connection = dataSource.getConnection();
            if (connection != null) {
                prepStmt = connection.prepareStatement(sqlStmt);
                resultSet = prepStmt.executeQuery();
                while (resultSet.next()) {
                    String name = resultSet.getString(2);
                    supportedAttributes.add(name);
                }
            }
        } catch (SQLException e) {
            throw new Exception("Error while initializing JDBC attribute Finder", e);
        }
    }

    /**
     * This returns the name of the module
     * @return Returns a String that represents the module name
     */
    public String getModuleName() {
        return "JDBCPIPAttributeFinder";
    }

    /**
     * This returns the Set of Strings the attributeId that are retrieved
     * in initialization
     *
     * @return Set of String
     */
    public Set getSupportedAttributes() {
        return supportedAttributes;
    }

    /**
     * This is the overloaded simplify version of the getAttributeValues() method. Any one who extends the
     * AbstractPIPAttributeFinder can implement this method and get use of the default
     * implementation of the getAttributeValues() method which has been implemented within
     * AbstractPIPAttributeFinder class
     *
     * @param subject Name of the subject the returned attributes should apply to.
     * @param resource The name of the resource the subject is trying to access.
     * @param action  The name of the action the subject is trying to execute on resource
     * @param environment The name of the environment the subject is trying to access the resource
     * @param attributeId The unique id of the required attribute.
     * @param issuer The attribute issuer.
     *
     * @return Returns a Set of Strings that represent the attribute
     *         values.
     * @throws Exception throws if fails
     */
     public Set getAttributeValues( String subject, String resource, String action, String environment, String attributeId, String issuer) throws Exception {

        String sqlStmt = "select ATTRIBUTE_VALUE from UM_USER_ATTRIBUTE where ATTRIBUTE_NAME='" + attributeId + "' and USER_ID=(select USER_ID from UM_USER where USER_NAME='" + subject + "');";

        Set values = new HashSet();
        PreparedStatement prepStmt = null;
        ResultSet resultSet = null;
        Connection connection = null;

        try {
            connection = dataSource.getConnection();
            if (connection != null) {
                prepStmt = connection.prepareStatement(sqlStmt);
                resultSet = prepStmt.executeQuery();
                while (resultSet.next()) {
                    values.add(resultSet.getString(1));
                }
            }
        } catch (SQLException e) {
            throw new Exception("Error while retrieving attribute values", e);
        }
        return values;
    }
}

You need to create your PIP module by using this class.

Register your PIP module in WSO2 IS

I will provide the action that you should follow to get your module running.

• Build your module and copy the jar to CARBON_HOME/repository/components/lib
• Copy the JDBC driver to CARBON_HOME/repository/components/lib ( here the mysql-connector )
• Register your attribute finder by adding it to CARBON_HOME/repository/conf/security/entitlement.properties as follow ( make sure that you change the dbUserName and userPassword )

PIP.AttributeDesignators.Designator.2=org.wso2.identity.samples.entitlement.pip.jdbc.JDBCAttributeFinder
org.wso2.identity.samples.entitlement.pip.jdbc.JDBCAttributeFinder.1=databaseUrl,jdbc:mysql://localhost:3306/piptestdb
org.wso2.identity.samples.entitlement.pip.jdbc.JDBCAttributeFinder.2=userName,dbUserName
org.wso2.identity.samples.entitlement.pip.jdbc.JDBCAttributeFinder.3=password,userPassword
org.wso2.identity.samples.entitlement.pip.jdbc.JDBCAttributeFinder.4=driverName,com.mysql.jdbc.Driver

I am attaching the db script that can be used to generate data which is required for this sample, the sample module and the mysql-connector jar to make your job more easy

Test your attribute finder

Given below is a sample policy which can be used to test your new JDBC attribute finder.
You can find the uploaded policy here.

 
 
 
 
 foo
 
 
 
 
 
 
 
 
 
 
 bar
 
 
 
 
 
 
 
 
 
 
 
 18
 
 
 
 
 
 30
 
 
 
 
 

This policy says that only the users whose age is between 18 and 30 can access the resource “foo” and perform action “bar”

-  Copy the above sample policy to an xml file, start WSO2 IS and upload the policy file through Policy Administrator.

         Follow Policy Administration > Add New Entitlement Policy > Import Existing Policy

-  Enable the policy through Policy View

-  Publish the policy through Policy Administrator.

-  Click on Tryit and send a request. ( given below is a sample request which you can download from here)

foo




Bob




bar

Now you would see that your newly created attribute finder has come into play :)

There are some configuration changes if you are trying a 3.x.x version. This would help you to identify those changes if you are using an earlier version.

Convert XML String to OMElement and extract values

A simple tip which saves a lot of your time...
You can simply convert XML string to an OMElement as below

      OMElement resultElement = AXIOMUtil.stringToOM(xmlString);

Extracting values from XML

Sample XML code

        Gambardella, Matthew
        
        Computer
        44.95
        2000-10-01
        An in-depth look at creating applicationswith XML.
    
    
        Ralls, Kim
        
        Fantasy
        5.95
        2000-12-16
        A former architect battles corporate zombies,an evil sorceress.
    
    
        Corets, Eva
        
        Fantasy
        5.95
        2000-11-17
        After the collapse of a nanotechnologysociety in England.
    
    
        Corets, Eva
        
        Fantasy
        5.95
        2001-03-10
        In post-apocalypse England, the mysteriousagent known only as Oberon.
    

Java code to retrieve values

OMElement resultElement = AXIOMUtil.stringToOM(xmlString);

        Iterator i = resultElement.getChildren();
        while (i.hasNext()) {
            OMElement book = (OMElement) i.next();
            Iterator properties = book.getChildren();
            System.out.println("====== book =======");
            while (properties.hasNext()) {
                OMElement property = (OMElement) properties.next();
                String localName = property.getLocalName();
                String value = property.getText();
                System.out.println(localName + ": " + value);
            }
        }

Result

====== book =======
author: Gambardella, Matthew
title: XML Developer's Guide
genre: Computer
price: 44.95
publish_date: 2000-10-01
description: An in-depth look at creating applicationswith XML.
====== book =======
author: Ralls, Kim
title: Midnight Rain
genre: Fantasy
price: 5.95
publish_date: 2000-12-16
description: A former architect battles corporate zombies,an evil sorceress.
====== book =======
author: Corets, Eva
title: Maeve Ascendant
genre: Fantasy
price: 5.95
publish_date: 2000-11-17
description: After the collapse of a nanotechnologysociety in England.
====== book =======
author: Corets, Eva
title: Oberon's Legacy
genre: Fantasy
price: 5.95
publish_date: 2001-03-10
description: In post-apocalypse England, the mysteriousagent known only as Oberon.

Get started with WSO2 App Factory

AppFactory is an elastic and self-service enterprise DevOps platform to mange applications from cradle to grave. This is a 100% free and open source solution developed by WSO2 which covers the whole lifecycle of an application. You will be facilitated by all the required resources for the application created in one go. Just use ones, you will see the difference :)

Getting started is very simple. It is all online running on cloud. But I will go through step by step, so you do not miss anything.

Create an account

Click the 'Register' link in App Factory live URL

You need to fill the following form to get registered.

Do not worry about the phone number field, just give some number there if you do not like to put the actual number.

If your registration is successful you will be asked to check your email.

Change the default password

Using the URL sent to your email address given at register time, log in to the system and change password.
(Remember, that log in URL can be used only ones and you need to change the default password in that log in)

Then we will change the default password with the one you gave

 Log in to AF

Use this link and log into the system using your new password. You will see the following page just after the log in.

But wait for few seconds, we are creating a default application for you :)

When you navigate to the application that just now got created, you will see the set of features and the functionalities that are bound with your application. 

I will discuss about how to manage the applications with App Factory in a recent post.