Moved - to Wordpress

Dear Blogger,

If someone come to visit me here, please let them know that I moved to a new place at WordPress.com. Please forward any letters/packages you may receive to my new home: https://thammegowda.wordpress.com/

Thanks and Good bye.

-
TG

Maven project with scala and Java sources

Some of the best tech I learned these days are scala, spark, spark-MLlib and graphx.
My work with spark forced me to learn scala even though I was very comfortable with Java8. I admit I had attempted to learn scala earlier but gave up with the prejudice that Java8 caught up with scala in terms of richness of language features. Actually now I figured out that Scala is 10x improvement over java (< 1.8). Java language with version 8 upgrade caught upto 4-5x and still missing bunch of rich features of scala language.

That is being said, i wanted to document how to setup a maven project which can build both the scala and java source code. (Do I need to say that we can call scala routines from java and viceversa?). This is very useful because we can just add the scala plugin to existing maven java projects and start to make use of scala without any extra effort.

mkdir my-project
cd my-project
mkdir -p src/main/java src/main/scala src/main/resources/ \
                src/test/java src/test/scala/ src/test/resource
emacs/edit pom.xml  # and copy paste the below content to

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    
    <groupId>com.example</groupId>
    <artifactId>test-project</artifactId>
    <version>1.0-SNAPSHOT</version>
    
    <packaging>jar</packaging>

    <name>Test Project</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <scala.version>2.11.8</scala.version>
        <exec.mainClass>JavaMain</exec.mainClass>
    </properties>

    <dependencies>
      <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
      </dependency>
      <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
      </dependency>
    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                  <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions> 
            </plugin>
        </plugins>
    </build>
</project>

Let's put a sample Scala code

emacs src/main/scala/ScalaMain.scala # paste the below conents

object ScalaMain{
     def hello(){
           println("Hello there from scala")
     }
}

Lets call scala code from java :

emacs src/main/java/JavaMain.java  # paste the below contents

public class JavaMain {
     public static void main(String[] args){
          System.out.println("Hello From Java");
     ScalaMain.hello();
     }
}

Lets Compile and run

mvn compile exec:java  # the class is already specified in pom.xml as exec.mainClass

[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Test Project 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
.........
[INFO] 
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ test-project ---
Hello From Java
Hello there from scala
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.239s
[INFO] Finished at: Mon Apr 04 00:42:25 PDT 2016
[INFO] Final Memory: 14M/216M
[INFO] ------------------------------------------------------------------------

Summary : 
I just documented the pom.xml configuration for a maven project which can compile both scala and java sources.

Multiplication table using matrix in spark

In this post, I am sharing a code snippet to demonstrate multiplication table in spark.

In addition, it also demonstrates the following:

• How to create a custom Spark RDD
• How to create an Iterator/Generator
• How to work with matrices in spark

Assumption :

• You are using JDK 8.
• You added spark and spark-mllib libraries to the classpath. I assume you use something like Maven and add these dependencies.

---

import org.apache.spark.Partition;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.TaskContext;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.mllib.linalg.distributed.CoordinateMatrix;
import org.apache.spark.mllib.linalg.distributed.MatrixEntry;
import org.apache.spark.rdd.RDD;
import scala.collection.AbstractIterator;
import scala.collection.Iterator;
import scala.collection.mutable.ArrayBuffer;
import scala.reflect.ClassManifestFactory$;

/**
 * This class illustrates multiplication table on spark!
 * @author  Thamme Gowda N
 *
 */
public class Main {

    /**
     * This RDD has numbers in it.
     * This RDD can be used for illustrating math operations on Spark
     */
    public static class SequenceRDD extends RDD<Long>{

        private long start;
        private long end;
        private Partition[] partitions;

        public SequenceRDD(SparkContext _sc, long start, long end) {
            super(_sc, new ArrayBuffer<> (),
                    ClassManifestFactory$.MODULE$.fromClass(Long.class));
            this.start = start;
            this.end = end;
            this.partitions = new Partition[]{
                    () -> 0 //just one part at index 0
            };
        }

        @Override
        public Iterator<Long> compute(Partition split, TaskContext ctx) {
            //we have only one part, so this is it
            return  new SequenceIterator(start, end);
        }

        @Override
        public Partition[] getPartitions() {
            return partitions;
        }
    }

    /**
     * This iterator yields numbers so we can build an RDD on it
     */
    private static class SequenceIterator
            extends AbstractIterator<Long>
            implements java.util.Iterator<Long> {

        private long nextStart;
        private long end;

        /**
         * Number generator for [start, end]
         * @param start the start
         * @param end the end, inclusive
         */
        public SequenceIterator(long start, long end) {
            this.nextStart = start;
            this.end = end;
            assert end >= start : "Invalid Args";
        }

        @Override
        public boolean hasNext(){
            return nextStart <= end;
        }

        @Override
        public Long next() {
            return nextStart++;
        }
    }

    public static void main(String[] args) {
        long n = 1_000;
        String outpath = "multiplication-matrix";

        long st = System.currentTimeMillis();
        SparkConf conf = new SparkConf()
                .setAppName("Large Matrix")
                .setMaster("local[2]");
        JavaSparkContext ctx = new JavaSparkContext(conf);
        JavaRDD<Long> rdd = new SequenceRDD(ctx.sc(), 0, n)
                .toJavaRDD()
                .cache();

        JavaPairRDD<Long, Long> pairs = rdd.cartesian(rdd);
        JavaRDD<MatrixEntry> entries = pairs.map(tup ->
                new MatrixEntry(tup._1(), tup._2(), tup._1() * tup._2()));
        CoordinateMatrix matrix = new CoordinateMatrix(entries.rdd());
        matrix.toIndexedRowMatrix()
                .rows()
                .saveAsTextFile(outpath);

        System.out.printf("n=%d\t outpath=%s\nTime taken : %dms\n", n,
                outpath, System.currentTimeMillis() - st);
        ctx.stop();
    }
}

Comparing cubic time complexity algorithm with a fourth-power complexity

   I thought of comparing n³ and n⁴ running times to feel what it means to have a slow algorithm.

Let us suppose we have an algorithm with cubic runtime complexity:

 int n = 2500;
 int[] arr = new int[n];
 System.out.println("Start");
 long st = System.currentTimeMillis();
 for (int i = 0; < n; i++) {
   for (int j = 0; j < n; j++) {
     for (int k = 0; k < n; k++) {
       arr[k]++;
    }
   } 
 }
 System.out.printf("%dms\nDone!", System.currentTimeMillis() - st);

For n = 2500, it completed in 1.827 seconds on a modern i5 CPU :

 Start  
 1827ms  
 Done!  

Now, an algorithm of power 4 time complexity,

 int n = 2500;
 int[] arr = new int[n];
 System.out.println("Start");
 long st = System.currentTimeMillis();
 for (int i = 0; i < n; i++) { 
   for (int j = 0; j < n; j++) {
     for (int k = 0; k < n; k++) {
       for (int l = 0; l < n; l++) {
         arr[l]++;
      }  
     }  
   }  
 }  
 System.out.printf("%dms\nDone!", System.currentTimeMillis() - st);

With just one additional loop, for the same n=2500, it took:

Well, it didn't complete... I waited 15 minutes, yet no answer!!
EDIT: It completed after 47 minutes!

 Start  
 2863936ms  
 Done! 

Why is this comparison important to me?
  This comparison became interesting to me because I have implemented a Tree Edit Distance algorithm which has a fourth-power time complexity of the input size (number of nodes in the trees). I used this algorithm on HTML DOM trees which have around 1000 to 2500 HTML elements.

Summary 
 Thus, an algorithm with power 4 is not an acceptable solution for tree edit distance for web pages with more than 1000 HTML elements.
I am seeking a better algorithm.

Digit grouping for very long numbers in Java

When we need to write a Billion on a sheet of paper we don't just write 1000000000, because it is hard to see if it is exactly a billion or  hundred million in the very first glance without counting the digits. Instead, grouping the digits is common practice so it can be written as 1,000,000,000. It is evident that grouping the digits improves the readability of numbers.

    However, all these days I never saw anywhere grouping the numeric literals in the software code.  Recently I was surprised to see that Java language supports digit groups by using the underscore ('_') characters as group separators. For example, the following is a perfectly valid way to store a decimal Billion in java.

int n = 1_000_000_000;

Some day I will do it - not a procrastination

    'Someday I will do it' - might seem like words of a procrastinator. But let's not judge so quickly from a single sentence. Maybe today is not the best day, so 'someday in the future' will be better than 'never'.  I agree that saying 'someday I will do it' is indefinite and I haven't come across any system that accept 'someday' for an event. What is interesting is - when we wake up in the morning, how nice it is to know that the day we hoped in the past has arrived!
     I have said these phrases many times - 'Someday I will do it', 'someday I will have it'. Fortunate enough, that day has arrived to fulfill many of my wishes. I had to pack my stuff and run when I realized it. Out from my huge 'basket of wishes', I am going to pick few of them out because they are now real. I wished for going to a grad school, meet people, make friends, for the sake of knowledge and experience in life. I am excited to have my dreams coming true. We make wishes all the times, but rarely we find time to fulfill them. Some wishes have deadlines; if we can not fulfill them within the bounds, we have to let go of them. I know that failing to fulfill dreams is very disappointing. I am glad to say I did not fail here.
    It's been a while since I wrote any post here in my blog. Let me catch up! Three weeks prior to this day, I flew from Bangalore to Los Angeles, I also visited London on the way! I am now in USC campus near downtown LA.

A lengthy day with 20 hours of daylight
Memorable is my first international flight
From a colony in the east
To a colony in the west
Via the capital of the kingdom
History in hand is ridding-off the boredom

     If I draw a circle centered at Bangalore International Airport with the radius of 40miles, the area inside the circle covers everything including the place I born, schools and colleges I went to, offices I worked for, relatives and friends I have visited. I seriously wanted to travel to a different part of the world for the sake of experiencing the change in my mid-20s. The someday that I hoped in the past is here with me in the present.
     My classes started this week. Many exciting events are happening in and around the university campus.

I will be back with a new post.

How did I land in startup?

I like to dwell upon the time, but this time I am not writing about time from cosmos, instead I am going to share few incidents from my recent past and the lessons I learned. This occasion marks the completion of my three years of working in a startup environment. I strongly believe that my tomorrow will be the result of the choices I make today. With the same belief, I can say whatever I am today is the result of choices I made in the past. People say having a dairy and writing down about the thoughts and opinions will shape the personality for better as that is the time one speak with the self. This is the time to look back at the self to appreciate the good while also correct the wrong doings. I don't have my dairy to tear few passages to stick together and tell you the story, but I try to be honest and recollect the facts from self-memory.

How did I land in SimplyPhi?

I feel my entry to SimplyPhi as a serendipitous chain of events that somehow happened. Let's begin by traveling back in time and I am sure you will be equally surprised too. 

October 7, 2011: I was offered a job from a company named TC*. They are the first guys to come to our college campus for recruitment. I was in final year of my undergrad studies. People who head our college placement department sold them like hot cakes by stating the facts that

   1) they are one of the biggest companies of our country with so many branches around the globe.

   2) They are one of the best and safest place to work as they ensure that our job is safe irrespective of recession or anything like that.

   3) Most of our seniors work there! They have a great brand value.

I did not experience the real sense of a job interview. My name was among the college toppers' list so I received a direct entry. The HR round interviewer looking at my records said that my records show my brilliance and annotating to my name said that I am a 'Gowda' and he is a 'Gowda' too, hence my interview was complete. Well, this point in the timeline, I was happy that I have a job; My parents were happy that I will be joining a big company to earn big money and the job will be safe. Things were normal!

January 20, 2012: Somebody named Gautham sent me a tweet asking for an email-ID to begin a private conversation. I shared my email-ID and then we had exchanged few emails. I was not sure who he was in the beginning. In another mail, he articulated about what he was doing, and his ambitions with his emerging startup Jnaapti.

January 21, 2012: Gautham and I had a long talk on a phone, he talked about so many topics, mostly related to startups. One thing he said was TC*(Company which hired me in campus) was not the best company for my level of caliber(In my mind – He is true, indeed! Google should've come to our campus for recruitment). He said I shall be placed to a better company who utilize my enthusiasm, skills while also offer a better stage to learn.

I even mentioned to him that my dream company to work is Google. In a later email, he shared many links about startups, about startups in the news and the startups to which he is connected.

And one thing he made clear - “Remember that Google, Yahoo, Facebook etc started as startups one day.”  'Why not think of building one?' I was sold to that thought in the very next moment. Thank you, Gautham, what you did that day changed my thinkings and offered me enough confidence to go for it.

February 2, 2012: A screening call from Monis Khan, Director of SimplyPhi.

We had a talk and he invited me to their office for discussions. I asked for time.

April 28, 2012: Interview at SimplyPhi Software Solutions.

I literally took many days to get back to SimplyPhi after the screening call. The interview went fine as they made me implement few algorithms. I wrote C code. This was my second interview and I had received my second offer. I liked the place, the liberty in work culture and the idea of being fully responsible. Moreover, I wanted to become a very good programmer. No, I wanted to become extremely good at programming. These people promised me that they will make me write code every day and will guide me as they are programmers too. I hated the idea of being idle on bench or taking support calls on the first offer from TC* instead of constructing something, I was not even sure whether I will get a programming role if I join(as I get to know about this from seniors)

May 2012: The decision-making time.

While very few of my friends said it's good to not join TC* and join a startup, but many of friends said it's not wise. Even my dad was not so pleased to digest that I am not gonna join TC* offer which is in hand. He might have learned from his friends circle about TC*, its hugeness and ubiquitousness. But my decision was already made in my mind and all I needed to do was to convince them. At last my parents said - don't worry much about the job, if something goes wrong we can look after you, you can look after our farming, we have this for you. That was bold; thank you dad, you let me do what I wanted to do! Summary of this - When you are not worried about losing job or failures, you get the courage to do whatever you wish to do. For the next two years, I want to be a programmer, I want to learn to build Softwares, and SimplyPhi felt like a better deal for it.

June 14, 2012: Last day in college, I had the final demo of my project. My last days in college were not very enticing(in other words - sad, I just wanted to get out of that). I didn't want to go dormant, as the dormancy would only hurt my mood. I was seeking something fresh, exciting, challenging, yet fun to keep me busy. I was thinking of rushing to Bangalore

June 16, 2012: I joined SimplyPhi Software Solutions.
 I was supposed to join few days later, but I was very eager.

My new life began from this day. I was a villager and now I live in a metropolitan city known for IT. I was a guy from a tier-3 college, but now I am conversing with best minds from IITs. They encouraged me to fill up my shabby fundamentals, I re-read algorithms, computation theory, data structures and some pieces of mathematics which are essential to think like a real programmer. At times, I used to have 40+ tabs open in chrome most of them were from Wikipedia. As promised they made me write code and solve the variety of problems. Be it web servers, web crawler, android apps, distributed systems, databases, search engines or anything like that, I didn't say No (let me be honest: I had liberty to say no, rarely I said NO for the tasks I didn't like to do. No to a client with a decade old windows server with all the shitty outdated enterprise stuff. No to some web interface while my mind was drifting to NLP research). The best part of this service oriented startup is  I got a variety of problems to solve in less time, which I can summarize in my own words – learn in the money of clients in the exchange of my time. Later on I became part of a product based startup, Datoin. All these cannot be completely comprehended here.

What people say - why not join a startup? 

Joining a startup as a fresher was a pretty scary thing to do for me about few years ago. Because it seems lot of work. Because it seems complicated work. Because my college didn't teach me well and I didn't know if I will be able to handle it. Because it seems risky and company may be shut down as they have more chances of being bankruptcy or miserably failing. Because every educated uncle and aunty I meet will ask me where I work, and they will counter me when I say a company whose name is not well known. Because there are lesser people at work and less possible new friends. Because startups don't have much money to do extravagant offers. Because they won't have a cab service or free food. Becuase they don't have onsite trips. Because....

Stop! I can now separate the genuine claims from all the speculations.

What I say - why work in a startup?

I use this as an opportunity to answer few questions which I am being asked by many. I like answering in line, one-on-one.

Because the work seems much in quantity, quality, and complexity.
I would like to tell you a story which I heard somewhere a while back. Once there was an eighteen-year-old boy, he was assigned the task to drive to a railway station on a rainy dark night to pick up a distant relative. Does this assignment seem too much for the boy? If the visiting guest is an old man who keeps yelling at the boy – Yes it is too much indeed. Suppose if the visiting guest is a girl in sweet sixteen whom the boy has developed crush recently – the boy will do it out of pleasure. Though the time and effort required for the trip will be same irrespective of an old man or his crush, the boy will have a different feeling about it based on the person. That means nothing is too complicated when you enjoy. The first few years may be the foundation for the rest; it has to be hard to be strong. I loved programming, I love building Softwares, when things get complicated, I shall learn to deal with them and there is a fun/kick in solving the complex problems. If you love building stuff, startups need your time and you need to do so called 'field work' in startups. 

Because I am not from an elite college, my college didn't teach me, I am not sure if I am ready for it.

 Agreed that elite colleges alumni had gone through a better environment, but the past is past.  Don't ruin your today by complaining that your yesterday was not elite. We all keep looking for opportunities and we should act when we see one. If you think there is something you can do, sure there will be many things you can do. If you think you can learn on your own, then there is no one to stop you on the internet. You should seriously worry when you are physically or mentally challenged for learning and picking new stuff. When you see people with good caliber quitting their high paying jobs to do something on own to realize their ideas, why not go and join them and put a hand of support? You don't need to do this just for the sake of feeling good. While you sit and work along with them you can catch up with them and see how you match with them. If things go well, it's gonna be mutual benefit. You help the team to realize their dreams, they help you become what you want to be by offering enough support and opportunities.

Because it seems risky and you can be out of the job any time.

Let me tell you the harsh truth: The corporates hire you, they alter your hobbies, they make you like whatever is good and profiting for them, they make you get used to laziness while they keep you as redundant candidate for risk recovery and finally when they are in trouble or they find a better candidate in the queue, they will fire you, for God's sake- you will be in trouble, not the company which hired. On the other hand it doesn't matter whether the company is big or small, whether well established or starting up, it doesn't matter whether your seat is safe today or for next few more days; if you are not useful, you will be let off by the companies as well as most people you know. Being in twenties without many family commitments(like marriage, kids and all) but thinking that you're doomed on the next day you're removed from the job is immature. You should seriously worry when you don't have the skills that are useful for others to employ you. The only thing to save you from this mess is the child inside you who keep on learning to be a better self and the grown up in you who is ready to adopt to changes. If you are working in a startup that is trying to solve the problems that are worth solving, you are going to be useful for many others too. For instance, just after one year of my endeavor in my first startup, I started to receive messages in LinkedIn asking if I am available for Job switch. Some of them are from HRs of well-known product based MNCs. Few years back I'd have loved to join them straight after the college. Recently I was offered an offer of almost twice the salary I draw currently, but I had to turn them down. The final word on this anecdote is - Keep a green tree in your heart and perhaps a singing bird will come.

Because every educated uncle and aunt asks where I work. I need to have a brand name.

People love to associate with brand names and love to speak about big things, but very few of them associate with the quality work. This is real, and I have faced it. Occasionally when I meet some of my relatives, people say their boy/girl work in well-known company and was offered to travel to other part of the world. Often I feel irritated when they boast that their boy work for X company which sent to Y country for onsite work, now he is earning Z lakhs. One of the worst conversation I had that I cannot forget is when one distant relative gave his remark – “I heard you're quite good with studies why didnt you go for a Government Job?”. I want to live for the day when people really talk about the kind of work they do before taking about its consequences. I say myself - You are not your resume, you are your work, so a better you is when you do better work. Anyways, it doesn't matter what all the uncles and aunts you meet on road feel about you, what it really matters is feelings of your family.

Because less people around and less friends.

 I accept the fact that these are the choices I made. I might have made only few friends, but most of the people I happened to work along are dreamers and achievers. My lunch time discussions with my colleagues were always enjoyable, yet there were serious topics involved. Some of these conversations have impacted so much on me that they changed the way I now perceive the world. Who knows the current startup might have offered common lunch time and place for a next Bill Gates and a next Steve Jobs! they are not many. 

No pickup and drop, No free food.

I agree that they are good to have. But, seriously you wish to join a company because they have a nice pickup bus or they serve your favorite dish at lunch? Now I see what these big companies are making. They want you to get used to them so you can not quit even if you would like to quit. I chose to be independent and free.

Startups don't have too much money and no extravagant offers and perks

This is true as well as false, depends on the startup, your role and the remaining people in your team. When you join an early stage startup which has not yet gone for fund raising, then the odds will most likely to be compensated with the stake. All the well versed startup people talk about the stake instead of salary in this space, but that is for well-versed people. What I recommend for a person who is just out of the college and is trying to make his mark is: don't let the money decide what you do and what you don't do at this phase in your life. Let your first focus be on learning and exploring what you love. I can promise you, there is lesser politics at the place where there are few people involved than at the place where many people are involved. If you happened to meet right people who appreciate your hard work, you will be given appraisals like every other company. However this is not for sure, I have come across examples of my friends who happened to be underpaid and their hard work is overlooked. If you are surrounded by a bunch of greedy businessmen who keep all the stake with themselves and also feel the effort/risk you put is insignificant compared to their's, then you shall have a second thought. If you ever feel you're not at the right place to grow old, learn what you can, pack your stuff, and devise an exit. But if you're around handful of good men who appreciate your presence during the journey of conceiving an idea to MVP to scalable product/service, if everything goes fine you can be unprecedentedly rich few years later.  or if everything goes down you have to start again. That means the money part is quite uncertain. As a newbie out of college, set your expectations right, don't expect to be a millionaire or a billionaire(like Mark or Bill) in a night. Just focus on learning everything about starting-up, scaling-up so you can either land in your next high paying dream job when you plan to surrender your skills for money, or use these learning for starting up your own endeavour.

EDIT :

Looks like I added so much spirit about startup. Every Tom, Dick and Harry aren't encouraged. In spite of saying this, join startup only if you can foresee what is waiting for you.

• More learning than what you did during your college. 
• More pressure than what you had during the night before your exam; occasionally when things break, slip or don't work the way they suppose to work.
• More complexity in the problems than you might face otherwise.
• More responsibilities, multiple roles to take care of. You will be an admin, a developer, an architect, and a tester. Hence, you will be completely responsible for whatever you do.

 I have more exciting news which will be coming up soon in another post. If you have any question worth asking and getting a reply from me, go ahead and use comment box below!!