10 reasons why commercial companies should do open source software

  1. Positive PR
  2. External targeted contributions. For example, most people submitting patches to open source software are facing some issue as a user. If they are experiencing an issue, others may also be experiencing similar problems. This is a much better way of driving development, compared to doing it post expensive market research.
  3. Marketing
  4. Recruiting vehicle
  5. Increasing the size of markets. For example Google is trying to bring more people online with Android to increase the size of search market.
  6. Commoditizing complements : http://www.joelonsoftware.com/articles/StrategyLetterV.html
  7. Reducing the cost of training new employees
  8. Influencing the direction of an industry. For example setting standards for server hardware with Open Compute Project.
  9. Free testers
  10. Improving employee morale

Agile software development process explained in simple terms

Agile is a combination of two processes that are designed to develop software. The two methodologies that form Agile are : Scrum and Kanban.
Key to understanding agile is understanding the objective tree. Here is an example objective tree for a company like amazon.com which aims to build an e-commerce website.

organization-objective-tree
Organization Objective Tree

There are many people who argue about the number of levels in the tree and how they should be called. But the key insight here is that :
a) Objectives form a tree
b) When child objectives are done, the parent objective is automatically done

There are two methodologies within Agile : scrum and kanban.

Scrum

Process

Objectives are broken into user stories. A prioritized list of user stories is called a backlog. Product management owns the backlog.

Objectives are achieved by working on them in 2 weeks sprints. Team pulls the maximum number of stories that can be completed into the sprint backlog and starts working on them when the sprint starts. Sprint backlog is owned by the team.

Every day of the sprint there is a 15 minute standup meeting where team members describe 3 things :

  1. What they worked on yesterday
  2. What they are planning to work on today
  3. Blockers ( if any )

This is an opportunity for micro-course-correction. If a story was done, team members can move to help with another story. If there is something that blocks a team member, management can move to unblock them. If a story might spillover, it can be called out.

At the end of the sprint there is a sprint retrospective to re-evaluate and course correct at a higher level.

Every N sprints ( for a finite value of N ) there is 1 refactor sprint. No new features are delivered in this sprint. Architectural runway is prepared for incoming stories and code is made more maintainable.

Why is the process like this ?

Reason for product management owning the backlog is to communicate clearly what the current priority is. This can be challenging in a changing environment. For example, management may ask the team to support an application on Windows 2012 platform and mark it high priority since largest customer base is on that platform. Afterwards it could turn out that highest paying customers are on RHEL 7.1. Management can communicate this information to engineering by adding a new story to support RHEL 7.1 and deprioritizing the story for Windows 2012 support.

Team owns the backlog because the team is responsible to ensure that the features ( stories ) committed at the beginning of sprint is done by end of sprint. This is why it’s the team that need to pull the stories into the sprint ( and not the manager ) so they feel a sense of ownership.

Standups give an idea about the pulse of the sprint : whether the team is on track, if it needs adjustment etc. Since software development is a collaborative process, it can get blocked on other people. For example : UI team can get blocked on UX mockups, developers can get blocked on APIs that were promised by other team members, one team member may be blocked for code review from another etc. Daily meetings help with dealing with these problems as they arise. It also enables the team to help others, for example, if someone airs a database problem, someone else on the team may have encountered and solved the problem before. Standup meetings should be short and sweet. If they are too long, since they occur everyday, they become a productivity drag.

The sprint retros are an opportunity to improve things at a higher level. For example, developers may notice that the CI system could run a larger portion of automated tests, reducing QA burden. Developers may point out that the code is getting too complex in one area and request refactor stories.

Expectations

1. If a story is in the backlog, it may get done. If a story is not in the backlog, it will never get done.

Why is this expectation there ? Sometimes product management may request a feature. Engineering does not know if that feature was important, whether it was a priority etc. Moreover, there could be multiple managers requesting different priorities for stories fuelling confusion. This expectation sets up management to close out the relative priority of objectives before people set out to achieve them.

There is no such thing as an implicit requirement. Management owns the product backlog.

2. Target spillover for any sprint is zero

Why is this expectation there ? For hard software engineering problems it is required that the team is a highly committed one. An uncommitted team member will usually bring down the whole team since the job is collaborative in nature. Target spillover of zero ensures that the team remains focussed on the problem at hand.

3. No activities are planned mid-sprint

Why is this expectation there ? This is to help the team remain focussed so that the cost of context switching is lowered. It is called a “Sprint”, not a “Trot” or “Gentle Walk”. Management is expected to defend the team from external influences. Course corrections are possible at the end of sprint. Therefore, it is important to keep the sprints as short as possible to get a meaningful amount of work done. The length of your sprint should be the minimum amount of time it takes you to respond to external influences. Usually, it is 2 weeks.

4. QA should raise a ticket if there is a problem.

Why is this expectation there ? It is possible that QA notices a problem and communicates it verbally or through email. Now people may go on vacation etc cutting the team’s ability to deliver.

There is no such thing as an implicit ticket. Raise it in issue management software.

5. No story is done until :
a) All code is reviewed
b) There are no open tickets
c) Documentation is complete

Why is this expectation there ? Code reviews are important because it lets someone else get familiar with that code base allowing them to take over in the event of the author’s absence. It is important to close all open tickets before calling a story done since otherwise your development process simply turns feature requests to bugs. Documentation should be developed lock step with code since that is when it is easiest to write it.

6. Do not test from feature branches

Why is this expectation there ? Someone could merge something else into the development branch making the feature tested in feature branches to be unviable in development branch. Development branch is what ultimately gets shipped to users so that is where things should be tested.

7. Stories are of the form “As a [user] I want to do [task] so that I can [accomplish goal]”. If they are not in that form, it can be safely deleted.

Why is this expectation there ? Often stories may be created with no clarity on what the next step is. Often developers may misunderstand the requirement and deliver a different feature requiring multiple unnecessary round trips and feedback loops to complete the feature. If the story is not in the prescribed form, it can be deleted and considered non-existent since it motivates people to state the objectives of the story explicitly.

A few points

  1. In Scrum the keywords are focus and course correction.
  2. Manager – employee relationship is co-operative. The team does everything to make the manager look good ( by delivering high quality features that delights users ) and managers does everything to setup the team and employees for success : removing impediments, giving them tools and access they need to meet the objectives.
  3. Dev – QA relationship is quasi-adversarial. QA tries to defend the customer demanding issues be fixed within reason. Dev team appreciates QA for finding the bugs they have missed that could have caused them to deliver an inferior product.
  4. Agile requires self motivated team. An under-motivated team may over inflate estimates or under commit into a sprint.
  5. There is no such thing as half-agile. Agile gives management a certain amount of power and engineers certain amount of power. This is to improve efficiency by reducing friction and miscommunication. If management says : ‘I will accept these parts of the process that gives me power and ignore these parts that require me to relinquish power’ this process will fail. It requires humility on the part of management and engineering.

Kanban

This is usually for support. After product is developed, support is trained on resolving bugs that get into production. Support can’t wait 2 weeks to resolve issues. When tickets come into the queue people take them on and try to get through them as fast as possible.

Scrum is more proactive and Kanban is more reactive. Kanban is for situations where a 2 week turnaround time is unacceptable.

Waterfall

Agile came about as a response to the Waterfall model of software development. It helps solve the most important problems with waterfall such as quicker course correction, improving focus and reducing miscommunication.

In waterfall, the software developed in following phases ( one after other ) :

  1. Gathering requirements
  2. Designing solution
  3. Implementing solution
  4. Testing solution

Problems:

  1. A mistake in early phase cannot be detected until last phase.
  2. There is little focus as work drags on for years.
  3. People working on each step only communicates with people in previous step increasing communication errors.

Should Indian Railways be privatized ?

Should Indian Railways be privatized ? Looking up the answer to this question on the Internet would lead one to conclude that no, it should not be privatized. I am not convinced, I think it should be privatized.

Lets look at the arguments against privatization, one by one.

  • It is not clear how it will improve situation

The argument is that privatization and increased competition will improve the quality of service, drive down prices and spur innovation. It should reduce corruption and force employees to do their best. For a government owned company the employees have no reason to work hard, treat its customers well and figure out how to make everything better. They get paid regardless. This causes them to mistreat their customers and generally be lazy about their work.

The Indian Railways lost ₹30,000 crore (US$5 billion) in 2015. This is despite the fact that it is a high demand business with a government sanctioned monopoly. Booking a train ticket is so painfully difficult that it a subject of so many degrading memes on the Internet. The compartments are noisy and sleeper classes are filled with bed bugs. The toilets are dirty. The staff is unpleasant to deal with. The train stations are riotous and unclean. It smells of shit and urine.

I wonder how many levels of incompetence it takes to suck so badly at a job.

Privatization has helped the Indian telecommunication sector and aviation sector. It is possible for it to work in the rail industry as well. Even if it doesn’t help, it is at least worth a try.

  • Look at the state of American and British Railways

They are significantly better, aren’t they ? But even if it isn’t, there are a thousand reasons it could work better in India. Yes, I know there are a thousand reasons why it couldn’t work also. But why not give it a try ?

Concerning America, their landmass is 3 times that of India and they are a very affluent and urbanized country. This means that most of the travel is very long and hence very inconvenient to be conducted by train. So rail is used mostly for goods and human beings pay a little more to travel by planes.

  • It is a natural monopoly due to high cost of entry and exit

This is partially true. The high cost of entry and exit is largely due to high cost of building and operating railways, so we may need to have some government assistance with regard to this.

One solution to this problem is to let the rails themselves be owned by one company and combine this with multiple companies owning the individual trains. The train companies lease the rails from the rail company to provide their service.

Sure, some government assistance will be needed to acquire more land, ensure that there is stiff competition etc but these are all problems that have well known solutions. It can definitely not be a fire and forget endeavor.

Also I don’t understand how it becomes harder for individual investors to exit the rail business as most companies are traded on the public stock market. If you want to enter the rail business buy a few shares of the company at the Mumbai Stock Exchange. Want to exit ? Sell them at the stock exchange.

  • Some routes may be under serviced because they may be unprofitable

Form a government company to service only these routes using tax payer money. There is no need to keep the entire business in the public sector and cause incalculable misery to everyone in India to benefit a few.

  • The railway employs a lot of people these jobs may be lost in privatization

Privatization does not automatically cause loss of jobs. If a private railway business can operate with fewer number of people and automate away a lot of these jobs it would benefit the customer with lower fares. This money saved by customers can be used to buy new things which creates new jobs. See the parable of broken windows. Another possibility is to tax the rail companies and pay unemployment benefits to these people directly. At least the public will be spared from their laziness and callousness.

  • Defense concern. Countries like China could buy up all Indian Rail businesses and in the event of a war, discontinue them all

We can repossess these businesses in the event of a war. The other problem is if they sabotage it silently while they are holding it. This problem can be solved by stronger policing.

  • National pride and unity

A lot of arguments center around how Indian Railways is a matter of pride for Indians. Sorry, I would rather travel comfortably than with pride. Also, I don’t think it is a matter of pride given how absurdly mismanaged it is.

So anyway, I think most of these arguments are bogus or ill-informed. I have similar thoughts about the power industry ( power cuts are uncommon in Western countries ) since there is no innovation in solar, wind, nuclear power in India despite us having no serious oil reserves. I think India is a poor country on account of power. But more on that later.

What do you think ? Let me know in the comments section.

“Man of Steel” movie explained

So last night we watched “Batman v Superman”. This got me thinking about the prequel : “Man of Steel”. I just rewatched it and found that it is in fact a very good movie. Let me tell you why.

superman
Man of Steel

Superman has always been one of my favorite characters. This is not to say that I have watched and enjoyed every version of Superman in existence. In fact, I may not even like some incarnations of this character. But the fact remains that the essence of the Superman story is very interesting. So I am always surprised to find that many people dislike Superman deeply.

One reason I have heard for this strange phenomenon is that they think Superman is too powerful. He seems invincible. It is true that a story would be uninteresting if the hero had no chance of losing or if he was pitted against a weak villain ( “Merlin” TV series anyone ? ). This is not the case for “Man of Steel”. Zod is every bit as powerful as Superman. It is in fact a vital part of the story that Superman is outgunned and outnumbered in this fight. He has to fight many Kryptonians who are just as strong as him but born and bred to be soldiers. These Kryptonians trained their entire life to be warriors and you can see Superman struggling to match them. The truly awesome power of Superman is the hook for the story. But this does not make it dull if he spends the majority of his time standing up to guys a lot stronger than himself.

A lot of people are angry that at the end of the movie Superman kills Zod. I think it is an important plot point : this implies that Superman was never invincible to Zod, Kryptonians can kill other Kryptonians on Earth with their bare hands. The fight could very much have ended with the death of Superman, it is unclear how he survived at all. Maybe Zod was still distraught over losing his purpose and his people. Maybe his heart wasn’t fully in the fight. Maybe it was just plain luck.

I think it is very brave that a simple farm boy is willing to take up such dreadful odds to do the right thing. He didn’t sign up for any of this.

Superman Flying
Superman is a reluctant hero

There is also another aspect to this story. When Zod asks Superman to join him, it would have been the easiest thing in the world to say ‘Yes’. The movie depicts throughout, the loneliness that Clark feels. He feels like an outsider, a freak. It would have been a welcome relief to live on the ‘New Krypton’ and feel belonged. So why does he say ‘No’ ?

The simplest answer would be to say that he didn’t want to kill all those humans. But he did finish off the Kryptonian race ( his own ) in the process. Is it any better ? This is where the back story kicks in again. Every Kryptonian’s life is predestined, every man and woman born to a single predefined purpose. There is no place for chance or choice. With such a system, the ‘New Krypton’ would inevitably meet the original Krypton’s fate : infinite self-destruction. This is why Jor-El does not leave Krypton with Kal-El. When a race is doomed, it can only be wiped out so that a new one can have a hope of survival. This point, however cruel is a fact in the story universe.

Human beings may not make the same mistakes Kryptonians made, which means they have a shot at survival and expansion, not the infinite destruction that Kryptonians face. This is also why Jor-El was OK with sending Kal-El away from Krypton : he was a natural birth. He is not part of the system. Jor-El may have tried to bargain with Zod to let the Kryptonians live alongside humans in a last ditch attempt, but he knew that only humans have a serious chance.

Humans lead by Superman
Superman is willing to defend human capacity for good

Finally, I also want to make the point that it may not be Superman a lot of people hate. This point has been argued before, but I think we find it difficult to accept even a fictional character who is both good and successful. The reason for this is that he reminds us of our own shortcomings. We feel that we are justified in doing evil, so that we can be successful or happy. We feel it is impossible to live in this unjust and unfair world by being good. It is this lie that we tell ourselves that is challenged when we watch a Superman movie. We are not angry at Superman, we are angry at ourselves. Sometimes we project this anger towards all that is good. If that is true, we have to do better than that, don’t you think ?

How to write good software

Here is my theory on an algorithm to build great software :

  1. Figure out how the software ought to work.

    This is the most important part in the software design process. It is important that the user experience be nailed down before we begin all other processes. If we don’t do this first, people will find the easiest way to make the software work. Such an approach may not deliver the best quality experience to the user. It is important to note that it is essential, to involve the users in this step. We need to clearly understand the problems they are facing. How should the optimal solution to their problem work ? Is it even possible with the available resources and at the current state of the art ?

  2. Make it work

    This is the part where you build the software according to the expectations set by the user. At this point you do not have to worry about code quality, performance etc.

  3. Make it better

    In this step you have to make sure that the code is extendable, has high quality tests and is secure

  4. Make it pretty

    In this step, make the software beautiful. Make the text conversational and friendly. Introduce fancy graphics and other visual elements if possible.

  5. Make it fast

    Finally it is time to seek out the worst performing parts of the software and make it fast.

If you follow these steps, and in that order, you should have a high quality piece of software in your hands.

What is polymorphism ?

Polymorphism is touted as one of the awesomest features of Object Oriented Programming. But what exactly is the point of polymorphism ?

Lets take the problem of trying to automate Google search. Lets say your test case is :

  1. Open the browser
  2. Navigate to “www.google.com”
  3. Type “facebook” to the search text box
  4. Click on “Google Search” button
  5. Wait for the page to load
  6. Check if the results page contain a link to “www.facebook.com”

Clearly we need to find some way to drive the browser to do these actions. But how will the automation software know where the ‘search text box’ is ? How will it know where the ‘Google Search button’ is ? So, there is a technology called Xpath that lets you identify a particular element in a web page.

So here is the procedural code that solves the problem ( Language : C ):

int search_google_facebook()
{
    int test_passed = 1;
    char *search_box_xpath = "//searchbox";
    char *search_button_xpath = "//searchbutton";

    firefox_navigate( "www.google.com" );
    firefox_type( search_box_xpath, "facebook" );
    firefox_click( search_button_xpath );
    firefox_wait_page_load();
    test_passed &&= firefox_search_page_link( "www.facebook.com" ):

    chrome_navigate( "www.google.com" );
    chrome_type( search_box_xpath, "facebook" );
    chrome_click( search_button_xpath );
    chrome_wait_page_load();
    test_passed &&= chrome_search_page_link( "www.facebook.com" ):

    return test_passed;
}

Now lets look at the object oriented solution to the problem. Please keep in mind the Liskov Subsitution Principle : base class pointers can point to derived class objects. The code is in Java, so if you are unfamiliar with Java, just keep in mind that classes that implement an interface can be thought of as a derived class of said interface :


public interface BrowserDriver {
    public void navigate( String URL );
    public void type ( String textBoxXpath, String textToType );
    public void click ( String buttonXpath );
    public void waitForPageLoad();
    public boolean searchForLink();
}


public class FirefoxBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Firefox browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Firefox
	}

	// etc
	
}

public class ChromeBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Chrome browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Chrome
	}

	// etc
	
}

public class GoogleSearchFaceBookTest {


	public static boolean testGoogleSearchFacebook( BrowserDriver driver ) {
		driver.navigate( "www.google.com" );
		driver.type( "//searchbox", "facebook" );
		driver.click( "//searchbutton" );
		driver.waitForPageLoad();
		return driver.searchForLink("www.facebook.com");

	}

}


public class TestMain {

	public static void main( String [] args ) {
		BrowserDriver browserDrivers[] = { new ChromeBrowserDriver(), new FirefoxBrowserDriver() };
		for( browserDriver : browserDrivers ) {
			if(!GoogleSearchFaceBookTest.testGoogleSearchFacebook( browserDriver )) {
				System.out.println( "GoogleSearchFaceBookTest has failed!" );
			}
		}

	}

}

If you have a very large test suites, there are a couple of advantages to this approach :

  1. This code is a lot less copy paste and hence less error prone.
  2. Code is easily extensible : Want to add a new test ? Just add it to the main for loop. New test runs with all supported browsers. Want to add a new browser to the test suite ? Just add the driver object to the browserDrivers array. Your entire test suite now runs against the new browser.

How does this magic happen ? If you look at these lines of code :

driver.navigate( "www.google.com" );
driver.type( "//searchbox", "facebook" );
driver.click( "//searchbutton" );
driver.waitForPageLoad();
return driver.searchForLink();

you can see that the code behaves differently ( polymorphism ) based on the type of the driver object. If the driver object is of type ChromeBrowserDriver, calls get dispatched to ChromeBrowserDriver methods. If driver is of type FirefoxBrowserDriver, calls get dispatched to FirefoxBrowserDriver methods. This is the power of polymorphism and it allows us to write modular code.

One important question that could come up at this point is : ‘My favorite language is C. Do I need to switch to Java to write object oriented code ?’. And my answer is : No, it is not the language that is important, but the technique. You can find object oriented code in C. Look at the write() system call in Linux :

ssize_t write(int fd, const void *buf, size_t count);

This is an object method operating on the file object pointed to by the “fd” parameter.

Since C is a simpler language compared to Java, if modularity problems are not common in the problems that you solve, use C. I believe we must use the simplest tool that gets the job done. This reduces chances for bugs and creates more performant software.

On the other hand, if modularity problems are common in the problem you are attempting to solve, it maybe helpful to use a language with support for object orientation. For example, if you say that ‘class A implements interface B’ and while implementing class A, if you forgot to implement a method specified in interface B, you get a nice compile time error in Java. In fact, if you use a good IDE, it will report the problem immediately, before you even decide to compile the program.

Honesty

How do we act ?

There are two categories in which we act :

Emotional

We are so overwhelmed by our emotions that we do stupid shit that sometimes seriously jeopardizes us.

I am not going to talk about this category of actions in this post.

Rational

  1. We have hopes, dreams and desires.
  2. We gather information from our surroundings.
  3. Based on this information we act in a way that we hope fulfills our hopes, dreams and desires.

This is why we get mad when someone lies to us, because they fucked with our step 2.

So we should never feed bad information to anyone, i.e, people should be allowed to form an accurate perception of their surroundings. Anything, including telling part of the truth, hiding the truth, saturating their mind with only one type of information so they think there is no other type ( for example, at a job fair, only talking about the perks of working at a company and not talking about the bad aspects ) etc is evil and should be avoided.

Done. Mystery Solved. Amirite ? Not quite.

So I was reading this entertaining story of Ms Karen Owen, who slept with half the men in Duke University and then proceeded to make a thesis paper out of it, which went viral.

What followed was what anyone would imagine : death/rape threats, lawsuits, harassment etc. I noted a comment made by some woman online. She said that the problem was that Ms Owen labelled the thesis as a ‘Fuck-List’. If she had labelled it as ‘The Best 15 men I ever had’ there would not have been that much of a blow back. I remember thinking to myself : ‘Well, that is not lying, but I still feel somewhat uneasy about it. I don’t like the idea that someone can control your actions by presenting the facts in a particular way, but I also do not feel that Ms Owen would be at fault if she had done that’.

Hannibal Lecter
Hannibal was a master of manipulation

So I started asking myself why I thought that. Now, it might be a good time point out that I am fairly liberal when it comes to this. I currently feel that women should be allowed to use their bodies in a way that she sees fit so long as it does not hurt others. I realized that there is only one way to respond to her actions : ‘Congratulate her on managing to have a good time at school’. On the other hand it was not OK to send her death/rape threats and force her to change her name or whatever. So in my mind since there was only one course of acceptable action, I was OK with someone presenting the data in a way that brought about that outcome.

Just as I cannot support Ted Bunty’s autonomy to have sex with fresh dead bodies of women he killed, I cannot support a mob’s autonomy to ruin someone’s life.

Let me try to explain this with another example.

Imagine, you are walking through the park, and you find a young boy running towards you screaming. There is a fork in the road and he takes the left fork.
Soon after, here comes a known child molester who is running after the boy. He sees the fork in the road and asks you : ‘Which way did he go ?’. The molester is already leaning towards the left fork and if you say nothing he will run after the boy and possibly rape him.

Should you be honest ? I think you should be dishonest and send him the wrong way to spare the child.

What we see here is that you do not have to respect the autonomy of others to make this world a worse place. Therefore ethics of manipulation can be defined as :

Respect the rights of others to ‘absorb correct information from their surroundings and act to fulfill their hopes and dreams in a way that cannot hurt others’.

Benevolent manipulation

Some people claim that there is a benevolent form of manipulation : motivating. I think it is alright, if executed carefully. I have taken motivation from the words of others and it has benefited me. The problem here is that you can easily misjudge what they want from life and end up sending them in a path that could cause them a lot of misery. Be careful of that.

Another exception to this ‘Respect the Autonomy of People‘ rule is when you find that someone is incapable of taking care of their own interests. For example, children, drug addicts, mental patients etc.

Hopefully with these rules in mind we can live an honest life.