Why human civilizations are always imperfect

As I was travelling I started to notice something that must have been obvious: there are more people living in some places (urban regions) and fewer people living in other(rural regions)? Why is that? I also noticed that there are virtually no societies without a government. If human beings value being free so much, why do they install governments for themselves? I always like to study things from first principles and I have been doing that for some time. So, I thought I would share with you my understanding of the process of evolution of human civilizations.

Imagine for a second that you are living in the fertile crescent during the time of the start of recorded history. Society has recently discovered farming. Instead of moving from one place to another in search of food and other necessities, you can simply plant some seeds in the soil, wait for a bit, and have food waiting for the taking in a short period of time. This ability to know where food will be available a year from now is useful, it helps you become more efficient. You no longer have to wander aimlessly in search of food hoping that you will find it before hunger gets the better of you. Hooray! Predictability! This means that you can settle down and have a nice quiet life.

All men want to eat and make merry. You realize that to do that, you have to do 4 simple steps:

  • Step 1: Plant your crops (work).
  • Step 2: Tend to the crops until it yields food (work).
  • Step 3: Harvest (work).
  • Step 4: Eat and make merry.

Lather, rinse, repeat!

But there is one problem: there is another way someone can ‘eat and make merry’ without having to go through steps 1, 2 and 3 which is a lot of work. That method goes something like this:

  • Step 1: Wait for someone else to do the work of planting crops, caring for and harvesting them.
  • Step 2: Use violence on them and take their stuff (work).

Step 2 can involve killing them, but if you kill them, they will not be around to plant crops for you in the future. So, prefer to not kill them, but in case they resist, kill a few of them in a public and brutal way to set an example. You are king and everyone works for your pleasure.

Let’s say that people who produce the crops are called Producers (P) and people who seize other people’s stuff for themselves are called the Warriors (W).

As long as the work of taking other people’s stuff is greater than the work of producing the stuff, warriors will take from producers.

If we denote the amount of effort required for Producers to be Work(P) and the amount of work required for warriors to be Work(W),

Work(W) < Work(P) => Seizure

Now, let’s say that workers notice that despite doing the hard labour of planting the crops, caring for them, and harvesting them, they do not get to ‘eat and make merry’. They realize that there is nothing that stops the warriors from taking their stuff. So, they decide to install a warrior clan for themselves and when another warrior clan tries to kill them and take their stuff, they turn around and kill them first. Therefore, to ‘eat and make merry’, there are now 5 steps:

  • Step 1: Plant your crops (work).
  • Step 2: Tend to the crops until it yields food (work).
  • Step 3: Harvest (work).
  • Step 4: Defend your harvest from others who may want to take their stuff (work).
  • Step 5: Eat and make merry.

This means that each sedentary society develops a production clan and a warrior clan. This is the evolution of a state. The state has some territory (land) where it conducts economic activity such as production and some people which can be broadly divided into a production class and warrior class.

Within a state, there is law and administration of justice. The purpose of the law is to incentivize people to produce more and try to be more efficient in production. To do this, the state declares that only you have rights to the fruits of your own labour. If you work in a field to produce a harvest, only you are entitled to the harvest produce. If someone steals the fruits of your labour, they are punished (administration of justice). It should cost more to steal from others than to produce goods and services on their own. Thus:

Work(W) > Work(P) => No Seizure

This means that it becomes more profitable to become a producer and not a warrior within a state. This leads to more people producing more goods and services leading to prosperity within the state.

Other people notice the prosperity of such a state and install similar governments for themselves to ensure their own prosperity. Thus, similar states develop in other territories forming an international system. Within the international system, there is still the rule of the mightiest. Whenever it is profitable for a state to steal the wealth of its neighbours through violence, it does so. This process of violence between states is called war. At the end of a war, the victorious state can loot and plunder the vanquished state and over time this system stabilizes for a brief period of time when each major state is evenly matched.

State power and corruption

Governments are installed to administer justice. To administer justice, law enforcement officers need to be hired, rehabilitation centres need to be built, court systems should be established, and criminals should be caught, tried and punished. All of this costs money and these costs are borne through the collection of taxes. Here, there is another problem. What if law enforcement officers cooperate with the plunderers to steal from producers? If law enforcement officers can make more money from co-operating with thieves than they can make from enforcing the law, they will co-operate with thieves and make more money that way. This problem is called corruption.

Let’s denote the benefit of co-operating with people who steal from producers to be Benefit(Co-operating with thieves) and the benefit of enforcing the law to be Benefit(Enforcing the Law). The benefit of enforcing the law is usually the salary and benefits awarded to law enforcement officers.

Benefit(Enforcing the law) < Benefit(stealing) => Corruption.

It will almost always be more profitable to steal than it is to enforce the law. Therefore, it is important to have virtue as a requirement before people are installed as law enforcement officers. Otherwise, there will be no administration of justice and this will cause the state to become inefficient. It will eventually, get overrun by a different state where administration of justice is possible, leading to prosperity for them and more money for them to pay for soldiers and thus a better and stronger army. The stronger army should then make war on the weaker army thus defeating it and installing its system of justice on the vanquished people thus ending the corruption.

Development of cities and trade

As people become more secure in their environments, they aspire to become wealthy through specialization and trade. People who are better at certain arts and crafts specialize in their areas of expertise and then trade the products of their work with others.

Different people may have different costs for producing goods and services. People with the lowest cost can produce the goods and services and then trade them for goods and services with others who have lower costs of production for those goods and services. This makes everyone better off as goods and services are produced at the lowest cost possible.

However, it might be very inefficient for people to travel to everyone else’s homes for trading. Therefore, usually, a common area is designated as a marketplace where anyone who has anything to buy and sell is required to show up for trading. This area is usually called a town. Each town will usually have the products and services that are required to support the population in its surrounding area (also called a catchment area). More specialized services (such as brain surgery etc) might require a larger catchment area as the service require more people to support running costs. As a result, some of these towns develop into cities and offer more specialized services.

Learn more

Here are some of the links which were informative in this enquiry:

Inevitability of imperfection

So, human civilizations seem to have evolved from an organic process that involves much competition and a search for efficiency. However, this process, like all organic processes is inefficient. Therefore, every human society is inefficient (with some injustice that cannot be eliminated) and the international system is always unstable as the capabilities of every country change through time thus disrupting the equilibrium. We just have to learn to live with all of it. However, I do hope that the system never delivers an unacceptable outcome, such as a nuclear holocaust or human extinction.

Nutanix and Hyper converged Infrastructure

Now that I am no longer working for Nutanix, I feel like I can talk about what I learned about Hyper converged infrastructure and cloud computing in the last 2 years. Let’s look at how hyper converged architecture is formed and then we will discuss the advantages of structuring your IT infrastructure this way. Most of what is mentioned here is a simplification of Nutanix Bible.

What is Hyper converged infrastructure ?

For a system to be hyper converged, it needs to have at least 4 components :

  1. Hardware
  2. Hypervisor
  3. Distributed storage software
  4. Management plane that can manage all of the above

So, let’s look at it from Nutanix’s point of view.

Hardware

All software needs hardware to run on and for hyper converged platforms, this is not different. You can buy hardware from Nutanix, Dell-EMC, Lenovo etc. Nutanix hardware is usually standard x86 servers from Supermicro ( although that may have changed ) with Nutanix branding. This hardware is not special, which is why you can swap it out with Dell-EMC, Lenovo gear.

Hypervisor

Hypervisors are usually purpose built operating systems that can create virtual machines ( VMs ). So, on the hardware, first you need to install a hypervisor. Here again, you have choice. You can install Nutanix’s own AHV , or you can install VMWare ESX or Microsoft Hyper-V.

Distributed Storage

Using the hypervisor above you can create a VM that has direct access to all the disks on the computer. This VM ( called a Controller Virtual Machine or a CVM ) talks to other CVMs in the data center, forming a cluster. On this VM, a set of software is installed which aggregates all the disks on the computer ( and the disks on all the other computers in the cluster ). This is then presented back to the hypervisor as a single unit. This component is the absolute minimum that must come from Nutanix if you want to use Nutanix platform.

Management plane

Some management software is installed on the CVM which can be accessed by going to https://CVM_IP_ADDRESS:9440 using a web browser. For Nutanix, this management plane is called PRISM. Besides serving the user interface, it also serves a RESTful API which can be used to automate tasks on the cluster. PRISM is a must for managing storage in Nutanix cluster, but you can still manage your VMs from VMWare vCenter for your ESX cluster.

Advantages of Hyper converged infrastructure

So why go through this at all ? This seems much more complicated than just running your applications directly on hardware. It is even more complicated than running applications on a virtual machine on the hardware. ‘Distributing storage ? Presenting the storage back to the hypervisor ? You are out of your mind, sir !’ I can hear you saying. Here are some reasons :

  1. Self healing : Hyper converged systems offer automatic data replication. When VMs write data to disk, the platform writes a copy elsewhere. So if your hard disk fails, just rip and replace it when you have time. The system notices that there is now only one copy of all the data that was in that hard disk and automatically builds a second copy. Same applies when a single computer in the cluster fails.
  2. Expandability : You can start with just a few computers and then grow from there as your computing needs grow. You don’t have worry about to buying a large number of computers upfront and then not needing it.
  3. Easier to manage : Hyper converged systems offer a single interface to manage all your IT infrastructure. Manage storage, compute etc from the same web based console.
  4. Efficiency : The platform can notice when duplicate data is being written to disk by virtual machines. For example, your VMs serving live traffic may be cloned by developers to debug issues or to develop new features. Most of the data on these VMs may be similar and can be deduplicated saving storage space.
  5. Security : Data can be encrypted automatically by the platform before they are written to disk.

There are many more advantages of course, but that should justify the extra complexity.

10 reasons why commercial companies should do open source software

  1. Positive PR
  2. External targeted contributions. For example, most people submitting patches to open source software are facing some issue as a user. If they are experiencing an issue, others may also be experiencing similar problems. This is a much better way of driving development, compared to doing it post expensive market research.
  3. Marketing
  4. Recruiting vehicle
  5. Increasing the size of markets. For example Google is trying to bring more people online with Android to increase the size of search market.
  6. Commoditizing complements : http://www.joelonsoftware.com/articles/StrategyLetterV.html
  7. Reducing the cost of training new employees
  8. Influencing the direction of an industry. For example setting standards for server hardware with Open Compute Project.
  9. Free testers
  10. Improving employee morale

Agile software development process explained in simple terms

Agile is a combination of two processes that are designed to develop software. The two methodologies that form Agile are : Scrum and Kanban.
Key to understanding agile is understanding the objective tree. Here is an example objective tree for a company like amazon.com which aims to build an e-commerce website.

organization-objective-tree
Organization Objective Tree

There are many people who argue about the number of levels in the tree and how they should be called. But the key insight here is that :
a) Objectives form a tree
b) When child objectives are done, the parent objective is automatically done

There are two methodologies within Agile : scrum and kanban.

Scrum

Process

Objectives are broken into user stories. A prioritized list of user stories is called a backlog. Product management owns the backlog.

Objectives are achieved by working on them in 2 weeks sprints. Team pulls the maximum number of stories that can be completed into the sprint backlog and starts working on them when the sprint starts. Sprint backlog is owned by the team.

Every day of the sprint there is a 15 minute standup meeting where team members describe 3 things :

  1. What they worked on yesterday
  2. What they are planning to work on today
  3. Blockers ( if any )

This is an opportunity for micro-course-correction. If a story was done, team members can move to help with another story. If there is something that blocks a team member, management can move to unblock them. If a story might spillover, it can be called out.

At the end of the sprint there is a sprint retrospective to re-evaluate and course correct at a higher level.

Every N sprints ( for a finite value of N ) there is 1 refactor sprint. No new features are delivered in this sprint. Architectural runway is prepared for incoming stories and code is made more maintainable.

Why is the process like this ?

Reason for product management owning the backlog is to communicate clearly what the current priority is. This can be challenging in a changing environment. For example, management may ask the team to support an application on Windows 2012 platform and mark it high priority since largest customer base is on that platform. Afterwards it could turn out that highest paying customers are on RHEL 7.1. Management can communicate this information to engineering by adding a new story to support RHEL 7.1 and deprioritizing the story for Windows 2012 support.

Team owns the backlog because the team is responsible to ensure that the features ( stories ) committed at the beginning of sprint is done by end of sprint. This is why it’s the team that need to pull the stories into the sprint ( and not the manager ) so they feel a sense of ownership.

Standups give an idea about the pulse of the sprint : whether the team is on track, if it needs adjustment etc. Since software development is a collaborative process, it can get blocked on other people. For example : UI team can get blocked on UX mockups, developers can get blocked on APIs that were promised by other team members, one team member may be blocked for code review from another etc. Daily meetings help with dealing with these problems as they arise. It also enables the team to help others, for example, if someone airs a database problem, someone else on the team may have encountered and solved the problem before. Standup meetings should be short and sweet. If they are too long, since they occur everyday, they become a productivity drag.

The sprint retros are an opportunity to improve things at a higher level. For example, developers may notice that the CI system could run a larger portion of automated tests, reducing QA burden. Developers may point out that the code is getting too complex in one area and request refactor stories.

Expectations

1. If a story is in the backlog, it may get done. If a story is not in the backlog, it will never get done.

Why is this expectation there ? Sometimes product management may request a feature. Engineering does not know if that feature was important, whether it was a priority etc. Moreover, there could be multiple managers requesting different priorities for stories fuelling confusion. This expectation sets up management to close out the relative priority of objectives before people set out to achieve them.

There is no such thing as an implicit requirement. Management owns the product backlog.

2. Target spillover for any sprint is zero

Why is this expectation there ? For hard software engineering problems it is required that the team is a highly committed one. An uncommitted team member will usually bring down the whole team since the job is collaborative in nature. Target spillover of zero ensures that the team remains focussed on the problem at hand.

3. No activities are planned mid-sprint

Why is this expectation there ? This is to help the team remain focussed so that the cost of context switching is lowered. It is called a “Sprint”, not a “Trot” or “Gentle Walk”. Management is expected to defend the team from external influences. Course corrections are possible at the end of sprint. Therefore, it is important to keep the sprints as short as possible to get a meaningful amount of work done. The length of your sprint should be the minimum amount of time it takes you to respond to external influences. Usually, it is 2 weeks.

4. QA should raise a ticket if there is a problem.

Why is this expectation there ? It is possible that QA notices a problem and communicates it verbally or through email. Now people may go on vacation etc cutting the team’s ability to deliver.

There is no such thing as an implicit ticket. Raise it in issue management software.

5. No story is done until :
a) All code is reviewed
b) There are no open tickets
c) Documentation is complete

Why is this expectation there ? Code reviews are important because it lets someone else get familiar with that code base allowing them to take over in the event of the author’s absence. It is important to close all open tickets before calling a story done since otherwise your development process simply turns feature requests to bugs. Documentation should be developed lock step with code since that is when it is easiest to write it.

6. Do not test from feature branches

Why is this expectation there ? Someone could merge something else into the development branch making the feature tested in feature branches to be unviable in development branch. Development branch is what ultimately gets shipped to users so that is where things should be tested.

7. Stories are of the form “As a [user] I want to do [task] so that I can [accomplish goal]”. If they are not in that form, it can be safely deleted.

Why is this expectation there ? Often stories may be created with no clarity on what the next step is. Often developers may misunderstand the requirement and deliver a different feature requiring multiple unnecessary round trips and feedback loops to complete the feature. If the story is not in the prescribed form, it can be deleted and considered non-existent since it motivates people to state the objectives of the story explicitly.

A few points

  1. In Scrum the keywords are focus and course correction.
  2. Manager – employee relationship is co-operative. The team does everything to make the manager look good ( by delivering high quality features that delights users ) and managers does everything to setup the team and employees for success : removing impediments, giving them tools and access they need to meet the objectives.
  3. Dev – QA relationship is quasi-adversarial. QA tries to defend the customer demanding issues be fixed within reason. Dev team appreciates QA for finding the bugs they have missed that could have caused them to deliver an inferior product.
  4. Agile requires self motivated team. An under-motivated team may over inflate estimates or under commit into a sprint.
  5. There is no such thing as half-agile. Agile gives management a certain amount of power and engineers certain amount of power. This is to improve efficiency by reducing friction and miscommunication. If management says : ‘I will accept these parts of the process that gives me power and ignore these parts that require me to relinquish power’ this process will fail. It requires humility on the part of management and engineering.

Kanban

This is usually for support. After product is developed, support is trained on resolving bugs that get into production. Support can’t wait 2 weeks to resolve issues. When tickets come into the queue people take them on and try to get through them as fast as possible.

Scrum is more proactive and Kanban is more reactive. Kanban is for situations where a 2 week turnaround time is unacceptable.

Waterfall

Agile came about as a response to the Waterfall model of software development. It helps solve the most important problems with waterfall such as quicker course correction, improving focus and reducing miscommunication.

In waterfall, the software developed in following phases ( one after other ) :

  1. Gathering requirements
  2. Designing solution
  3. Implementing solution
  4. Testing solution

Problems:

  1. A mistake in early phase cannot be detected until last phase.
  2. There is little focus as work drags on for years.
  3. People working on each step only communicates with people in previous step increasing communication errors.

Should Indian Railways be privatized ?

Should Indian Railways be privatized ? Looking up the answer to this question on the Internet would lead one to conclude that no, it should not be privatized. I am not convinced, I think it should be privatized.

Lets look at the arguments against privatization, one by one.

  • It is not clear how it will improve situation

The argument is that privatization and increased competition will improve the quality of service, drive down prices and spur innovation. It should reduce corruption and force employees to do their best. For a government owned company the employees have no reason to work hard, treat its customers well and figure out how to make everything better. They get paid regardless. This causes them to mistreat their customers and generally be lazy about their work.

The Indian Railways lost ₹30,000 crore (US$5 billion) in 2015. This is despite the fact that it is a high demand business with a government sanctioned monopoly. Booking a train ticket is so painfully difficult that it a subject of so many degrading memes on the Internet. The compartments are noisy and sleeper classes are filled with bed bugs. The toilets are dirty. The staff is unpleasant to deal with. The train stations are riotous and unclean. It smells of shit and urine.

I wonder how many levels of incompetence it takes to suck so badly at a job.

Privatization has helped the Indian telecommunication sector and aviation sector. It is possible for it to work in the rail industry as well. Even if it doesn’t help, it is at least worth a try.

  • Look at the state of American and British Railways

They are significantly better, aren’t they ? But even if it isn’t, there are a thousand reasons it could work better in India. Yes, I know there are a thousand reasons why it couldn’t work also. But why not give it a try ?

Concerning America, their landmass is 3 times that of India and they are a very affluent and urbanized country. This means that most of the travel is very long and hence very inconvenient to be conducted by train. So rail is used mostly for goods and human beings pay a little more to travel by planes.

  • It is a natural monopoly due to high cost of entry and exit

This is partially true. The high cost of entry and exit is largely due to high cost of building and operating railways, so we may need to have some government assistance with regard to this.

One solution to this problem is to let the rails themselves be owned by one company and combine this with multiple companies owning the individual trains. The train companies lease the rails from the rail company to provide their service.

Sure, some government assistance will be needed to acquire more land, ensure that there is stiff competition etc but these are all problems that have well known solutions. It can definitely not be a fire and forget endeavor.

Also I don’t understand how it becomes harder for individual investors to exit the rail business as most companies are traded on the public stock market. If you want to enter the rail business buy a few shares of the company at the Mumbai Stock Exchange. Want to exit ? Sell them at the stock exchange.

  • Some routes may be under serviced because they may be unprofitable

Form a government company to service only these routes using tax payer money. There is no need to keep the entire business in the public sector and cause incalculable misery to everyone in India to benefit a few.

  • The railway employs a lot of people these jobs may be lost in privatization

Privatization does not automatically cause loss of jobs. If a private railway business can operate with fewer number of people and automate away a lot of these jobs it would benefit the customer with lower fares. This money saved by customers can be used to buy new things which creates new jobs. See the parable of broken windows. Another possibility is to tax the rail companies and pay unemployment benefits to these people directly. At least the public will be spared from their laziness and callousness.

  • Defense concern. Countries like China could buy up all Indian Rail businesses and in the event of a war, discontinue them all

We can repossess these businesses in the event of a war. The other problem is if they sabotage it silently while they are holding it. This problem can be solved by stronger policing.

  • National pride and unity

A lot of arguments center around how Indian Railways is a matter of pride for Indians. Sorry, I would rather travel comfortably than with pride. Also, I don’t think it is a matter of pride given how absurdly mismanaged it is.

So anyway, I think most of these arguments are bogus or ill-informed. I have similar thoughts about the power industry ( power cuts are uncommon in Western countries ) since there is no innovation in solar, wind, nuclear power in India despite us having no serious oil reserves. I think India is a poor country on account of power. But more on that later.

What do you think ? Let me know in the comments section.

“Man of Steel” movie explained

So last night we watched “Batman v Superman”. This got me thinking about the prequel : “Man of Steel”. I just rewatched it and found that it is in fact a very good movie. Let me tell you why.

Superman has always been one of my favorite characters. This is not to say that I have watched and enjoyed every version of Superman in existence. In fact, I may not even like some incarnations of this character. But the fact remains that the essence of the Superman story is very interesting. So I am always surprised to find that many people dislike Superman deeply.

One reason I have heard for this strange phenomenon is that they think Superman is too powerful. He seems invincible. It is true that a story would be uninteresting if the hero had no chance of losing or if he was pitted against a weak villain ( “Merlin” TV series anyone ? ). This is not the case for “Man of Steel”. Zod is every bit as powerful as Superman. It is in fact a vital part of the story that Superman is outgunned and outnumbered in this fight. He has to fight many Kryptonians who are just as strong as him but born and bred to be soldiers. These Kryptonians trained their entire life to be warriors and you can see Superman struggling to match them. The truly awesome power of Superman is the hook for the story. But this does not make it dull if he spends the majority of his time standing up to guys a lot stronger than himself.

A lot of people are angry that at the end of the movie Superman kills Zod. I think it is an important plot point : this implies that Superman was never invincible to Zod, Kryptonians can kill other Kryptonians on Earth with their bare hands. The fight could very much have ended with the death of Superman, it is unclear how he survived at all. Maybe Zod was still distraught over losing his purpose and his people. Maybe his heart wasn’t fully in the fight. Maybe it was just plain luck.

I think it is very brave that a simple farm boy is willing to take up such dreadful odds to do the right thing. He didn’t sign up for any of this.

There is also another aspect to this story. When Zod asks Superman to join him, it would have been the easiest thing in the world to say ‘Yes’. The movie depicts throughout, the loneliness that Clark feels. He feels like an outsider, a freak. It would have been a welcome relief to live on the ‘New Krypton’ and feel belonged. So why does he say ‘No’ ?

The simplest answer would be to say that he didn’t want to kill all those humans. But he did finish off the Kryptonian race ( his own ) in the process. Is it any better ? This is where the back story kicks in again. Every Kryptonian’s life is predestined, every man and woman born to a single predefined purpose. There is no place for chance or choice. With such a system, the ‘New Krypton’ would inevitably meet the original Krypton’s fate : infinite self-destruction. This is why Jor-El does not leave Krypton with Kal-El. When a race is doomed, it can only be wiped out so that a new one can have a hope of survival. This point, however cruel is a fact in the story universe.

Human beings may not make the same mistakes Kryptonians made, which means they have a shot at survival and expansion, not the infinite destruction that Kryptonians face. This is also why Jor-El was OK with sending Kal-El away from Krypton : he was a natural birth. He is not part of the system. Jor-El may have tried to bargain with Zod to let the Kryptonians live alongside humans in a last ditch attempt, but he knew that only humans have a serious chance.

Finally, I also want to make the point that it may not be Superman a lot of people hate. This point has been argued before, but I think we find it difficult to accept even a fictional character who is both good and successful. The reason for this is that he reminds us of our own shortcomings. We feel that we are justified in doing evil, so that we can be successful or happy. We feel it is impossible to live in this unjust and unfair world by being good. It is this lie that we tell ourselves that is challenged when we watch a Superman movie. We are not angry at Superman, we are angry at ourselves. Sometimes we project this anger towards all that is good. If that is true, we have to do better than that, don’t you think ?

How to write good software

Here is my theory on an algorithm to build great software :

  1. Figure out how the software ought to work.

    This is the most important part in the software design process. It is important that the user experience be nailed down before we begin all other processes. If we don’t do this first, people will find the easiest way to make the software work. Such an approach may not deliver the best quality experience to the user. It is important to note that it is essential, to involve the users in this step. We need to clearly understand the problems they are facing. How should the optimal solution to their problem work ? Is it even possible with the available resources and at the current state of the art ?

  2. Make it work

    This is the part where you build the software according to the expectations set by the user. At this point you do not have to worry about code quality, performance etc.

  3. Make it better

    In this step you have to make sure that the code is extendable, has high quality tests and is secure

  4. Make it pretty

    In this step, make the software beautiful. Make the text conversational and friendly. Introduce fancy graphics and other visual elements if possible.

  5. Make it fast

    Finally it is time to seek out the worst performing parts of the software and make it fast.

If you follow these steps, and in that order, you should have a high quality piece of software in your hands.

What is polymorphism ?

Polymorphism is touted as one of the awesomest features of Object Oriented Programming. But what exactly is the point of polymorphism ?

Lets take the problem of trying to automate Google search. Lets say your test case is :

  1. Open the browser
  2. Navigate to “www.google.com”
  3. Type “facebook” to the search text box
  4. Click on “Google Search” button
  5. Wait for the page to load
  6. Check if the results page contain a link to “www.facebook.com”

Clearly we need to find some way to drive the browser to do these actions. But how will the automation software know where the ‘search text box’ is ? How will it know where the ‘Google Search button’ is ? So, there is a technology called Xpath that lets you identify a particular element in a web page.

So here is the procedural code that solves the problem ( Language : C ):

int search_google_facebook()
{
    int test_passed = 1;
    char *search_box_xpath = "//searchbox";
    char *search_button_xpath = "//searchbutton";

    firefox_navigate( "www.google.com" );
    firefox_type( search_box_xpath, "facebook" );
    firefox_click( search_button_xpath );
    firefox_wait_page_load();
    test_passed &&= firefox_search_page_link( "www.facebook.com" ):

    chrome_navigate( "www.google.com" );
    chrome_type( search_box_xpath, "facebook" );
    chrome_click( search_button_xpath );
    chrome_wait_page_load();
    test_passed &&= chrome_search_page_link( "www.facebook.com" ):

    return test_passed;
}

Now lets look at the object oriented solution to the problem. Please keep in mind the Liskov Subsitution Principle : base class pointers can point to derived class objects. The code is in Java, so if you are unfamiliar with Java, just keep in mind that classes that implement an interface can be thought of as a derived class of said interface :


public interface BrowserDriver {
    public void navigate( String URL );
    public void type ( String textBoxXpath, String textToType );
    public void click ( String buttonXpath );
    public void waitForPageLoad();
    public boolean searchForLink();
}


public class FirefoxBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Firefox browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Firefox
	}

	// etc
	
}

public class ChromeBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Chrome browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Chrome
	}

	// etc
	
}

public class GoogleSearchFaceBookTest {


	public static boolean testGoogleSearchFacebook( BrowserDriver driver ) {
		driver.navigate( "www.google.com" );
		driver.type( "//searchbox", "facebook" );
		driver.click( "//searchbutton" );
		driver.waitForPageLoad();
		return driver.searchForLink("www.facebook.com");

	}

}


public class TestMain {

	public static void main( String [] args ) {
		BrowserDriver browserDrivers[] = { new ChromeBrowserDriver(), new FirefoxBrowserDriver() };
		for( browserDriver : browserDrivers ) {
			if(!GoogleSearchFaceBookTest.testGoogleSearchFacebook( browserDriver )) {
				System.out.println( "GoogleSearchFaceBookTest has failed!" );
			}
		}

	}

}

If you have a very large test suites, there are a couple of advantages to this approach :

  1. This code is a lot less copy paste and hence less error prone.
  2. Code is easily extensible : Want to add a new test ? Just add it to the main for loop. New test runs with all supported browsers. Want to add a new browser to the test suite ? Just add the driver object to the browserDrivers array. Your entire test suite now runs against the new browser.

How does this magic happen ? If you look at these lines of code :

driver.navigate( "www.google.com" );
driver.type( "//searchbox", "facebook" );
driver.click( "//searchbutton" );
driver.waitForPageLoad();
return driver.searchForLink();

you can see that the code behaves differently ( polymorphism ) based on the type of the driver object. If the driver object is of type ChromeBrowserDriver, calls get dispatched to ChromeBrowserDriver methods. If driver is of type FirefoxBrowserDriver, calls get dispatched to FirefoxBrowserDriver methods. This is the power of polymorphism and it allows us to write modular code.

One important question that could come up at this point is : ‘My favorite language is C. Do I need to switch to Java to write object oriented code ?’. And my answer is : No, it is not the language that is important, but the technique. You can find object oriented code in C. Look at the write() system call in Linux :

ssize_t write(int fd, const void *buf, size_t count);

This is an object method operating on the file object pointed to by the “fd” parameter.

Since C is a simpler language compared to Java, if modularity problems are not common in the problems that you solve, use C. I believe we must use the simplest tool that gets the job done. This reduces chances for bugs and creates more performant software.

On the other hand, if modularity problems are common in the problem you are attempting to solve, it maybe helpful to use a language with support for object orientation. For example, if you say that ‘class A implements interface B’ and while implementing class A, if you forgot to implement a method specified in interface B, you get a nice compile time error in Java. In fact, if you use a good IDE, it will report the problem immediately, before you even decide to compile the program.

Ten pieces of advice to managers of software companies; from a programmer

How would you want manage a software company ? Software is unlike many other industries in the sense that it is a creative process. In other industries ( say a Coca-Cola bottling plant ) there is a linear relationship between labor and result. If it takes 1 hour for a person to fill 100 Coca-Cola bottles, it will most probably take him 2 hours to fill the a total of 200 bottles and so on. If you want 200 Coca-Cola bottles filled in an hour, you should therefore employ 2 people. It is fairly predictable how much time it would take to fill 300 Coca-cola bottles with one person : 3. None of this translates well into software. In software, once a software is developed, it is trivial to create a copy of it. Software companies make money by making new things : adding features, fixing bugs etc ( I am excluding SaaS business models for the moment, though they also require new features to stay on top of competition ). One important feature of this type of creative work is that, by its very nature, it is unpredictable. If you know what code will solve your problem, and you know that it will work, typing in the code is the simplest thing in the world. Majority of software development involves figuring out how to solve the problem. And this process is venturing into the unknown, taking educated guesses, trying what works and on failures start fresh. What kind of management does this lend itself into ?

First off, it is pointless to ask people to put more time at work. When you try something and fails, it is important to take a step back and relax. The problem needs to be emptied from your mind for a while so that on the next approach you will not go down the same rabbit hole you went before. Software companies that require employees to put their bums on their seats as much as possible needs to go out of business, and fast.

Secondly, it is very hard ( and most times, impossible ) to estimate how much time it would take to do something. To give you an example, lets say I want you to make me some “Japanese Kuhoo Kuhoo”. I won’t tell you what it is, or how to make it, but I want you to tell me how much time it would take to make it. You don’t know what you would need to make it. You don’t even know if it is a real thing or something I made up entirely to mess with you. How will you know how much time it would take ? You can’t. You can’t plan for the unknown. If you absolutely want to know how much time it would take, please consult an Oracle. Because their estimates can have as much bearing on reality as anyone else’s.

Thirdly, you must never reward drama. Drama is always a sign of dysfunction, not capacity. Imagine two teams. Team 1 is a well run team. There are unit tests for almost all important functionality. Commits that break unit tests are disallowed automatically from nightly builds. Team 2 gives zero fucks to any sort of automated testing. Result is that days before release, testing reveals lots and lots of bugs. Developers are asked to stay back late to fix issues. Management sees that Team 2is toiling away, burning the midnight oil while people from Team 1 goes home on time. Management rewards Team 2 with bonuses. Until of course the house of cards built by Team 2 just crumbles under its own weight.

Fourthly, and this is not specific to software companies. Accountability should never get lost in layers and layers of management. For example, imagine people found critical security issues during testing just before release. Management is not supposed to say ‘We will focus on these trivial stylistic changes which are more visible to my manager and release the software with security problems’. In big companies it is very much possible for this to happen. Managers know that even if word gets out that the software was released with known security defects, the resulting inquiry would probably find nothing.

Fifthly, there should be penalty for risk-avoidance. For example, lets say that an employee comes to internal IT department with a request to install the Operating System on their Solid State Device. The IT department refuses to install it saying that it is a risk for them since they do not know how to configure backups properly. The Operating System being installed on the SSD can save developer time which is valuable to the company. But the problem here is that there is no downside for IT department to refuse the request. They have only upsides. Ideally, there should be a penalty for the internal IT in the company so that if the company is spending the money for an SSD, if IT is refusing to make use of it they should get penalized.

Sixthly, people should not be allowed to screw company for short term gain. For example, a new CEO can cut R&D spending for software companies if those projects would only lead to products in a few years. Cutting those projects would mean that the expenses would be reduced ( with no reduction in immediate profits ). Slump in profits comes a few years later when the company has no more new products coming out of the pipeline by which time the executive would have left the company. This problem can probably be solved by paying the executive even after him leaving the company; based on the performance of the team he used to manage.

Seventhly, responsibilities must be handed off properly. Lets say ‘Project A’ uses library built by ‘Project B’. When ‘Project B’ obsoletes a version of the library, ‘Project A’ must be given permission to take over maintenance of the library. Or the manager for ‘Project A’ should force the engineers to move away from ‘Project B’ library before it goes out of support.

Eighthly, you must not use metrics such as ‘number of cases solved’ as a metric for productivity. This is the Coca-Cola bottle mentality. If you do this, you will get the behavior that you incentivize. People will pick up easier cases since they can be solved faster. This is bad since when faced with a problem, it is the skill of your programmers that will decide how well and in how much time it gets solved. This skill is like any other muscle, it gets better the more you use it. It is probably a better idea to encourage your programmers to pick hard problems. ‘Number of cases solved’ in no way represents the amount of work done : the quality of the solution matters, the complexity of the problem matters.

Ninethly, you mustn’t assume that if you put more people into a project it will run into completion faster. Putting more people into a late project makes it later. See ‘Mythical Man Month’.

Tenthly, you must not waste your employee’s time. People who want to live meaningful lives will rue it when their work is wasted. They want to do a good job and when they see their work is duplicated, or wasted because of say, management indecision, they are less amused. This problem is compounded by the recent ‘Agile Movement’, which has taken to mean, the managers can change their mind whenever they want. Project direction, once set, should only be changed if market conditions change.

Wish

Glen and Sally were having coffee at the coffee shop.

“If you could have one wish what would it be ?” Sally asked.

“I would wish to have infinite wishes”, Glen said without hesitation. He had thought of this before.

“I mean apart from that” Sally said.

“I would wish to have another wish without the restriction to have infinite wishes and then wish to have infinite wishes, but I get your point” Glen said. “I would wish for perpetual perfect regeneration with the ability to end my life whenever I please” he continued.

“That is very particular. Why ?” Sally asked.

“Well, first I would have wanted to live forever, but then what would be the point if you age to say 90 and then live in that state forever ? Definitely you would want to stay young. But then, what if someone cuts off your arms and legs ? You wouldn’t want to live like that forever. See, immortality brings its own set of curses. You wouldn’t want to be forced to end your life because your are handicapped. You would want everything to return to normal. So the ideal way to phrase a good immortality wish is ‘perpetual perfect regeneration’. But what if someone caught you and catapulted you into space ? You would float through space forever having nothing to do. So you would probably want to end your life then. You would probably want to do that anyway at some point.”

“But what would you do with immortality ?” Sally had asked.

“Well, first off I would quit my job.” Glenn said with a laugh.

“Of course” Sally laughed “And then what would you do?”

“Well I would go out and learn everything about everything” Glen said.

“Everything ?” asked Sally.

“Yes, everything.” Glen replied “Physics, Chemistry, Philosophy, Psychology, Sociology, Voodoo magic, you name it.”

“And then what ?” Sally asked.

“Well, I have no idea how long that will take” Glen replied “Step two would be to obtain power and money. I bet being immortal would lend itself nicely to doing really dangerous jobs that has a potential for high pay off. But I am really hoping that there would be some connection in all the knowledge that is available in the world that nobody has made. A connection nobody can possibly make in one lifetime. A source of large amount of power.

After that I would have to try and take over. See, I would have lots of time. All I would need to do is wait for right opportunities. I bet the best places to take over first are the war torn areas in Africa where a power vacuum develops every few decades. The point is try to install people who are obliged to you to serve as your puppets.

Form a secret society, with a hierarchical tree like structure. At each level, you have a number of people overseeing the work of one or more people. Anyone power hungry can join, and take a pseudonym. Nobody knows each others name, how many people there are at each level or how many levels there are. You can kind of ask these people to help each other and act as a power broker. It is important to stay away from the light though”

“Why ?” asked Sally.

“Well, that kind of power would simply attract too much attention. I can definitely see people sacrificing their entire lives to make my life miserable if they can detect my involvement” Glen replied.

“OK” Sally was now starting to regret ever having asked the question.

“Well the next step is to optimize” Glen continued on, unaware of the uneasiness developing in Sally, “Obviously, I feel the worst thing in the world is war. When people are constantly fighting each other they have nothing to look forward to. First thing I would do is stop all the wars. When people feel safe, and they know what they build won’t be taken away from them using force, they will start building for the future. Next step is to educate. The benefits of Enlightenment should percolate to the whole world. Just imagine, billions of people in the developing and under-developed nations joining the quest for Scientific Inquiry. How much more the average human condition would improve! How the international productivity would go through the roof!”

“What then ?” asked Sally

“Well, the next step is to expand outwards, of course. Colonize the moon. Colonize Mars. It might be possible to sustain human life in harsh conditions by then. Expand outward. We have no idea how vast the Universe is. I could be doing this for a while”, Glen finished with a smile.

There was a silence. “Do you know what you are planning to do ? You are planning to become God!” said Sally.

“Not really, for all we know God’s powers could be infinite” said Glen. He was an atheist himself, but he did not want to offend Sally’s feelings.

“But do you think that what you asked for is right ?” asked Sally.

“Ah, I thought you wanted to hear about the most interesting wish I could come up with” said Glen taking a sip of his coffee. “I was not thinking about the morality aspect of it. I wouldn’t do this if the opportunity was real. If a genie appeared before me and granted me one wish, I would first ask him ‘Can you go back in time and fix stuff there ?’, if his answer is ‘Yes’, I would wish for ‘Go back to Big Bang and fix everything from there onward, if there was time before that fix that also, if there are other Universes fix everything there as well. FIX EVERYTHING'” he said waving his hands to express his enthusiasm. “If he can’t go back in time and fix stuff – I mean what a crappy genie, right ? – then I would ask him to fix everything instantly”.

“OK” Sally was smiling approvingly now.

“That is the only moral thing you can do, Sally. Everything else is immoral. Since we are just chatting, I just inferred the question as ‘What is the most interesting thing you would wish for?'” said Glen and then he smiled as well.

**************

Sally walked into the room where G.E.N.I.E ( Generic Electronic Nature Influencer Engine ) was housed. She wanted to take Glen’s advice on what to wish for before entering her wish. Sally had felt a huge burden of responsibility when her research was complete and she realized that they could build exactly one machine that could in fact grant someone one actual wish. The machine could not do wish recursion and therefore wishing for more wishes was futile, that was the first thing she tried. But now she had figured out the ideal wish. The only wish any moral person could make.

She pulled out the console for G.E.N.I.E and typed in her wish :

“FIX EVERYTHING”

The computer blinked, and then started computing.

After a while it shut down, and changed nothing.