Agile software development process explained in simple terms

Agile is a combination of two processes that are designed to develop software. The two methodologies that form Agile are : Scrum and Kanban.
Key to understanding agile is understanding the objective tree. Here is an example objective tree for a company like amazon.com which aims to build an e-commerce website.

organization-objective-tree
Organization Objective Tree

There are many people who argue about the number of levels in the tree and how they should be called. But the key insight here is that :
a) Objectives form a tree
b) When child objectives are done, the parent objective is automatically done

There are two methodologies within Agile : scrum and kanban.

Scrum

Process

Objectives are broken into user stories. A prioritized list of user stories is called a backlog. Product management owns the backlog.

Objectives are achieved by working on them in 2 weeks sprints. Team pulls the maximum number of stories that can be completed into the sprint backlog and starts working on them when the sprint starts. Sprint backlog is owned by the team.

Every day of the sprint there is a 15 minute standup meeting where team members describe 3 things :

  1. What they worked on yesterday
  2. What they are planning to work on today
  3. Blockers ( if any )

This is an opportunity for micro-course-correction. If a story was done, team members can move to help with another story. If there is something that blocks a team member, management can move to unblock them. If a story might spillover, it can be called out.

At the end of the sprint there is a sprint retrospective to re-evaluate and course correct at a higher level.

Every N sprints ( for a finite value of N ) there is 1 refactor sprint. No new features are delivered in this sprint. Architectural runway is prepared for incoming stories and code is made more maintainable.

Why is the process like this ?

Reason for product management owning the backlog is to communicate clearly what the current priority is. This can be challenging in a changing environment. For example, management may ask the team to support an application on Windows 2012 platform and mark it high priority since largest customer base is on that platform. Afterwards it could turn out that highest paying customers are on RHEL 7.1. Management can communicate this information to engineering by adding a new story to support RHEL 7.1 and deprioritizing the story for Windows 2012 support.

Team owns the backlog because the team is responsible to ensure that the features ( stories ) committed at the beginning of sprint is done by end of sprint. This is why it’s the team that need to pull the stories into the sprint ( and not the manager ) so they feel a sense of ownership.

Standups give an idea about the pulse of the sprint : whether the team is on track, if it needs adjustment etc. Since software development is a collaborative process, it can get blocked on other people. For example : UI team can get blocked on UX mockups, developers can get blocked on APIs that were promised by other team members, one team member may be blocked for code review from another etc. Daily meetings help with dealing with these problems as they arise. It also enables the team to help others, for example, if someone airs a database problem, someone else on the team may have encountered and solved the problem before. Standup meetings should be short and sweet. If they are too long, since they occur everyday, they become a productivity drag.

The sprint retros are an opportunity to improve things at a higher level. For example, developers may notice that the CI system could run a larger portion of automated tests, reducing QA burden. Developers may point out that the code is getting too complex in one area and request refactor stories.

Expectations

1. If a story is in the backlog, it may get done. If a story is not in the backlog, it will never get done.

Why is this expectation there ? Sometimes product management may request a feature. Engineering does not know if that feature was important, whether it was a priority etc. Moreover, there could be multiple managers requesting different priorities for stories fuelling confusion. This expectation sets up management to close out the relative priority of objectives before people set out to achieve them.

There is no such thing as an implicit requirement. Management owns the product backlog.

2. Target spillover for any sprint is zero

Why is this expectation there ? For hard software engineering problems it is required that the team is a highly committed one. An uncommitted team member will usually bring down the whole team since the job is collaborative in nature. Target spillover of zero ensures that the team remains focussed on the problem at hand.

3. No activities are planned mid-sprint

Why is this expectation there ? This is to help the team remain focussed so that the cost of context switching is lowered. It is called a “Sprint”, not a “Trot” or “Gentle Walk”. Management is expected to defend the team from external influences. Course corrections are possible at the end of sprint. Therefore, it is important to keep the sprints as short as possible to get a meaningful amount of work done. The length of your sprint should be the minimum amount of time it takes you to respond to external influences. Usually, it is 2 weeks.

4. QA should raise a ticket if there is a problem.

Why is this expectation there ? It is possible that QA notices a problem and communicates it verbally or through email. Now people may go on vacation etc cutting the team’s ability to deliver.

There is no such thing as an implicit ticket. Raise it in issue management software.

5. No story is done until :
a) All code is reviewed
b) There are no open tickets
c) Documentation is complete

Why is this expectation there ? Code reviews are important because it lets someone else get familiar with that code base allowing them to take over in the event of the author’s absence. It is important to close all open tickets before calling a story done since otherwise your development process simply turns feature requests to bugs. Documentation should be developed lock step with code since that is when it is easiest to write it.

6. Do not test from feature branches

Why is this expectation there ? Someone could merge something else into the development branch making the feature tested in feature branches to be unviable in development branch. Development branch is what ultimately gets shipped to users so that is where things should be tested.

7. Stories are of the form “As a [user] I want to do [task] so that I can [accomplish goal]”. If they are not in that form, it can be safely deleted.

Why is this expectation there ? Often stories may be created with no clarity on what the next step is. Often developers may misunderstand the requirement and deliver a different feature requiring multiple unnecessary round trips and feedback loops to complete the feature. If the story is not in the prescribed form, it can be deleted and considered non-existent since it motivates people to state the objectives of the story explicitly.

A few points

  1. In Scrum the keywords are focus and course correction.
  2. Manager – employee relationship is co-operative. The team does everything to make the manager look good ( by delivering high quality features that delights users ) and managers does everything to setup the team and employees for success : removing impediments, giving them tools and access they need to meet the objectives.
  3. Dev – QA relationship is quasi-adversarial. QA tries to defend the customer demanding issues be fixed within reason. Dev team appreciates QA for finding the bugs they have missed that could have caused them to deliver an inferior product.
  4. Agile requires self motivated team. An under-motivated team may over inflate estimates or under commit into a sprint.
  5. There is no such thing as half-agile. Agile gives management a certain amount of power and engineers certain amount of power. This is to improve efficiency by reducing friction and miscommunication. If management says : ‘I will accept these parts of the process that gives me power and ignore these parts that require me to relinquish power’ this process will fail. It requires humility on the part of management and engineering.

Kanban

This is usually for support. After product is developed, support is trained on resolving bugs that get into production. Support can’t wait 2 weeks to resolve issues. When tickets come into the queue people take them on and try to get through them as fast as possible.

Scrum is more proactive and Kanban is more reactive. Kanban is for situations where a 2 week turnaround time is unacceptable.

Waterfall

Agile came about as a response to the Waterfall model of software development. It helps solve the most important problems with waterfall such as quicker course correction, improving focus and reducing miscommunication.

In waterfall, the software developed in following phases ( one after other ) :

  1. Gathering requirements
  2. Designing solution
  3. Implementing solution
  4. Testing solution

Problems:

  1. A mistake in early phase cannot be detected until last phase.
  2. There is little focus as work drags on for years.
  3. People working on each step only communicates with people in previous step increasing communication errors.

Should Indian Railways be privatized ?

Should Indian Railways be privatized ? Looking up the answer to this question on the Internet would lead one to conclude that no, it should not be privatized. I am not convinced, I think it should be privatized.

Lets look at the arguments against privatization, one by one.

  • It is not clear how it will improve situation

The argument is that privatization and increased competition will improve the quality of service, drive down prices and spur innovation. It should reduce corruption and force employees to do their best. For a government owned company the employees have no reason to work hard, treat its customers well and figure out how to make everything better. They get paid regardless. This causes them to mistreat their customers and generally be lazy about their work.

The Indian Railways lost ₹30,000 crore (US$5 billion) in 2015. This is despite the fact that it is a high demand business with a government sanctioned monopoly. Booking a train ticket is so painfully difficult that it a subject of so many degrading memes on the Internet. The compartments are noisy and sleeper classes are filled with bed bugs. The toilets are dirty. The staff is unpleasant to deal with. The train stations are riotous and unclean. It smells of shit and urine.

I wonder how many levels of incompetence it takes to suck so badly at a job.

Privatization has helped the Indian telecommunication sector and aviation sector. It is possible for it to work in the rail industry as well. Even if it doesn’t help, it is at least worth a try.

  • Look at the state of American and British Railways

They are significantly better, aren’t they ? But even if it isn’t, there are a thousand reasons it could work better in India. Yes, I know there are a thousand reasons why it couldn’t work also. But why not give it a try ?

Concerning America, their landmass is 3 times that of India and they are a very affluent and urbanized country. This means that most of the travel is very long and hence very inconvenient to be conducted by train. So rail is used mostly for goods and human beings pay a little more to travel by planes.

  • It is a natural monopoly due to high cost of entry and exit

This is partially true. The high cost of entry and exit is largely due to high cost of building and operating railways, so we may need to have some government assistance with regard to this.

One solution to this problem is to let the rails themselves be owned by one company and combine this with multiple companies owning the individual trains. The train companies lease the rails from the rail company to provide their service.

Sure, some government assistance will be needed to acquire more land, ensure that there is stiff competition etc but these are all problems that have well known solutions. It can definitely not be a fire and forget endeavor.

Also I don’t understand how it becomes harder for individual investors to exit the rail business as most companies are traded on the public stock market. If you want to enter the rail business buy a few shares of the company at the Mumbai Stock Exchange. Want to exit ? Sell them at the stock exchange.

  • Some routes may be under serviced because they may be unprofitable

Form a government company to service only these routes using tax payer money. There is no need to keep the entire business in the public sector and cause incalculable misery to everyone in India to benefit a few.

  • The railway employs a lot of people these jobs may be lost in privatization

Privatization does not automatically cause loss of jobs. If a private railway business can operate with fewer number of people and automate away a lot of these jobs it would benefit the customer with lower fares. This money saved by customers can be used to buy new things which creates new jobs. See the parable of broken windows. Another possibility is to tax the rail companies and pay unemployment benefits to these people directly. At least the public will be spared from their laziness and callousness.

  • Defense concern. Countries like China could buy up all Indian Rail businesses and in the event of a war, discontinue them all

We can repossess these businesses in the event of a war. The other problem is if they sabotage it silently while they are holding it. This problem can be solved by stronger policing.

  • National pride and unity

A lot of arguments center around how Indian Railways is a matter of pride for Indians. Sorry, I would rather travel comfortably than with pride. Also, I don’t think it is a matter of pride given how absurdly mismanaged it is.

So anyway, I think most of these arguments are bogus or ill-informed. I have similar thoughts about the power industry ( power cuts are uncommon in Western countries ) since there is no innovation in solar, wind, nuclear power in India despite us having no serious oil reserves. I think India is a poor country on account of power. But more on that later.

What do you think ? Let me know in the comments section.

How to write good software

Here is my theory on an algorithm to build great software :

  1. Figure out how the software ought to work.

    This is the most important part in the software design process. It is important that the user experience be nailed down before we begin all other processes. If we don’t do this first, people will find the easiest way to make the software work. Such an approach may not deliver the best quality experience to the user. It is important to note that it is essential, to involve the users in this step. We need to clearly understand the problems they are facing. How should the optimal solution to their problem work ? Is it even possible with the available resources and at the current state of the art ?

  2. Make it work

    This is the part where you build the software according to the expectations set by the user. At this point you do not have to worry about code quality, performance etc.

  3. Make it better

    In this step you have to make sure that the code is extendable, has high quality tests and is secure

  4. Make it pretty

    In this step, make the software beautiful. Make the text conversational and friendly. Introduce fancy graphics and other visual elements if possible.

  5. Make it fast

    Finally it is time to seek out the worst performing parts of the software and make it fast.

If you follow these steps, and in that order, you should have a high quality piece of software in your hands.

What is polymorphism ?

Polymorphism is touted as one of the awesomest features of Object Oriented Programming. But what exactly is the point of polymorphism ?

Lets take the problem of trying to automate Google search. Lets say your test case is :

  1. Open the browser
  2. Navigate to “www.google.com”
  3. Type “facebook” to the search text box
  4. Click on “Google Search” button
  5. Wait for the page to load
  6. Check if the results page contain a link to “www.facebook.com”

Clearly we need to find some way to drive the browser to do these actions. But how will the automation software know where the ‘search text box’ is ? How will it know where the ‘Google Search button’ is ? So, there is a technology called Xpath that lets you identify a particular element in a web page.

So here is the procedural code that solves the problem ( Language : C ):

int search_google_facebook()
{
    int test_passed = 1;
    char *search_box_xpath = "//searchbox";
    char *search_button_xpath = "//searchbutton";

    firefox_navigate( "www.google.com" );
    firefox_type( search_box_xpath, "facebook" );
    firefox_click( search_button_xpath );
    firefox_wait_page_load();
    test_passed &&= firefox_search_page_link( "www.facebook.com" ):

    chrome_navigate( "www.google.com" );
    chrome_type( search_box_xpath, "facebook" );
    chrome_click( search_button_xpath );
    chrome_wait_page_load();
    test_passed &&= chrome_search_page_link( "www.facebook.com" ):

    return test_passed;
}

Now lets look at the object oriented solution to the problem. Please keep in mind the Liskov Subsitution Principle : base class pointers can point to derived class objects. The code is in Java, so if you are unfamiliar with Java, just keep in mind that classes that implement an interface can be thought of as a derived class of said interface :


public interface BrowserDriver {
    public void navigate( String URL );
    public void type ( String textBoxXpath, String textToType );
    public void click ( String buttonXpath );
    public void waitForPageLoad();
    public boolean searchForLink();
}


public class FirefoxBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Firefox browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Firefox
	}

	// etc
	
}

public class ChromeBrowserDriver implements BrowserDriver {
	
	public void navigate( String URL ) {
	// Code to navigate Chrome browser to the URL
	}

	public void type ( String textBoxXpath, String textToType ) {
	// Code to type "textToType" into text box identified by textBoxXpath in Chrome
	}

	// etc
	
}

public class GoogleSearchFaceBookTest {


	public static boolean testGoogleSearchFacebook( BrowserDriver driver ) {
		driver.navigate( "www.google.com" );
		driver.type( "//searchbox", "facebook" );
		driver.click( "//searchbutton" );
		driver.waitForPageLoad();
		return driver.searchForLink("www.facebook.com");

	}

}


public class TestMain {

	public static void main( String [] args ) {
		BrowserDriver browserDrivers[] = { new ChromeBrowserDriver(), new FirefoxBrowserDriver() };
		for( browserDriver : browserDrivers ) {
			if(!GoogleSearchFaceBookTest.testGoogleSearchFacebook( browserDriver )) {
				System.out.println( "GoogleSearchFaceBookTest has failed!" );
			}
		}

	}

}

If you have a very large test suites, there are a couple of advantages to this approach :

  1. This code is a lot less copy paste and hence less error prone.
  2. Code is easily extensible : Want to add a new test ? Just add it to the main for loop. New test runs with all supported browsers. Want to add a new browser to the test suite ? Just add the driver object to the browserDrivers array. Your entire test suite now runs against the new browser.

How does this magic happen ? If you look at these lines of code :

driver.navigate( "www.google.com" );
driver.type( "//searchbox", "facebook" );
driver.click( "//searchbutton" );
driver.waitForPageLoad();
return driver.searchForLink();

you can see that the code behaves differently ( polymorphism ) based on the type of the driver object. If the driver object is of type ChromeBrowserDriver, calls get dispatched to ChromeBrowserDriver methods. If driver is of type FirefoxBrowserDriver, calls get dispatched to FirefoxBrowserDriver methods. This is the power of polymorphism and it allows us to write modular code.

One important question that could come up at this point is : ‘My favorite language is C. Do I need to switch to Java to write object oriented code ?’. And my answer is : No, it is not the language that is important, but the technique. You can find object oriented code in C. Look at the write() system call in Linux :

ssize_t write(int fd, const void *buf, size_t count);

This is an object method operating on the file object pointed to by the “fd” parameter.

Since C is a simpler language compared to Java, if modularity problems are not common in the problems that you solve, use C. I believe we must use the simplest tool that gets the job done. This reduces chances for bugs and creates more performant software.

On the other hand, if modularity problems are common in the problem you are attempting to solve, it maybe helpful to use a language with support for object orientation. For example, if you say that ‘class A implements interface B’ and while implementing class A, if you forgot to implement a method specified in interface B, you get a nice compile time error in Java. In fact, if you use a good IDE, it will report the problem immediately, before you even decide to compile the program.

How is The Scientific Method useful ?

Almost everyone knows what “The Scientific Method” is. But people are often confounded by why it is such a big deal. For example, when you ask people to prove their theories, they come up with a variety of evasive maneuvers to get out of doing so. So, what purpose does The Method serve ? Is it applicable always ? Does it have limitations ?

Francis Bacon
Francis Bacon was one of the first men to describe The Scientific Method

To answer these questions, we must first ask ourselves, should we prefer some courses of actions over others ? If you answered ‘yes, we should’, we must ask ourselves : why ? The people who are dedicated to The Scientific Method cares about these things because they consider that these principles have, and will serve them well in life. If you do not wish to improve your life, there is no reason to care about The Scientific Method, nor will it be the only principle that will improve your life. This much should be pretty obvious, that a certain and strong desire to win is a prerequisite to being interested in the Scientific Method. So with that let us build the case for The Scientific Method.

So my first step is to convince you of the idea that “Efficiency Is Always A Good Thing” *. So the first question is what is efficiency ? Imagine that you want to travel from Mumbai to New York and there are two ways of doing so :

1. Get from Mumbai to Frankfurt and then from Frankfurt to New York ( Flight time 23 hours, Price : $5000 )

2. Get from Mumbai to Sydney and from Sydney to New York ( Flight time 32 hours, Price : $2000 )

Do you have all the information you need to figure out what is the more efficient choice ? Nope. You also need to know what is the motivation. If I tell you that you are Bill Gates, the richest man on the planet, then you will probably chose option 1. On the other hand if you are considerably poorer but don’t mind an extra 9 hours of flight time you may chose option 2.

The most efficient solution is that solution which helps you achieve your goal using the least amount of resources possible.

I wanted to drive home the point that efficiency depends on motivation since at this point a lot of interesting questions can come around. For example, you may ask me if I should only say to my children : “Eat your breakfast”, “Do your homework”, “Go to bed”, “Go to school” in the name of efficiency. The answer is no, since in this case, your goal is to try and be a good parent and a strong parental bond developed at an early age is essential for children to turn into healthy adults. Idle talk is an excellent way to form bonds with other human beings.

The simplest idea here is that there are usually multiple ways to achieve your goal and the most efficient ones must always be picked if you care about winning. With that in mind, I am ready to try to convince you how The Scientific Method is useful to us in real life.

The Scientific Method is useful to us in three slightly different but kind of inter-related ways :

1. Since it helps us predict future outcomes, it allows us to make arrangements that can helps us achieve our goals by spending fewer resources ( efficiency ).

Eg : Consider the transitive property of debt. If Alice owes Bob $100 and Bob owes Carol $100, then it is the same as Alice owes Carol $100. Imagine, if Alice and Carol lives together ( or nearby ) and Bob lives halfway around the world. Without our understanding of the transitive nature of debt, Alice would have to travel halfway around the world, give the money to Bob, then they would both have to travel back halfway around the world, and Bob would have to hand the money over to Carol and then go all the way back. What a waste ! But if we can predict that the outcome of the above action and the outcome of Alice giving the money directly to Carol are going to be the same, then we can just choose that option and save two full globe trots. Bob and Alice can use the money saved to buy a new car, or some new fancy clothes !

2. Being able to predict outcomes also helps us control them. For example, imagine a Civil Engineer studying the effect of force on a material. By conducting tests on smaller quantities of a material if he is able to predict in advance what is the amount of cement needed to build a bridge ( or whether to use steel reinforcements ) or how many columns are needed to support a building, then we can build giant buildings and bridges without having to lose many human lives.

3. Since The Scientific Method helps us improve our probabilities ( all the tips and techniques for doing proper science, such as using double blind experiments, computing the consequence of a law and then comparing it to nature, are used to increase the value of “likelihood” { the term P(E|H) } in the diachronic interpretation of Bayes’ Theorem ) it can also help resolve arguments and convince others.

* It is very important to understand the side effects during your chase for efficiency. For example, in the case of flight routes from Mumbai to New York depicted in this post, if everyone where to get rich and always chose Option 1 ( via Frankfurt ) the company running the Australian airways may go into a loss and the people employed by them may get fired and try to make your life difficult for the heck of it. Naturally the probability of this happening is very low in this case, but they must also be accounted for when you are computing your decision.

What features would be present in your dream programming language ?

OK kids, back to programming.

I am stating some of my opinions on what should be in my dream programming language here. This will be an evolving set of ideas, I will be updating them as I learn more.

Functional Requirements

  1. Lexical scoping
  2. Static typing
  3. Pointers
  4. Object Orientation
  5. Java Interfaces ( Contracts )
  6. One class per file
  7. Java style namespaces ( packages )
  8. Checked Exceptions
  9. No mandatory garbage collection ( facilitate RAII )
  10. Single inheritance, multiple implementation
  11. Data types : String ( buffer + size ), int, float, char, boolean, hashes, arrays, structures
  12. Reflection
  13. Annotations
  14. C style syntax where possible ( no white space based scoping )
  15. Automatic destructor invocation ( avoid @Predestroy from Java )
  16. Specification and test suite, but also a reference implementation
  17. Generics

Non-functional requirements

  1. Integrated Development Environment ( IDE ) friendly
  2. Blue collar language
  3. Fast start up
  4. Fast execution
  5. Easy to hot load code
  6. Compiled
    1. Leave portability to LLVM
    2. After decades of “Sufficiently smart profiling JIT compiler should run code faster than sufficiently smart optimizing AOT compiler” it still has not materialized

This language should be developed under a Benevolent Dictator For Life ( BDFL ) model -as opposed to design by committee- and should also consist of a maven style declarative build system with sensible defaults.

What features would be present in your dream programming language ?

Perl 6 evaluation

Perl was the first ever scripting language that I picked up. I love it, but I kind of feel constrained by it. I knew about Perl 6 but I had never looked at it before. So I decided to take a look.

Perl 6 project was started 14 ( FOURTEEN !!! ) years ago after the now famous mug throwing incident. Perl 6 itself is a specification for which multiple implementations exist. There are four major (incomplete) implementations right now.

Pluses

  • Class keyword.
  • No flattening of lists.
  • Sigil invariance.
  • Gradual typing.
  • It is claimed that .NET implementation ( Nieceza ) has the ability to call .NET libraries.
  • v5 mode and JVM mode could potentially help Perl 6 code to inter-operate with Perl 5 and JVM code respectively.
  • Feels like Perl 5 ( potentially easier for Perl 5 devs to pick up ? ).

Minuses

  • No IDE support ( It may be hard to write something on Intellij, since only Perl 6 can parse Perl 6 ) ( Only vim syntax highlighting available now ).
  • Very few packages available ( one or two I have tried are bad ).
  • Nieceza seems to be falling behind.
  • Does it have something to constrain TIMTOWTDI ( for example contracts ) ?

Misc

  • Documentation problem is being addressed.
  • Devs on Perl 5 may not care since, contrary to popular belief, Perl 5 development is still going well.
  • MoarVM could be a good fit for client-side apps and one-off scripts and JVM port for server-side/Android.
  • Not a second system ( Perl 5 was an almost complete re-write of Perl 4 ).

How to try

Do not use your Linux distribution’s package.
Use Rakudo Star :
http://rakudo.org/downloads/star/

To install :

sudo perl Configure.pl --gen-parrot --prefix=/opt/perl6
sudo make
sudo make install

Perl 6 binaries, panda (==cpan) and some modules get installed to /opt/perl6
Try out like this :

simula67@hades ~/repos/perl6/1test $ cat test.pl6
#!/opt/perl6/bin/perl6
use v6;
class SampleClass
{
	has $.x is rw;
	method get_x()
	{
		$!x;
	}
}

my $sample_obj = SampleClass.new;
$sample_obj.x = 10;
say $sample_obj.get_x;

simula67@hades ~/repos/perl6/1test $ ./test.pl6 
10

If you love Perl you should definitely check it out.