Now that I am no longer working for Nutanix, I feel like I can talk about what I learned about Hyper converged infrastructure and cloud computing in the last 2 years. Let’s look at how hyper converged architecture is formed and then we will discuss the advantages of structuring your IT infrastructure this way. Most of what is mentioned here is a simplification of Nutanix Bible.
What is Hyper converged infrastructure ?
For a system to be hyper converged, it needs to have at least 4 components :
Distributed storage software
Management plane that can manage all of the above
So, let’s look at it from Nutanix’s point of view.
All software needs hardware to run on and for hyper converged platforms, this is not different. You can buy hardware from Nutanix, Dell-EMC, Lenovo etc. Nutanix hardware is usually standard x86 servers from Supermicro ( although that may have changed ) with Nutanix branding. This hardware is not special, which is why you can swap it out with Dell-EMC, Lenovo gear.
Hypervisors are usually purpose built operating systems that can create virtual machines ( VMs ). So, on the hardware, first you need to install a hypervisor. Here again, you have choice. You can install Nutanix’s own AHV , or you can install VMWare ESX or Microsoft Hyper-V.
Using the hypervisor above you can create a VM that has direct access to all the disks on the computer. This VM ( called a Controller Virtual Machine or a CVM ) talks to other CVMs in the data center, forming a cluster. On this VM, a set of software is installed which aggregates all the disks on the computer ( and the disks on all the other computers in the cluster ). This is then presented back to the hypervisor as a single unit. This component is the absolute minimum that must come from Nutanix if you want to use Nutanix platform.
Some management software is installed on the CVM which can be accessed by going to https://CVM_IP_ADDRESS:9440 using a web browser. For Nutanix, this management plane is called PRISM. Besides serving the user interface, it also serves a RESTful API which can be used to automate tasks on the cluster. PRISM is a must for managing storage in Nutanix cluster, but you can still manage your VMs from VMWare vCenter for your ESX cluster.
Advantages of Hyper converged infrastructure
So why go through this at all ? This seems much more complicated than just running your applications directly on hardware. It is even more complicated than running applications on a virtual machine on the hardware. ‘Distributing storage ? Presenting the storage back to the hypervisor ? You are out of your mind, sir !’ I can hear you saying. Here are some reasons :
Self healing : Hyper converged systems offer automatic data replication. When VMs write data to disk, the platform writes a copy elsewhere. So if your hard disk fails, just rip and replace it when you have time. The system notices that there is now only one copy of all the data that was in that hard disk and automatically builds a second copy. Same applies when a single computer in the cluster fails.
Expandability : You can start with just a few computers and then grow from there as your computing needs grow. You don’t have worry about to buying a large number of computers upfront and then not needing it.
Easier to manage : Hyper converged systems offer a single interface to manage all your IT infrastructure. Manage storage, compute etc from the same web based console.
Efficiency : The platform can notice when duplicate data is being written to disk by virtual machines. For example, your VMs serving live traffic may be cloned by developers to debug issues or to develop new features. Most of the data on these VMs may be similar and can be deduplicated saving storage space.
Security : Data can be encrypted automatically by the platform before they are written to disk.
There are many more advantages of course, but that should justify the extra complexity.
Server Image Manager is used to generate answer file for sysprep
Server Core only has a command line ( cmd.exe ). It’s a stripped down version of windows. Some services or roles do not work on server core so make sure the services you want work before choosing server core
Windows installation ISO contains a ‘.wim’ file which is used to install Windows
This ‘.wim’ file can be used to install windows using PXE ( this uses windows deployment services [WDS] )
‘.wim’ files can also be created from a reference computer using the Image Capture Wizard
In enterprise environment you should ideally use Windows Software Update Services ( wsus ) to update Windows
Run “oobe” in “Run” line to get the Initial Configuration Tasks
UAC asks for permission if you are running as administrator. It asks for administrator username and password if running as normal user.
This can be turned off by going to “Control Panel” -> “User Accounts” -> “Change User Account Control Settings”
System settings is Right Click on “My Computer” and select “Properties”
WORKGROUPS and used when you have smaller networks ( 10 -15 ) -> decentralized
Active Directory is used when you have larger networks to manage -> centralized
RAID0 stripes data across all drives. if any drives fail all the data is gone
Host Bus Adapter is a network adapter that is used specifically to access storage network
Monitoring and Troubleshooting
“System Information” gives you information about all the individual pieces of the system
“Event Viewer” gives you details on all the interesting events in the system
“Master Boot Record” is the first 512 bytes of a partitioned storage device such as hard disk
Code written here passes control to Volume Boot Record
Volume Boot Record ( VBR ) contains the code to load NTLDR and BOOTMGR
It used to be boot.ini file that told Windows how to load
Now you can use BCDEdit to modify it ( BCD = Boot Configuration Data )
“System Configuration” ( msconfig ) is boot and trouble shooting tool which is service based
Turn on safe boot, turn services off etc
Once you figure out what the problem, set the settings back to “Normal Startup”
Virtual memory settings are shown in “system properties”
Usually page file is at : C:\pagefile.sys
Control Panel” -> “Administrative Tools” -> “Windows Server Backup” to use builtin backup tool ( not enterprise ready )
UPDATE: This post should not be considered an endorsement of Fideism
First of all, let me say that I am not an official spokesperson for the Roman Catholic Church. I tell this story because I feel it needs to be said, but it may contain inaccuracies.
Part I : French Revolution
In 1689, St. Margaret Mary Alacoque receives a vision of Jesus. Jesus reveals to St. Margaret that despite the love he has shown to men we only repay Him with ungratefulness and wickedness. He orders St. Margaret to ask the King of France to consecrate France to His sacred heart. The King Louis XIV ignores this order.
The Napoleonic wars severely depopulates France with a large number of people perishing in Napoleon’s thirst for power
Catholic Church suffers major setbacks : it’s nuns and priests turned out, leaders killed or put in exile, it’s property seized and controlled by enemies
There is an attempt to de-Christianize France
However, the effects were partially reversed in time. Some effects still remain to this day. Most important of these are “liberalism” and the idea of replacing God with reason. See : The Cult of Reason.
France has corrupted the universe, one day it will be punished. The faith will die out in France: three quarters of France will not practice religion anymore, or almost no more, the other part will practice it without really practicing it. Then, after [that], nations will convert, the faith will be rekindled everywhere.
A great country in the north of Europe, now Protestant, will be converted; by the support of this country all the other nations of the world will be converted
I see that there are many versions of this text and some readings of these messages will lead you to consider the notion that there is nothing much to it. Be warned that interpreting these messages are difficult for the best of us, but at least we can see that half of this has already come true. If you look at religion in France today, only a quarter believe in God.
So now, the question is which is that great country to the north of Europe ?
Part III: Our Lady and Fátima
In 13 May 1917, three children ( among them Lúcia Santos )in Fátima, Portugal reported a vision of Mother Mary.
The Lady revealed Three Secrets to St. Lúcia. Here is some text from the second secret :
The war is going to end: but if people do not cease offending God, a worse one will break out during the Pontificate of Pope Pius XI. When you see a night illumined by an unknown light, know that this is the great sign given you by God that he is about to punish the world for its crimes, by means of war, famine, and persecutions of the Church and of the Holy Father. To prevent this, I shall come to ask for the Consecration of Russia to my Immaculate Heart, and the Communion of reparation on the First Saturdays. If my requests are heeded, Russia will be converted, and there will be peace; if not, she will spread her errors throughout the world, causing wars and persecutions of the Church. The good will be martyred; the Holy Father will have much to suffer; various nations will be annihilated. In the end, my Immaculate Heart will triumph. The Holy Father will consecrate Russia to me, and she shall be converted, and a period of peace will be granted to the world.
To me it looks like the ‘war that is going to end’ is World War I and the other one that ‘breaks out during the Pontificate of Pope Pius XI’ is World War II.
Our Lady fulfills her promise, asking “for the consecration of Russia to Her Immaculate Heart, promising its conversion through this means and the hindering of the propagation of its errors
At Rianjo, Spain in August 1931, Our Lord communicated to Sister Lucia His dissatisfaction. Here is what He said :
Make it known to My ministers, given that they follow the example of the King of France in delaying the execution of My requests, they will follow him into misfortune.
They did not wish to heed My request! Like the King of France they will repent of it, and they will do it, but it will be late. Russia will have already spread its errors in the world, provoking wars and persecutions against the Church. The Holy Father will have much to suffer.
Many consecrations of Russia are attempted ( in 1942, 1952, 1964, 1982 ). St. Lúcia informs that none has fulfilled the conditions ( the conditions are that all bishops in union with the Pope and Russia must be specially mentioned )
In March 25, 1984, Pope John Paul II, “united with all the pastors of the Church in a particular bond whereby we constitute a body and a college,” consecrates “the whole world, especially the peoples for which by reason of their situation you have particular love and solicitude.” Both the Pope and Sr. Lucia initially seemed uncertain that the consecration has been fulfilled, but shortly thereafter Sr. Lucia tells the papal nuncio to Portugal that the Consecration is fulfilled.
Technically Russia is to the north of Europe and technically the country is Protestant, in the sense that Orthodox Christians do not accept the authority of the Pope. Since the fall of Communism religion is returning to Russia.
Part IV: Why did I tell this story ?
Regardless of what you think about the interpretation of these events, what you have to accept that there are currently two major threats to our faith :
Replacing God with the State
This is mostly a Communist idea. We can see echoes of this idea in modern left wing and liberal movements. Since I went to a government college and grew up in a Communist influenced state, I know this is a bad idea already. I do not think governments are very good at their jobs. There may be good people in the government, but mostly it is an instrument used by people with power to take advantage of people without power. Incentive structures within the government are setup so that people get paid well even if they mistreat the people who have to deal with these bureaucrats.
Replacing God with Reason
This is the modern vision put forward by New Atheists. We are required by our professions to maintain and upgrade our capacity to reason, and for a long time, we have done so. We have built our minds into a Fortress of Reason, and this benefits us. Unfortunately, to maintain our relationship with God, it is upon this Fortress of Reason that we must commence our siege.
I have been reading more and more about American politics and recently been pushed more and more to the right of the political spectrum. So, I looked at how the system fares in my home state of Kerala and was surprised to learn that the Congress ( the supposed right wing in Kerala politics ) is actually center-left. So I want to talk to you about the importance of having a proper right wing party in Kerala.
The term first appeared after French Revolution when the people who supported the King sat on the right side of the President in the National Assembly while the people who supported the Revolution sat on the left. This was because the supporters of the King wanted to avoid the shouts, swearing and other indecencies that enjoyed a free reign on the left. The Left was the ‘party of movement/change’ and the Right was the ‘party of order’. But over time left right division has been associated with different political ideologies.
What is the Left-Right divide in American politics ?
If you look at contemporary American politics, it seems to me like the Left’s central ideology is ‘Social Justice’ ( supported by the Democratic party ) while the Right’s central ideology is ‘Individual Liberty and Individual Responsibility’ ( supported by the Republican party ). This division is absent in Indian or Kerala politics. Also the problem here is that since French Revolution’s slogan was ‘Liberty, Fraternity and Equality’ how can the Libertarians be on the Right ? The best explanation I can think of, is since the country is majority Christians and since most ( not all ) Christians believe in gentle persuasion as the only acceptable tactic for spreading their ideas it boils down to traditionalists enabling liberty. I have talked about the three rules of freedoms and rights. Most libertarians are for maximizing freedom, not for anarchy. They understand the symbiotic relationship between self-control and freedom. If I have a belief that I must not take your life, you have the liberty to your life. If you want liberty to your property then you want everyone to be convinced that it is immoral to steal. So since most religious people are mindful of other people’s rights this maximizes the freedom available to everyone and therefore they are on the Right side of the spectrum.
The Left side of the political spectrum consists of Socialists, Communists etc. But even though I am for equality of opportunity I can not be for equality of outcomes. I suspect that inequality is a natural state and different people have different motivations, ambitions and inclinations. Ensuring equality of outcomes will be oppressive rather than liberating. That being said, I agree with a lot on the American Left. There is an intellectual corruption on the American Right, with their support for creationism and denial of climate change. Unfortunately, as far as I can tell, nobody on the Right is attempting to reform it like Dave Rubin is attempting on the Left.
After I noticed the problems with the Left’s argument on pro-choice I started to learn about Republican positions. For the longest time, I thought that if I went to US, I would be a leftist ( or a liberal ), but now I think I would be a right-winger ( or a conservative ). I think shows like ‘Last Week Tonight’, ‘The Newsroom’ and ‘The Daily Show’ misrepresent the Republican viewpoints.
What is the major problem with Leftist philosophy ?
To me the main oversight in Leftist philosophy is the potential for evil in human beings. Most Leftist parties are ‘for poor people’ and ‘for oppressed minorities’. If you start a party and you spend a large amount of time trying to promote it as the ‘Party of the oppressed’ or ‘Party of the poor’, then if these people who vote for you stop being oppressed or stop being poor, then they will no longer vote for you. And since most important thing that politicians care about is ‘coming to power’, parties for the poor tend to become parties for poverty. Parties for the oppressed tend to become parties for oppression.
How is all of this relevant to Indian or Kerala politics ?
In Kerala our choice is between Congress ( center-left ) and Communists ( far left ). Why is there no party that supports individual liberty ? All political parties want to expand their power while we know the government is a corrupt machinery incapable of getting much done. Why is no one advocating for limited government ? Even if you believe that leftist philosophy is good, you must believe that there needs to be a strong opposition to the left, for purposes of balance. Otherwise those of us who are Keralites will have to live as an alien outside Kerala to further our career ambitions and live in a corrupt state when we return.
External targeted contributions. For example, most people submitting patches to open source software are facing some issue as a user. If they are experiencing an issue, others may also be experiencing similar problems. This is a much better way of driving development, compared to doing it post expensive market research.
Increasing the size of markets. For example Google is trying to bring more people online with Android to increase the size of search market.
Agile is a combination of two processes that are designed to develop software. The two methodologies that form Agile are : Scrum and Kanban.
Key to understanding agile is understanding the objective tree. Here is an example objective tree for a company like amazon.com which aims to build an e-commerce website.
There are many people who argue about the number of levels in the tree and how they should be called. But the key insight here is that :
a) Objectives form a tree
b) When child objectives are done, the parent objective is automatically done
There are two methodologies within Agile : scrum and kanban.
Objectives are broken into user stories. A prioritized list of user stories is called a backlog. Product management owns the backlog.
Objectives are achieved by working on them in 2 weeks sprints. Team pulls the maximum number of stories that can be completed into the sprint backlog and starts working on them when the sprint starts. Sprint backlog is owned by the team.
Every day of the sprint there is a 15 minute standup meeting where team members describe 3 things :
What they worked on yesterday
What they are planning to work on today
Blockers ( if any )
This is an opportunity for micro-course-correction. If a story was done, team members can move to help with another story. If there is something that blocks a team member, management can move to unblock them. If a story might spillover, it can be called out.
At the end of the sprint there is a sprint retrospective to re-evaluate and course correct at a higher level.
Every N sprints ( for a finite value of N ) there is 1 refactor sprint. No new features are delivered in this sprint. Architectural runway is prepared for incoming stories and code is made more maintainable.
Why is the process like this ?
Reason for product management owning the backlog is to communicate clearly what the current priority is. This can be challenging in a changing environment. For example, management may ask the team to support an application on Windows 2012 platform and mark it high priority since largest customer base is on that platform. Afterwards it could turn out that highest paying customers are on RHEL 7.1. Management can communicate this information to engineering by adding a new story to support RHEL 7.1 and deprioritizing the story for Windows 2012 support.
Team owns the backlog because the team is responsible to ensure that the features ( stories ) committed at the beginning of sprint is done by end of sprint. This is why it’s the team that need to pull the stories into the sprint ( and not the manager ) so they feel a sense of ownership.
Standups give an idea about the pulse of the sprint : whether the team is on track, if it needs adjustment etc. Since software development is a collaborative process, it can get blocked on other people. For example : UI team can get blocked on UX mockups, developers can get blocked on APIs that were promised by other team members, one team member may be blocked for code review from another etc. Daily meetings help with dealing with these problems as they arise. It also enables the team to help others, for example, if someone airs a database problem, someone else on the team may have encountered and solved the problem before. Standup meetings should be short and sweet. If they are too long, since they occur everyday, they become a productivity drag.
The sprint retros are an opportunity to improve things at a higher level. For example, developers may notice that the CI system could run a larger portion of automated tests, reducing QA burden. Developers may point out that the code is getting too complex in one area and request refactor stories.
1. If a story is in the backlog, it may get done. If a story is not in the backlog, it will never get done.
Why is this expectation there ? Sometimes product management may request a feature. Engineering does not know if that feature was important, whether it was a priority etc. Moreover, there could be multiple managers requesting different priorities for stories fuelling confusion. This expectation sets up management to close out the relative priority of objectives before people set out to achieve them.
There is no such thing as an implicit requirement. Management owns the product backlog.
2. Target spillover for any sprint is zero
Why is this expectation there ? For hard software engineering problems it is required that the team is a highly committed one. An uncommitted team member will usually bring down the whole team since the job is collaborative in nature. Target spillover of zero ensures that the team remains focussed on the problem at hand.
3. No activities are planned mid-sprint
Why is this expectation there ? This is to help the team remain focussed so that the cost of context switching is lowered. It is called a “Sprint”, not a “Trot” or “Gentle Walk”. Management is expected to defend the team from external influences. Course corrections are possible at the end of sprint. Therefore, it is important to keep the sprints as short as possible to get a meaningful amount of work done. The length of your sprint should be the minimum amount of time it takes you to respond to external influences. Usually, it is 2 weeks.
4. QA should raise a ticket if there is a problem.
Why is this expectation there ? It is possible that QA notices a problem and communicates it verbally or through email. Now people may go on vacation etc cutting the team’s ability to deliver.
There is no such thing as an implicit ticket. Raise it in issue management software.
5. No story is done until :
a) All code is reviewed
b) There are no open tickets
c) Documentation is complete
Why is this expectation there ? Code reviews are important because it lets someone else get familiar with that code base allowing them to take over in the event of the author’s absence. It is important to close all open tickets before calling a story done since otherwise your development process simply turns feature requests to bugs. Documentation should be developed lock step with code since that is when it is easiest to write it.
6. Do not test from feature branches
Why is this expectation there ? Someone could merge something else into the development branch making the feature tested in feature branches to be unviable in development branch. Development branch is what ultimately gets shipped to users so that is where things should be tested.
7. Stories are of the form “As a [user] I want to do [task] so that I can [accomplish goal]”. If they are not in that form, it can be safely deleted.
Why is this expectation there ? Often stories may be created with no clarity on what the next step is. Often developers may misunderstand the requirement and deliver a different feature requiring multiple unnecessary round trips and feedback loops to complete the feature. If the story is not in the prescribed form, it can be deleted and considered non-existent since it motivates people to state the objectives of the story explicitly.
A few points
In Scrum the keywords are focus and course correction.
Manager – employee relationship is co-operative. The team does everything to make the manager look good ( by delivering high quality features that delights users ) and managers does everything to setup the team and employees for success : removing impediments, giving them tools and access they need to meet the objectives.
Dev – QA relationship is quasi-adversarial. QA tries to defend the customer demanding issues be fixed within reason. Dev team appreciates QA for finding the bugs they have missed that could have caused them to deliver an inferior product.
Agile requires self motivated team. An under-motivated team may over inflate estimates or under commit into a sprint.
There is no such thing as half-agile. Agile gives management a certain amount of power and engineers certain amount of power. This is to improve efficiency by reducing friction and miscommunication. If management says : ‘I will accept these parts of the process that gives me power and ignore these parts that require me to relinquish power’ this process will fail. It requires humility on the part of management and engineering.
This is usually for support. After product is developed, support is trained on resolving bugs that get into production. Support can’t wait 2 weeks to resolve issues. When tickets come into the queue people take them on and try to get through them as fast as possible.
Scrum is more proactive and Kanban is more reactive. Kanban is for situations where a 2 week turnaround time is unacceptable.
Agile came about as a response to the Waterfall model of software development. It helps solve the most important problems with waterfall such as quicker course correction, improving focus and reducing miscommunication.
In waterfall, the software developed in following phases ( one after other ) :
A mistake in early phase cannot be detected until last phase.
There is little focus as work drags on for years.
People working on each step only communicates with people in previous step increasing communication errors.
Should Indian Railways be privatized ? Looking up the answer to this question on the Internet would lead one to conclude that no, it should not be privatized. I am not convinced, I think it should be privatized.
Lets look at the arguments against privatization, one by one.
It is not clear how it will improve situation
The argument is that privatization and increased competition will improve the quality of service, drive down prices and spur innovation. It should reduce corruption and force employees to do their best. For a government owned company the employees have no reason to work hard, treat its customers well and figure out how to make everything better. They get paid regardless. This causes them to mistreat their customers and generally be lazy about their work.
The Indian Railways lost ₹30,000 crore (US$5 billion) in 2015. This is despite the fact that it is a high demand business with a government sanctioned monopoly. Booking a train ticket is so painfully difficult that it a subject of so many degrading memes on the Internet. The compartments are noisy and sleeper classes are filled with bed bugs. The toilets are dirty. The staff is unpleasant to deal with. The train stations are riotous and unclean. It smells of shit and urine.
I wonder how many levels of incompetence it takes to suck so badly at a job.
Privatization has helped the Indian telecommunication sector and aviation sector. It is possible for it to work in the rail industry as well. Even if it doesn’t help, it is at least worth a try.
Look at the state of American and British Railways
They are significantly better, aren’t they ? But even if it isn’t, there are a thousand reasons it could work better in India. Yes, I know there are a thousand reasons why it couldn’t work also. But why not give it a try ?
Concerning America, their landmass is 3 times that of India and they are a very affluent and urbanized country. This means that most of the travel is very long and hence very inconvenient to be conducted by train. So rail is used mostly for goods and human beings pay a little more to travel by planes.
It is a natural monopoly due to high cost of entry and exit
This is partially true. The high cost of entry and exit is largely due to high cost of building and operating railways, so we may need to have some government assistance with regard to this.
One solution to this problem is to let the rails themselves be owned by one company and combine this with multiple companies owning the individual trains. The train companies lease the rails from the rail company to provide their service.
Sure, some government assistance will be needed to acquire more land, ensure that there is stiff competition etc but these are all problems that have well known solutions. It can definitely not be a fire and forget endeavor.
Also I don’t understand how it becomes harder for individual investors to exit the rail business as most companies are traded on the public stock market. If you want to enter the rail business buy a few shares of the company at the Mumbai Stock Exchange. Want to exit ? Sell them at the stock exchange.
Some routes may be under serviced because they may be unprofitable
Form a government company to service only these routes using tax payer money. There is no need to keep the entire business in the public sector and cause incalculable misery to everyone in India to benefit a few.
The railway employs a lot of people these jobs may be lost in privatization
Privatization does not automatically cause loss of jobs. If a private railway business can operate with fewer number of people and automate away a lot of these jobs it would benefit the customer with lower fares. This money saved by customers can be used to buy new things which creates new jobs. See the parable of broken windows. Another possibility is to tax the rail companies and pay unemployment benefits to these people directly. At least the public will be spared from their laziness and callousness.
Defense concern. Countries like China could buy up all Indian Rail businesses and in the event of a war, discontinue them all
We can repossess these businesses in the event of a war. The other problem is if they sabotage it silently while they are holding it. This problem can be solved by stronger policing.
National pride and unity
A lot of arguments center around how Indian Railways is a matter of pride for Indians. Sorry, I would rather travel comfortably than with pride. Also, I don’t think it is a matter of pride given how absurdly mismanaged it is.
So anyway, I think most of these arguments are bogus or ill-informed. I have similar thoughts about the power industry ( power cuts are uncommon in Western countries ) since there is no innovation in solar, wind, nuclear power in India despite us having no serious oil reserves. I think India is a poor country on account of power. But more on that later.
What do you think ? Let me know in the comments section.