• Regular scrums – usually daily
• Micro-Issue tracking
• Measurements

There are of course many other aspects to Agile methods and Evan Leybourn @ The Agile Director has alot of experience in implementing Agile methods in software development teams. He has a few courses running in Canberra and Sydney. Check them out:

• Sydney – 2-3 April12 – Agile Methods
• Sydney – 4 April12 – Advance Agile Methods
• Canberra – 10-11 April12 – Agile Methods
• Canberra 12 April12 – Advance Agile Methods

So what exactly is the cloud, and why will it provide the computing capabilities which has been dominated by super-computers over the last 2 decades. And why will cloud computing succeed where grid computing failed.

In May 17 1999, SETI@Home was release, and it gave the public a glimpse of how inter-connected computers could be leveraged to perform very large tasks. Grid technology was encapsulated in technologies like SunGrid and xGrid, but largely failed to gain traction. The internet was only starting to go mainstream, and computers were still expensive items.

A decade and a bit later in 2012, Cloud-computing is making headlines, and it seems that cloud-computing may succeed where grid computing failed. So what has changed since 1999?

• Computers are cheaper
• The Internet is much faster
• VMWare and Virtualisation is gaining inroads into organisations
• Hosting and Infrastructure companies are virtualising
• Accessing virtual services like email, social media, SaaS is common place.
• Increased awareness of online computing via Amazon Web Services, SalesCloud, Azure.

So will super-computing be replaced? Will there be reduced demand in running parallel jobs on multiple computing nodes? NO. There is significantly increased demand in running computer-intensive and parallel jobs. However the way in how a super-computer might be implemented will change. Instead of proprietary platforms, super-computer will evolve to open-platforms and be built on the cloud. The proprietary bits of super-computing will be the charging mechanisms for the utility.

Will grid computing be replaced? Grid computing will fade away. Grid computing addresses the same type of distributed super-computing that cloud computing would replace. The traditional super-computer might still serve a purpose for tightly-coupled applications which are difficult to distribute to the cloud or grid.

Consumers are not interested in a technology, but rather what they can do with it. In Cloud computing, this becomes more apparent with products like:

• Database processing
• Running an algorithm
• Getting an answer

See photos here.

1) Virtualisation is going to happen whether we like it or not

• This was driven initially by under-utilized servers, but ease of management and configuration has taken over as the leading reasons for virtualisation.
• Currently only 30% of organisational server infrastructure is in the virtualised environment. If an organisation doesn’t reach 80% virtualised, it doesn’t gets the efficiency benefits of virtual infrastructure, but ends up with large overheads managing both virtual infrastructure and traditional infrastructure.
• VMware hopes to push this to 50% in 2011-12
• The issues with adoption are confidence levels in application-infrastructure interoperability, and security.
• VMware has notoriously low security, and is itself a gateway to accessing the entire virtualised infrastructure. (Search Google for “vmware hack”)

2) Overheads

• Virtualisation comes with overheads.
• If installing Vista, or Windows 7 was not enough, virtualisation can help by adding 10-20% overhead to CPU usage.
• VMs also generate alot more network traffic.

3) Configuration

• VM configuration is going to be crucial as “the server” is spread over a VM, SAN storage, and network “bus”, and actual physical locations. So when we have slow VMs, it could be the result of alot of different factors now.
• A clone of VM for failover/failback scenarios can also generate alot of network traffic. So virtualisation increases network overheads.

4) The Virtual Desktop

• VMWare hopes to bring back thin-client computing with virtualised downloadable profiles from VM infrastructure.
• Personally, I think this is a shot in the dark, as the PC-era is gone, and computing is already transitioning to the fragmented plethora of thin-clients (eg mobile devices, ipads, netbooks) with profiles stored in SaaS applications.
• The benefits of centralized profiles is supposedly in data security, however, with SaaS, fragmenting application, platforms, I doubt the virtual desktop will make it to the enterprise before iDesktops.

5) Capability

• VMWare ESX 5 now support 32 cores, and upto 1TB RAM per VM. These are called the “Big VMs” (or “Monster VMs” if you were a VMware sales person) that VMware has now released.
• This may support the more computationally intensive applications, but only if the virtual infrastructure has been upgraded.

6) Visibility

• From an application development point-of-view, understanding the performance and capability of an application in the virtual infrastructure.
• Processor CPU utility within a Windows/Unix VM is not an accurate reflection of the actual processing capability available to your application.
• So VM infrastructure performance statistics needs to be actively shared (in real-time) with application teams.
• Using SPEC CPU benchmarking tools is another way to measure application-infrastructure performance.
• However let’s hope for an open environment with open information sharing.

7) Super-Computing / Grid Computing

• Although there has not been any noted implementation of supercomputing in VM infrastructure, there are no reasons why this is not possible.
• Grid Computing, and maybe some aspects of super-computing is probably possible on VM infrastructure with the appropriate HPC software in place.

The Carbon Footprint

• The Carbon Footprint is now the new driver for VM infrastructure.
• Running un-optimised / under-utilitzed servers kills the environment.
• If electricity prices go up by 30% in the next 2-5 years, what will organisations have to do to mitigate that?

We can compare Orange to the Trident Platform from Microsoft. The only difference is that its open source and works better.

Orange is free software; you can redistribute it and/or modify i under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or Orange is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

• The progress if Napoleon’s Army vs Distance vs Temperature in the Russian Campaign of 1812
• The Origin of Cattle destined for Paris

Charles was trained as a civil engineer. Cartographia has a good list of Minard’s work.

For in-depth study, check out this.

An Example

You have a dataset on radiation levels. (thanks to fallout from nuclear powerstations). A very quick and common question that demands immediate answers would be “What is the impact of increased radiation?”. That is a very broad question, and even with skillful narrowing of the scope of the question, this question still needs to be answered. Even the basic remaining key perspectives on the question may be:

• Effect on population?
• Effect within a radius of 100km?
• Effect on transportation within 100km?
• Effect on travel?
• Effect on tourism?
• Effect on agriculture?

All these questions will require the custodians of co-related datasets to make their data available. The negotiations to acquire the data would probably take time. Followed by the data modeling, loading and analysis. The final outcomes would still be achieved, but under the strain of time and effort.

We can reduce some of this time by having open data, and configured data. Consider plug and play data. Consider being able to draw data from established datasets with minimal processing, and be able to derive results quickly. This is where Glitchdata would advocate data by configuration.

• The App Presentation Layer
• The App Web Services Layer
• The App Business Logic Layer
• The App Database Layer

As database systems have evolved rapidly over the last decade, we see database systems providing features like foreign key enforcement, indexing, view, triggers, data transformation, fulltext indexing, spatial capabilities, and more.

The problem here that databases start getting bloated, and stop focusing on the key value that they provide. Data storage and retrieval.

So it stands to reason why Amazons Web Services have offered SimpleDB has its key database offering for Cloud services. Of course they also offer other relational database services.

So why does Amazons prefer SimpleDB? Scalability, and lower costs/GB of data stored.