Kyo Lee

Open-Source Cloud Blog

Tag: agile

Docker Strange Dev: How I learned to stop worrying and love the VM

A couple of months ago, my laptop died. I remember the day I brought it home for the first time. Still in college. I made a big decision—despite of my marginal finances—to go with the top model. “It should last at least five years,” I convinced myself. Seven years later, I got the news from the certified Apple repair person, “Well, you should just buy a new laptop than trying to fix this one.” So I did.

Now, I have a new laptop, which is currently placed on a pedestal. Watching how fast applications open and close on this laptop makes me cry. The future has arrived. I tell myself, “No! I will not f*** this one up this time! I will never install any software on this laptop ever!” Of course, I am in denial. This pure awesomeness will eventually decay. I predict the rate of decay exponentially accelerated by the number of software installed on the laptop (Disclaimer: No supporting data exists for this assertion—other than my paranoia).

I begin searching for the answer to the quest: Install no software. So I try Docker.

Docker_logo

The noticeable difference between using Docker, which runs Linux container, and using the virtual machine (VM) is speed.

Both options provide the ability to isolate software’s running environment from the base operating system. Thus both deliver the desirable paradigm: Build once and run everywhere. And both address my concern—that the base operation system must remain minimal and untouched. However, Docker stands out from the classic VM approach by allowing the cloud application—which is a virtual image with desired software installed—to run within milliseconds. Compare that to the minutes of time it takes to boot a VM instance.

Speed_Racer

What significance does this difference make?

Docker removes a big chunk of mental roadblock for developers. Thanks to Docker’s superior response time, developers can barely identify the perceptual distinction between running applications on a virtual image (a virtual container, to be precise) and running applications on the base OS. In addition to its responsiveness, Docker appeals to the developers by obsoleting the tedious procedures of booting the VM instance and managing the instance’s life cycle. Unlike previous VM-centric approach, Docker embraces the application-centric design principal. Docker’s command line interface (CLI) below asks us two simple questions:

1. Which image to use?

2. What command to run?

docker run [options] <image> <command>

With Docker, I can type single-line commands to run cloud applications, similar to other Linux commands. Then I will have my databases servers running. I will have my web servers running. I will have my API servers running. And I will have my file servers running. All these are done by using single-line commands. The entire web-stack can now be running on my laptop within seconds. The best part of all this? When I’m done, there will be no trace left on my laptop—as if they never existed. The pure awesomeness prevails.

docker-whaleeuca_new_logo

Running Eucalyptus Console on Docker

 

For those who want to check out Eucalyptus Console to access Amazon Web Services, here are the steps to launch Eucalyptus Console using Docker on OS X.

Step 1. Install Docker on your laptop

Here is a great link that walks you through how to install Docker on OS X:

http://docs.docker.com/installation/mac/

Step 2. Pull Eucalyptus Console Docker image repository

Run the command below to pull Eucalyptus Console Docker images (it will take some time to download about 1.5 G image files):

docker pull kyolee310/eucaconsole

Run the command below to verify that the eucaconsole images have been pulled:

docker images

Screen Shot 2014-09-14 at 10.52.11 PMFor those who want to build the images from scratch, here is the link to the Dockerfile used:

https://github.com/eucalyptus/dockereuca

Step 3.  Update Docker VM’s clock

When running Docker on OS X, make sure that OS X’s clock is synchronized properly. A skewed clock can cause problems for some applications on Docker. In order to fix this issue, you will need to log into the Docker VM and synchronize the clock manually.

You can SSH into Docker VM using the command below:

boot2docker ssh

Once logged in, run the command below to sync the clock:

sudo ntpclient -s -h pool.ntp.org

Run the command below to verify that the clock has been sync’ed

date

One more patching work to do is to create an empty “/etc/localtime” file so that you can link your OS X’s localtime file to Docker VM’s localtime file at runtime:

sudo touch /etc/localtime

Exit the SSH session:

exit

This issue is being tracked here:

https://github.com/boot2docker/boot2docker/issues/476

Screen Shot 2014-09-14 at 10.53.04 PM

Step 4. Launch Eucalyptus Console via Docker

Run the command below to launch Eucalyptus Console on Docker:

docker run -i -t -v /etc/localtime:/etc/localtime:ro -p 8888:8888 kyolee310/eucaconsole:package-4.0 bash

It’s a shame, but running a live “bash” session is not Docker’s way of doing things, but excuse me for the moment until I figure out how to run Eucalyptus Console properly without using the “service” command.

The command above will open a bash shell session for the eucaconsole image, then run the command below to launch Eucalyptus Console:

service eucaconsole start

Step 5.  Open Eucalyptus Console on a browser

Run the command below to find out the IP of Docker VM:

boot2docker ip

, whose output would look like:

    The VM’s Host only interface IP address is: 192.168.59.103

Using the IP above, access Eucalyptus Console at port 8888:

ex. http://192.168.59.103:8888/

Screen Shot 2014-09-14 at 8.54.09 PM

In order to access AWS, you will need to obtain your AWS access key and secret key. Here is the link by AWS on howto:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html

Step 6. Log into AWS using Eucalyptus Console

Screen Shot 2014-09-14 at 8.54.28 PM

Advertisements

A Developer’s Story on How Eucalyptus Saved the Day

This is a short story on how a UI developer at Eucalyptus was able to use Eucalyptus to save his time on development of Eucalyptus.

travis-mascot-200px

The agenda of the day is to set up Travis CI for the latest Eucalyptus user console, Koala. Quoted from Wikipedia, “Travis CI is a hosted, distributed continuous integration service used to build and test projects hosted at GitHub.” In other words, we want to set up an automated service hook on Koala’s GitHub repository so that whenever developers commit new code, “auto-magic” takes places somewhere in the Internet, which ensures that the developers did not screw things up by mistake, which, in turn, allows us developers to sit back and enjoy a warm cup of “post-commit-victory” tea while our thoughts are drifting away on the sea of Reddit.

But, before arriving at such Utopia, first things first. Must read the instructions on Travis CI.

Luckily, Travis CI put together a nice and comforting set of documentations on how to hook a project on Travis CI (http://about.travis-ci.org/docs/user/getting-started/) as well as its impressive, pain-free registration interface. Things are as easy as clicking buttons for the first few steps so far.

And, of course, nothing ever comes that easy. Now I am looking at the part where I need to create the XML configuration file for Koala’s build procedures. Done with the button-clicking. Time to put down the cup of tea because now I got some reading and thinking to do.

A few minutes after, I am stunned by the line below:

“Travis CI virtual machines are based on Ubuntu 12.04 LTS Server Edition 64 bit.”

Oh, bummer.

The main development platform for Koala has been on CentOS 6, meaning Koala’s build dependencies and scripts have been targeted toward running on CentOS 6 environment. It means that I need to go over the build procedures and dependency setting so that Koala can be built and tested on Ubuntu 12.04. But, first, where do I find those Ubuntu machines?

Then, the realization, ‘Wait a second here. I have Eucalyptus.’

I open up a browser and log into the Eucalyptus system for which I have been using as backend to develop Koala. I launch a couple of Ubuntu 12.04 instances. Within a minute, I have 2 fresh instances of Ubuntu 12.04 virtual machines up and running.

Immediately I log in to the first instance and start installing Koala to validate the build procedures on Ubuntu 12.04 environment. Along the way, I discover various little issues in this new environment and tweak things around to fine-tune Koala’s build procedures. Once felt ready, I log into the other Ubuntu instance to verify the newly adjusted build procedures under its fresh setting. More mistakes and issues are captured, and more adjustments are made. Meanwhile, the first instance has been shut down and a new Ubuntu instance has been brought up. With this new instance, I am able to rinse and repeat the validation of the build procedures. Of course, there are some mistakes again. They get fixed and adjusted. Meanwhile, another instance goes down and comes up fresh.

The juggling of the instances lasts a couple more times until the build procedures are perfected. Now I am confident that Koala will build successfully on Ubuntu 12.04 environment. Commit the new build XML script to GitHub. It’s time for the warm cup of “post-commit-victory” tea.

images-tea

[Cloud Application] Run Eucalyptus UI Tester on your Mac using Vagrant

Eucalyptus User Console

Initially, this blog was written to be a technical blog that describes the instructions on how to run Eucalyptus UI Tester (se34euca) on your Mac using Vagrant and Virtual Box. However, writing this blog has made me reiterate the benefits on running or developing applications on a virtual machine.

Background: Automated Tester As An Application (ATAAA)

When developing software, there is a need for having an automated test suite readily available; with the click of a button, a developer should be able to run a sequence of automated tests to perform a speedy sanity check on the code that is being worked on.

Traditionally, a couple of in-house machines would be dedicated to serve as automated testers, shared by all developers. In such setup, it would require the developers to interact with the tester machines over VPN, which can get quite hectic sometimes — especially for those developers who like to hang out at a local coffee shop.

Now, with Vagrant and Virtual Box, you can have your own personal automated tester as a “cloud application” running on a laptop. In this scenario, when the code is ready for testing, you can quickly run a set of automated tests on your laptop by launching a virtual image that has been pre-configured to be the automated tester for the project/software. When finished, the virtual instance can be killed immediately to free up the resources on the laptop.

Screen shot 2013-07-09 at 9.46.04 PM

Benefits of Running Applications on a Virtual Instance

As mentioned in the introduction, while preparing this Eucalyptus UI Tester to run as a cloud application, I rediscovered the appreciation for using virtual machines as part of the software development environment. The fact that the application runs on a virtual image brings the following benefits: contain-ability, snapshot-ability, and portability of the application.

1. Contain-ability

Running the application on a virtual instance means that no matter how messy dependencies the application requires, they all get to be installed on a contained virtual environment. This means that you get to keep your precious laptop clean and tidy, protecting it from all those unwanted unstable, experimental packages.

2. Snapshot-ability

When working with a virtual instance, at some point, you should be able to stabilize the application, polish it up to be a known state, and take a snapshot of the virtual image in order to freeze up the moment. Once the snapshot is taken and preserved, you have the ability to bring the application back to the such known state at any time. It’s just like having a time machine.

groundhogday

3. Portability

When working with a team or a community, the portability of the application on a virtual image might be the most appealing benefit of all. Once you polish up the application to run nicely on a virtual image, then the promise is that it will also run smoothly on any other virtual machines out there — including on your fellow developers’ laptops as well as on the massive server farms in a data center, or in the cloud somewhere. Truly your application becomes “write once, run everywhere.”

Screen shot 2013-07-09 at 9.47.51 PM

Running Eucalyptus UI Tester on Your Mac Laptop via Vagrant

If you would like to run Eucalyptus UI Tester from scratch, follow the steps below:

1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps

and

2. Installing Eucalyptus UI Tester on CentOS 6 image via Vagrant

If you would like to run Eucalyptus UI Tester from the pre-baked Vagrant image, follow the steps below:

1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps

then

3. Running PreBaked Eucalyptus UI Tester Image using Vagrant

, and see 4. Creating a New Vagrant Package Image if you are interested in creating a new image via Vagrant.

Instructions

1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps

https://github.com/eucalyptus/se34euca/wiki/Installing-Virtual-Box-and-Vagrant-on-Mac-OS-X

2. Installing Eucalyptus UI Tester on CentOS 6 image via Vagrant

https://github.com/eucalyptus/se34euca/wiki/Installing-se34euca-on-Centos-6

3. Running PreBaked Eucalyptus UI Tester Image using Vagrant

https://github.com/eucalyptus/se34euca/wiki/Running-PreBaked-se34euca-Image-using-Vagrant

4. Creating a New Vagrant Package Image

https://github.com/eucalyptus/se34euca/wiki/Creating-a-New-Vagrant-Package-Image

euca_new_logo

Introducing Metaleuca

Nuclear-Devil-Horns

What is Metaleuca?

Metaleuca is a bare-metal provision management system that interacts with open-source software Cobbler via EC2-like CLI.

Using Metaleuca, users can communicate with Cobbler to self-provision a group of bare-metal machines to boot up with new, fresh OS images. The main appeal of Metaleuca is that it allows users to manage the bare-metal machines like EC2’s virtual instances via the command-lines that feel much like ec2-tools, or euca2ools.

euca_new_logo

ec2_logocobbler_logo

Metaleuca Command-Line Tools

Metaleuca consists of a set of command-lines that mirror some of the command-lines in ec2-tool or euca2ools. The list below shows a number of the core command-lines used in Metaleuca:

  • metaleuca-describe-profiles – Describe all the profiles provided in Cobbler
  • metaleuca-describe-systems – Describe all the bare-metal systems registered in Cobbler
  • metaleuca-reboot-system – Reboot the selected bare-metal system
  • metaleuca-run-instances – Initiate the provision sequence on the selected bare-metal systems
  • metaleuca-describe-instances – Describe the statuses of the provisioned bare-metal systems
  • metaleuca-terminate-instances – Terminate the bare-metal systems, returning them back to the resource pool

Metaleuca Configuration

Prior to installing Metaleuca, it is required that you have already configured Cobbler to provision a group of bare-metal machines in your datacenter. If you are new to Cobbler, please visit the Cobbler’s homepage http://cobbler.github.com/ for more information on how to set up Cobbler.

Once you have Cobbler running in the datacenter, you will need to install Metaleuca on a Ubuntu machine, or virtual machine. The installation guide for Metaleuca is provided at:

https://github.com/eucalyptus/metaleuca/wiki/Metaleuca-Installation-Guide

Metaleuca Walkthrough

Now that you have Metaleuca configured in the datacenter, let’s go over a scenario where you will want to launch two bare-metal instances with fresh CentOS 6.3 images.

First, you might want to use the command “metaleuca-describe-systems” to survey all the available systems registered in Cobbler.

Picture 3.png 03-30-32-133

Metaleuca allows users to directly select which bare-metal machines to provision — by using the machines’ IPs. However, those who are familiar with AWS will contest that this approach is not how virtual machines are provisioned on EC2; rather than specifying the IPs, the AWS users are to provide the number of instances to launch. For this reason, here, we will cover the EC2’s approach to provision the instances.

In Metaleuca, first you will need to find out which ‘profile‘ is set to install the CentOS 6.3 image. In Cobbler, a profile maps to a preconfigured ‘kickstart‘ file that contains the netboot instructions on which OS to install when a machine boots up and initiates PXEBOOT. In other words, you may compare the profiles on Cobbler to the instance images on EC2. In Metaleuca, you can display the available profiles using the command ‘metaleuca-describe-profiles‘:

Picture 7

Let’s say that the profile “qa-centos6u3-x86_64-striped-drives” is what we want to use.

Next, you will want to determine which “system-group” you want the machines to be selected from. In Metaleuca, the bare metal machines can be grouped into different resource pools. For instance, in our QA system at Eucalyptus, which utilizes Metaleuca, we partitioned the machines in the datacenter into 6 groups: qa00, qa01, dev00, dev01, test00, and test01. Such grouping allows us to provide semantics on the machine pools based on their usages, in which we created the resource allocation policy for the users, who are mainly developers and QA engineers. The command “metaleuca-describe-system-groups” displays all the machines and the groups accordingly:

Picture 8

However, keep in mind that not all machines in the list will be available to be provisioned; some of the machines might be in use by other users. Thus, you will want to run the command “metaleuca-describe-system-groups -f” to discover which machines are free to use. Fortunately, at this moment, among those 6 system-groups mentioned above, the group ‘test01′ has 2 machines available, which are labeled as “FREED” in the screen shot below:

Picture 6

Once you figure out the profile and the system-group availability information, you are ready to provision the bare-metal machines. The command “metaleuca-run-instances” takes input of how many instances you want to launch, which profile to use, which group to call from, and finally, the user string to mark the machines:

./metaleuca-run-instances -n 2 -g test01 -p qa-centos6u3-x86_64-striped-drives -u kyo_machines_for_demo

Picture 10

And, similar to ec2-tools and euca2ools, you can monitor the progress of the provisioned machines using the command “metaleuca-describe-instances -u“:

Picture 11

Notice that the instances are at the “pending” state at the moment. Soon, in about 8 minutes after launching, the instances will be shown as “running”:

Picture 12

At that point, you may ssh into the bare metal machines to verify that they are up and running with the fresh CentOS 6.3 OS installed.

Later, the command “metaleuca-describe-system-user -u” comes in handy when you want to find out which machines are provisioned under your name:

Picture 13

When you are done with the machines, you may “free” the machines so that they can return to the resource pool; the command for that case is “metaleuca-terminate-instances -u“:Picture 14

When running the command “metaleuca-describe-instance -u“, you will notice that the machines have been successfully freed:

Picture 15

Metaleuca as an Open Source Project

BARE METAL FOIL2

Metaleuca was evolved out of the internal usage — the development and test environment for engineers — at Eucalyptus, and is now available as an open source project on Github under the Apache license. The goal of the project is to complete the integration of Metaleuca into the Eucalyptus system so that it can be served as a “bare-metal only” zone in Eucalyptus. Your contribution is much appreciated!

Check out the project at:

https://github.com/eucalyptus/metaleuca

\m/

Beyond Continuous Integration: Locking Steps with Dev, QA, and Release

Continuous integration: the practice of frequently integrating one’s new or changed code with the existing code repository [wikipedia]

In this blog we will talk about how the continuous integration process was put in place for the new component, Eucalyptus User Console, in order to collaborate the efforts among the dev, QA and release teams throughout the development cycle of Eucalyptus 3.2.

Backgrounduserconsoleconponentview

Eucalyptus User Console is a newly introduced component in Eucalyptus, whose main goal is to provide an easy-to-use, intuitive browser-based interface to the cloud users, thus assisting in the dev/test cloud deployments among IT organizations and enterprises. Eucalyptus User Console consists of two components: javascript-based client-side application and Tornado-based user console proxy server.

Early Involvement

The first phase of the development was to come up with a quick prototype to demonstrate how the user console would work under the given initial design of the architecture (see the Eucalyptus Console components layout diagram above). As soon as the prototype was evaluated and its feasibility was verified, the release team started creating the packages for two major Linux OS platforms: Ubuntu and Centos/RHEL.

The early involvement of the release team turned out to be the best help any developers or QA engineers could ask for; since the very beginning stage of the development, the release team was able to provide invaluable information that served as guardrail for the fast-moving development. Such information included advising on how the files should be named and organized and identifying which dependencies should or should not be used in order to meet the requirements for various Linux distributions. Dealing with such issues at the later stage of the development would have been undoubtedly a major pain in the back-end.

jenkins_logo

Further more, the release team was able to ensure that the development of the new user console would never go off the track against the Linux distro requirements by setting up the automated daily package-building process using Jenkins — which utilizes the VM resources from our Release cloud that runs on Eucalyptus.

Keeping Up With Eucalyptus

Setting up the automated process to build the packages would allow the release team to keep an eye on the progress of the user console’s development in terms of the ability to build the packages according to the constraints set by the Linux distributions. However, it would not guarantee whether the newly built packages contain the version of the user console that works with the current, up-to-date Eucalyptus cloud that was also in development.

Thus, the challenge was to ensure that the latest built user console packages work with the latest built Eucalyptus throughout the development.

In order to solve this issue, the QA team created a testunit that automatically installs the latest user console packages on a newly built Eucalyptus. Then, the testunit was added to the main test sequences used by the Eucalyptus 3.2 development in our automated QA system, making the installation of the latest user console packages accessible by all developers at Eucalyptus.

This setup encouraged a failure in the user console package installation to be seen by any developers throughout the development, thus allowing the failure to be detected fast and reported with quickness.

Screen shot 2012-12-10 at 5.50.02 AM

The testunit ui_setup can be seen in action above in the table which displays the results of the test sequence ran by the automated QA system.. Check out the link below for more details of this testunit:

https://github.com/eucalyptus-qa/ui_setup

Circle of Trust

As the user console evolved out of its prototype state and took the form of a more product-like shape, the QA team was working in parallel, figuring out how to set up the automated testing process for the user console. The blog here talks in detail about how Selenium was used to create the automated web-browser testing tools, se34euca.

big-logo

In the mid-stage of the development, as the features of the user console started functioning in reasonably stable manners, 3 automated tests were added — incrementally — to ensure that the working state of the user console throughout the development.

Screen shot 2012-12-10 at 6.41.28 AMThose 3 tests are:

  1. user_console_view_page_testhttps://github.com/eucalyptus-qa/user_console_view_page_test
  2. user_console_generate_keypair_testhttps://github.com/eucalyptus-qa/user_console_generate_keypair_test
  3. user_console_launch_instance_test https://github.com/eucalyptus-qa/user_console_launch_instance_test

These automated tests were to ask the 3 simple questions below on a daily basis:

  1. Can the user log in and see all the landing pages on the latest user console?
  2. Can the user generate a new keypair using the latest user console?
  3. Can the user launch a VM instance using the latest user console?

Of course, it would be possible, and desirable, to ask more questions in a more complicated fashion. However, during the rapid development phase, asking those 3 simple questions on a daily basis, turned out to be sufficient, and effective, to understand whether something terrible had happened to the user console or not.

traffic_light

The goal of these automated tests at this stage of the development was not to detect every little defect in the product. Not too soon at the moment.

The main purpose is rather to serve as an indicator for the developers, QA engineers, and release engineers to assure ourselves that the change that went in the code earlier today did not ruin the delicate trust among the three groups, meaning that the build, installation, and configuration procedures are still in tact. Having such assurance in check by mechanical means has made the three groups extremely effective in discovering issues during the development since it allowed each member to narrow down exactly what was responsible for the defects in a finely reduced time frame, which was in hours, rather than days or weeks.

Guardrail For Development

Having the automated package build process and the automated installation/configuration process in place at the early stage of the development was proven to be extremely useful; rather than agreeing on the written procedures, the dev, QA, and release team materialized such agreements into the actual implementation, and put them into work by using various automated mechanics that run on a daily basis. Therefore, throughout the development, we were able to witness and assure ourselves that we were making progress in accordance with the plan and our self-imposed restrictions.

Check out the Eucalyptus Open QA webpage to see the continuous integration at Eucalyptus in action:

Eucalyptus Open QA (beta) – http://ec2-50-112-61-121.us-west-2.compute.amazonaws.com/open_qa.php

10 Steps to Euca Monkey

Euca Monkey is an easy-to-deploy test tool designed for performing stress-test on Eucalyptus Cloud. The tool repeatedly generates and tears down 6 types of cloud user resources: running instances, volumes, snapshots, security groups, keypairs, and IP addresses. As the resources are being populated and released, the cloud is actively queried to validate such resources are indeed being allocated correctly per request. Then, the tool renders the progress of the stress testing using Gnuplot, an open source graphing tool, and displays the graphs as webservice in real time.

Euca Monkey uses cloud-resource-populator — which utilizes Eutester, which is based on Boto, thus making the tool AWS-API-compatible  — to populate and release resources from Eucalyptus as a user. The input for cloud-resource-populator looks below:

[USER INFO]
account: ui-test-acct-23
user: user-23
password: mypassword23
[RESOURCES]
running instances: 2
volumes:  2
snapshots: 1
security groups: 10
keypairs: 3
ip addresses: 2
[ITERATIONS]
iterations: 200

With the given input above, cloud-resource-populator will generate the resources as specified in the [RESOURCES] section as the user ‘user-23′ under the account ‘ui-test-acct-23′. When viewed from Eucalyptus user console, it will look as below:

As soon as the resources are populated according to the specification, cloud-resource-populator will immediately send requests to the cloud to release all the allocated resources, which makes the console view look as below:

And, as you would have guessed, the process of populating and releasing of the resources is repeated for [ITERATIONS] times.

The most appealing feature of Euca Monkey is that it launches a webservice to render the progress of the stress-testing in real time.

The graph above is showing the input values of a few iterations of the resource population and tear-down process. When tracing the running instance line, which is in red, this graph is telling us that 20 instance were started on the first request, then those 20 instances were all terminated on the second mark, thus brought the count down to 0. And, 20 instances were started again, then terminated, and so on. Such operations were repeated 7 times on the graph above. And, also notice that there are 4 other resources being populated and released 7 times as well.

When the cloud is behaving nicely, the actual resources should be populated and released in match with the input values in the graph above, thus resulting in the output graph as below:

While the first graph shown above renders the input values to the tool cloud-resource-populator, this graph is showing the actual values reported by the cloud. The fact that these two graphs look the same means “Yay Cloud!!”

However, occasionally, during the development of Eucalyptus, you would see the graph like below on a rainy day in Santa Barbara:

The graph above reveals an interesting state of the cloud. Notice that the “running instance” line went from the “nice” behavior pattern to a flat line. It means that the cloud was able to launch and terminate 20 instances during the first phase, but somehow it got stuck to the state where it was not able to release instances, nor launch more instances, thus stuck with 18 running instances. But, notice that other resources were making the usual progress as before, except the security groups. It turned out what we were witnessing was the deadlock case in Eucalyptus Cluster Controller, which occurred around the 8th hour of the stress-testing. R.I.P CC. 😦

The purpose of stress-testing is to push the limit of the system to the point where malfunctions and faulty behaviors of the system can be observed. Such stress testing is crucial for the development of a distributed system like Eucalyptus; many unknowns and bugs are constantly introduced to the system as new features from various components are being integrated. Thus, having an easily deployable stress-testing tool with visualization support, such as Euca Monkey, yields tremendous benefits for the developers in an agile development environment since the tool aims to ensure the system’s stability and reliability throughout the rapid development cycle.

If you would like to take Euca Monkey for a spin, feel free to check out the GitHub link below. On a fresh Centos 6 machine or VM, it will take 10 simple steps to launch your own monkey.

https://github.com/eucalyptus/euca-monkey

Other Resources:

Eutester – https://github.com/eucalyptus/eutester

Boto – https://github.com/boto/boto

Eucalyptus QA – https://github.com/eucalyptus-qa

Open QA for Eucalyptus – https://github.com/eucalyptus/open-qa

Implication of Fragmentation in Linux

At Eucalyptus we have been proud to say,

“Eucalyptus Cloud runs on almost all major Linux distros: Ubuntu, Debian, CentOS, Red Hat, Fedora, etc.. You name it, we will support it!”

The crowd erupts in cheers, with occasional tears of joy. There will be a parade later.

Yes, it sounds wonderfully majestic as it should be.

When developing open-source software, you need to support all major, stable Linux platforms so that the software can reach out to every single open-source enthusiast who is often loyal to a certain flavor of Linux.

But, what does all this mean to software developers?

A nightmare.

A nightmare accompanied with a horrible migraine.

In the ideal software utopia, all Linux platforms behave in an identical manner; you should be able to run things on Fedora in the same exact way you run those things on Ubuntu. After all, they are all “Linux”, aren’t they?

Welcome to the harsh reality called “fragmentation” in the software world.

It is true that all those distros feel like Linux; they share the same core — Linux kernel — and provide the same level of abstraction which can be described as “Linux experience.”

But, in reality, no two Linux distros are never the same.

The biggest problem with this inconvenient truth is that no one knows for sure what the exact differences are when going from one distro to another distro.

Let’s say we use CentOS as a default Linux distro for developing and testing software. If every function and feature works well on CentOS, can we safely assume that the software will also behave nicely with other Linux distros, such as Red Hat, Ubuntu, and Debian?

The answers to this simple question always fall somewhere in between “Maybe”, “It depends”, “Possibly yes”, “Theoretically it should”, “What is Linux?”, and “NO!”

The only assured way to discover the correct answer is to run tests on the software under all distros.

At present Eucalyptus officially supports three main Linux distros — Ubuntu, Redhat, and CentOS — and is working on adding two more distros: Debian and Fedora. Every time a new distro is added, we are to repeat the entire set of test suite for the new distro. If running a whole set of test suite takes X amount of resources, supporting 5 distros would mean 5X resources. Even further, per distro, we support two of the latest versions — for instance, for CentOS, Eucalyptus supports its version 5 and 6. This additional requirement brings up the total amount testing resources to be 10X.

When translated to the operation cost, if running a complete set of test suite under one Linux environment takes one day, due to the fragmentation of Linux distros, we need to add nine more days of testing in order to completely cover all corner cases in various Linux distros.

The real world implication of this nightmare is that whenever a little tweak goes into Eucalyptus, it might take up to 10 days, in the worst case scenario, to ensure ourselves that this seemingly innocent tweak will not bring down the house under some other Linux distros in strange, unpredictable ways.

In school we are taught to celebrate diversity, but they often forget to emphasize the beauty in simplicity. Handling the issue of the Linux fragmentation remains to be one of many challenges that Eucalyptus has to overcome.

%d bloggers like this: