Kyo Lee

Open-Source Cloud Blog

JavaScript Unit Test: Steps toward more perfect code

Why unit test?

Given the event-driven nature of javascript front-end architecture, over time it is inevitable that the project will slowly spiral into ever greater “event handler” entropy — to a point that even the Wise Ones would scratch their heads wondering how the update of a seemingly trivial object in one component had possibly resulted in the creation of the Universe in another.


Since the beginning, developers have tried to prevent such nightmare scenarios by writing detailed comments in attempt to explain entangled relationships among the events. However, when event dependencies are spread among multiple modules, what suppose to be helpful comments often turn into a lengthy short story — deciding on where to maintain such information itself becomes a challenge.

For those who are not fully aware of the seriousness of this problem, here is a simple scenario: Let’s look at an event chain A -> B -> C. When an object gets updated, it would trigger the event A, which in turn broadcasts a signal that wakes up the event B. Then, the event B updates a variable that the event C was keeping eye on, thus invoking the event C. Now you can guess how deep the rabbit hole could possibly go from here. And, it should be considered lucky if the chain was only linear — often you end up dealing with a chain of events that seems like an episode of CSI.


With the scenario above, let’s bring in new, innocent developers who fail to recognize the Holiness that is A -> B -> C. Now, unknowingly the chain is broken, and this critical mistake slips away unnoticed; it turns out that the triggering of the event C was conditional. After a month has gone by, the problem surfaces. Everyone is clueless and devastated. The Holiness was never to be broken. It was clearly written in the comments. How could it be?

This is when unit test comes in and saves the project.

The original author could easily prevent the event chain A -> B -> C from derailing by writing a set of unit test to secure the event flow A -> B -> C. If any changes are made in the code that interfere the event flow, then the unit test would raise the red flag immediately and warn the developer about the consequence of the recent changes. With automated test runners, such as Karma, feedback can take place almost instantly as soon as the code is edited — Karma runs in the background on the developer’s machine, keeping an eye on the javascript files, and triggers unit test automatically upon the detection of the update in the files.


If comments and documentations are to be served as the source of knowledge, unit test works as an enforcer that actively imposes a set of rules defined in the previous development. You can imagine unit test being “live comments working in action”.


How to Write Jasmine Unit-test for Angular Module

Basic Jasmine Unit-test Setup

A simple Jasmine unit-test setup consists of describe() and it() blocks:

describe("<test category>", function() {
    it("<literal description of test>", function() {
        // test procedure comes here

The list of the Jasmine native matchers can be found in Jasmine 2.0 Documentation

Angular Mock and beforeEach()

See below Jasmine unit-test sample code for setting up Angular mock module in beforeEach(). The function beforeEach() will be called before each function it() is run within describe().

Jasmine Unit-test:

describe("SecurityGroupRules", function() {

    var scope, httpBackend, ctrl;
    beforeEach(angular.mock.inject(function($rootScope, $httpBackend, $controller) {
        scope = $rootScope.$new();
        httpBackend = $httpBackend;
        ctrl = $controller('SecurityGroupRulesCtrl', {
            $scope: scope

    describe("Initial Values Test", function() {
        it("Initial value of isRuleNotComplete is true", function() {

spyOn() and toHaveBeenCalled()

Notice the Angular function resetValues() below contains a function call cleanupSelections(). If you want to write a unit-test to ensure that the function cleanupSelections() gets executed whenever resetValues() is invoked, Jasmine’s spyOn() and toHaveBeenCalled() can be used as below:

Angular Module:

$scope.resetValues = function () {

Jasmine Unit-test:

describe("Function resetValues() Test", function() {

    it("Should call cleanupSelections() after resetting values", function() {
        spyOn(scope, 'cleanupSelections');

setFixtures() – Template Loading

Not only ensuring function procedures, unit-test can also be used to prevent critical elements on a template from being altered.

The beforeEach() block below shows how to load a template before running unit-test. The template will be loaded onto PhantomJS‘s environment so that a jQuery call, such as $('#inbound-rules-tab'), can be called to grab the static element on the template.

Jasmine Unit-test:

beforeEach(function() {
    var template = window.__html__['templates/panels/'];
    // remove <script src> and <link> tags to avoid phantomJS error
    template = template.replace(/script src/g, "script ignore_src"); 
    template = template.replace(/\<link/g, "\<ignore_link"); 

describe("Template Label Test", function() {

    it("Should #inbound-rules-tab link be labeled 'Inbound'", function() {

Notice above that template.replace() lines update the template’s elements to disable<script src=""></script> and <link></link>. When the template is loaded ontoPhantomJS, PhantomJS tries to continue loading other JS or CSS files appeared on the template. The loading of such files becomes an issue if their locations are not properly provided in the template — for instance, the files contain dynamic paths, then PhantomJS results in error since it will not be able to locate the files. A workaround for this issue is to disable <script> and <link> elements on the template and, instead, load such files directly using the karma configuration list karma.conf.js.

Template is required

In some cases, when a function contains calls that interact with elements on the template, then you will have to provide the template so that the function call can complete without error. For instance, the function cleanupSelections below contains jQuery calls in the middle of procedure. Without the template provided, the function will not be able to complete the execution since those jQuery lines will error out.

Angular Module:

$scope.cleanupSelections = function () {
    if( $('#ip-protocol-select').children('option').first().html() == '' ){
    // Section needs to be tested

setFixtures() – Direct HTML Fixtures

In some situations, the static elements provided by the template will be not satisfy the needed condition for testing the function. For instance, the functiongetInstanceVPCName below expects the select element vpc_network to be populated with options. In a real scenario, the options will be populated by AJAX calls on load — mocking such AJAX calls is described in the section below. However, if the intention is to limit the scope of testing for this specific function only, then you could directly provide the necessary HTML content in order to simulate the populated select options as seen in the setFixtures() call below:

Angular Module:

 $scope.getInstanceVPCName = function (vpcID) {
     var vpcOptions = $('#vpc_network').find('option');
     vpcOptions.each(function() {
         if (this.value == vpcID) {
             $scope.instanceVPCName = this.text;

Jasmine Unit-test:

beforeEach(function() {
    setFixtures('<select id="vpc_network">\
        <option value="vpc-12345678">VPC-01</option>\
        <option value="vpc-12345679">VPC-02</option>\

it("Should update instanceVPCName when getInstanceVPCName is called", function() {

Mock HTTP Backend

When writing unit-test for Angular modules, often it becomes necessary to simulate the interaction with the backend server. In that case, $httpBackend module can be used to set up the responses from the backend server for predetermined AJAX calls.

Angular Module:

    $scope.getAllSecurityGroups = function (vpc) {
        var csrf_token = $('#csrf_token').val();
        var data = "csrf_token=" + csrf_token + "&vpc_id=" + vpc;
            method:'POST', url:$scope.securityGroupJsonEndpoint, data:data,
            headers: {'Content-Type': 'application/x-www-form-urlencoded'}
        }).success(function(oData) {
            var results = oData ? oData.results : [];
            $scope.securityGroupCollection = results;

Jasmine Unit-test:

describe("Function getAllSecurityGroups Test", function() {

    var vpc = 'vpc-12345678';

    beforeEach(function() {
        setFixtures('<input id="csrf_token" name="csrf_token" type="hidden" value="2a06f17d6872143ed806a695caa5e5701a127ade">');
        var jsonEndpoint  = "securitygroup_json";
        var data = 'csrf_token=2a06f17d6872143ed806a695caa5e5701a127ade&vpc_id=' + vpc 
        httpBackend.expect('POST', jsonEndpoint, data)
            .respond(200, {
                "success": true,
                "results": ["SSH", "HTTP", "HTTPS"]

    afterEach(function() {

    it("Should have securityGroupCollection[] initialized after getAllSecurityGroups() is successful", function() {
        scope.securityGroupJsonEndpoint = "securitygroup_json";

Also notice how setFixtures() is used in beforeEach() to prepare for the jQueryline var csrf_token = $('#csrf_token').val(); in the functiongetAllSecurityGroups().

Angular $watch test

$watch() is one of the most frequently used functions in Angular, which triggers events when it detects update in the watched object. When you need $watch() function to react in unit-test, you could call $apply() to have the latest update to be detected by the Angular module.

Angular Module:

$scope.setWatchers = function () {
    $scope.$watch('securityGroupVPC', function() {

Jasmine Unit-test:

it("Should call getAllSecurityGroupVPC when securityGroupVPC is updated", function() {
    spyOn(scope, 'getAllSecurityGroups');
    scope.securityGroupVPC = "vpc-12345678";

Angular $on test

In Angular,$on() is used to detect any broadcast signal from other Angular modules. For testing such setup, you could directly send out the signal by using $broadcast()call.

Angular Module:

$scope.setWatchers = function () {
    $scope.$on('updateVPC', function($event, vpc) {
        $scope.securityGroupVPC = vpc;

Jasmine Unit-test:

it("Should update securityGroupVPC when updateVPC is called", function() {
    scope.$broadcast('updateVPC', 'vpc-12345678');

Angular $broadcast test

Paired with $on(), you would also want to write unit-test for ensuring the$broadcast() call’s condition. For such purpose, spyOn() andtoHaveBeenCalledWith() setup can be used on $broadcast() to check for its proper signal signatures.

Angular Module:

$scope.setWatcher = function () {
    $scope.$watch('securityGroupVPC', function () {
        $scope.$broadcast('updateVPC', $scope.securityGroupVPC);

Jasmine Unit-test:

it("Should broadcast updateVPC when securityGroupVPC is updated", function() {
    spyOn(scope, '$broadcast');
    scope.securityGroupVPC = 'vpc-12345678';
    expect(scope.$broadcast).toHaveBeenCalledWith('updateVPC', scope.securityGroupVPC);

For more detailed wiki on how to write Jasmine Unit Test for Angular Module, check out the link below:



Docker Strange Dev: How I learned to stop worrying and love the VM

A couple of months back, my laptop died. I recall the day I brought it home for the first time; still in college, making a big decision to go with the top model at the time, thinking, “Well, it should last at least 5 years.”  — it lasted 7 years.  7 years later, a certified Apple repair person told me, “Well, you should just buy a new laptop than trying to fix this one.” So I did.

Now I have a new laptop, which is currently placed on a pedestal. How fast applications open and close on this laptop makes me cry with tears of joy. The future has arrived. I keep telling myself, “No, I will not f*** this one up this time. I will never install “software” on this laptop ever!” I am in denial of the fact that this pure awesomeness will inevitably start decaying at an exponential rate in relation to the number of new software installed and run on the laptop [Disclaimer: there exist no supporting data, but paranoia].

This is when I tried this Docker thing, in a desperate search of finding the promise land of “install no software.”


The noticeable difference in using Docker, which utilizes Linux containers, vs. using direct virtual machines approach is the speed.

Both options provide the ability to isolate software’s running environment away from the base operating system, thus delivering ever-desirable “build once and run everywhere” paradigm and addressing my paranoia that the system that provides the base OS must remain minimal and untouched. However, what makes Docker stand out from the classic VM approach is that now the cloud application (a virtual image + installed software) will run within milliseconds. Try to compare that to the pre-historic requirement of taking “minutes” of time to boot a virtual instance.


How significant does this difference make?

Docker removes a big chunk of mental roadblock for developers. With Docker, there exists no perceptual distinction between running applications on the base OS and on a virtual image (a virtual container, to be more precise) because the developers will not be able to tell the difference when it comes to the response time. It’s not only the response time, but the biggest differentiator comes from the fact that Docker strips away the tedious procedures of booting VM instances and managing the instance’s life cycle, which is not the forte of the developers. Unlike other VM-centric approaches, Docker embraces its application-centric design principal. For instance, Docker’s CLI below asks the user two simple questions:

1. Which image to use?

2. Which command to run?

docker run [options] <image> <command>

Using Docker, I can just type away a single line command to run a cloud application, as I would with any other Linux commands, then I will have my database servers running, web servers running, API servers running, file servers running. etc — all be done in single lines. The entire distributed web-stack can now be running on my laptop within seconds. The best part of all this? When I’m done, there will be no trace left on my laptop — as if they never existed. The pure awesomeness prevails.


Running Eucalyptus Console on Docker


For those who want to check out Eucalyptus Console to access Amazon Web Services, here are the steps to launch Eucalyptus Console using Docker on OS X.

Step 1. Install Docker on your laptop

Here is a great link that walks you through how to install Docker on OS X:

Step 2. Pull Eucalyptus Console Docker image repository

Run the command below to pull Eucalyptus Console Docker images (it will take some time to download about 1.5 G image files):

docker pull kyolee310/eucaconsole

Run the command below to verify that the eucaconsole images have been pulled:

docker images

Screen Shot 2014-09-14 at 10.52.11 PMFor those who want to build the images from scratch, here is the link to the Dockerfile used:

Step 3.  Update Docker VM’s clock

One issue that I faced while running Docker on OS X was that the clock wasn’t sync’ed properly, which was problematic for some applications. In order to fix this issue, you will need to log into the Docker VM and sync the clock manually.

You can SSH into Docker VM using the command below:

boot2docker ssh

Once logged in, run the command below to sync the clock:

sudo ntpclient -s -h

Run the command below to verify that the clock has been sync’ed


One more patching work to do is to create an empty “/etc/localtime” file so that you can link your OS X’s localtime file to Docker VM’s localtime file at runtime:

sudo touch /etc/localtime

Exit the SSH session:


This issue is being tracked here:

Screen Shot 2014-09-14 at 10.53.04 PM

Step 4. Launch Eucalyptus Console via Docker

Run the command below to launch Eucalyptus Console on Docker:

docker run -i -t -v /etc/localtime:/etc/localtime:ro -p 8888:8888 kyolee310/eucaconsole:package-4.0 bash

It’s a shame, but running a live “bash” session is not Docker’s way of doing things, but excuse me for the moment until I figure out how to run Eucalyptus Console properly without using the “service” command.

The command above will open a bash shell session for the eucaconsole image, then run the command below to launch Eucalyptus Console:

service eucaconsole start

Step 5.  Open Eucalyptus Console on a browser

Run the command below to find out the IP of Docker VM:

boot2docker ip

, whose output would look like:

    The VM’s Host only interface IP address is:

Using the IP above, access Eucalyptus Console at port 8888:


Screen Shot 2014-09-14 at 8.54.09 PM

In order to access AWS, you will need to obtain your AWS access key and secret key. Here is the link by AWS on howto:

Step 6. Log into AWS using Eucalyptus Console

Screen Shot 2014-09-14 at 8.54.28 PM

A Developer’s Story on How Eucalyptus Saved the Day

This is a short story on how a UI developer at Eucalyptus was able to use Eucalyptus to save his time on development of Eucalyptus.


The agenda of the day is to set up Travis CI for the latest Eucalyptus user console, Koala. Quoted from Wikipedia, “Travis CI is a hosted, distributed continuous integration service used to build and test projects hosted at GitHub.” In other words, we want to set up an automated service hook on Koala’s GitHub repository so that whenever developers commit new code, “auto-magic” takes places somewhere in the Internet, which ensures that the developers did not screw things up by mistake, which, in turn, allows us developers to sit back and enjoy a warm cup of “post-commit-victory” tea while our thoughts are drifting away on the sea of Reddit.

But, before arriving at such Utopia, first things first. Must read the instructions on Travis CI.

Luckily, Travis CI put together a nice and comforting set of documentations on how to hook a project on Travis CI ( as well as its impressive, pain-free registration interface. Things are as easy as clicking buttons for the first few steps so far.

And, of course, nothing ever comes that easy. Now I am looking at the part where I need to create the XML configuration file for Koala’s build procedures. Done with the button-clicking. Time to put down the cup of tea because now I got some reading and thinking to do.

A few minutes after, I am stunned by the line below:

“Travis CI virtual machines are based on Ubuntu 12.04 LTS Server Edition 64 bit.”

Oh, bummer.

The main development platform for Koala has been on CentOS 6, meaning Koala’s build dependencies and scripts have been targeted toward running on CentOS 6 environment. It means that I need to go over the build procedures and dependency setting so that Koala can be built and tested on Ubuntu 12.04. But, first, where do I find those Ubuntu machines?

Then, the realization, ‘Wait a second here. I have Eucalyptus.’

I open up a browser and log into the Eucalyptus system for which I have been using as backend to develop Koala. I launch a couple of Ubuntu 12.04 instances. Within a minute, I have 2 fresh instances of Ubuntu 12.04 virtual machines up and running.

Immediately I log in to the first instance and start installing Koala to validate the build procedures on Ubuntu 12.04 environment. Along the way, I discover various little issues in this new environment and tweak things around to fine-tune Koala’s build procedures. Once felt ready, I log into the other Ubuntu instance to verify the newly adjusted build procedures under its fresh setting. More mistakes and issues are captured, and more adjustments are made. Meanwhile, the first instance has been shut down and a new Ubuntu instance has been brought up. With this new instance, I am able to rinse and repeat the validation of the build procedures. Of course, there are some mistakes again. They get fixed and adjusted. Meanwhile, another instance goes down and comes up fresh.

The juggling of the instances lasts a couple more times until the build procedures are perfected. Now I am confident that Koala will build successfully on Ubuntu 12.04 environment. Commit the new build XML script to GitHub. It’s time for the warm cup of “post-commit-victory” tea.


Eucalyptus Console is for Developers

Screen shot 2013-09-18 at 9.28.06 PM

One major event taken place in preparation for the upcoming release of Eucalyptus 3.4 is that Eucalyptus Console has found its own nest in GitHub ( and is available as an independent project. This separation aims Eucalyptus Console to be developed and released at a different pace than its much bigger core system, Eucalyptus ( Having faster cycles for the project expects Eucalyptus Console to be able to deliver patches and introduce new features in a much responsive manner.

Another appealing characteristic of Eucalyptus Console for this release is that now it can be used as a stand-alone application — running on a developer’s laptop — to manage the cloud resources on AWS. When provided a pair of an access key and a secret key into Eucalyptus Console’s login window, the console sets the end-point to AWS and allows the user to interact with the AWS cloud under the provided AWS credentials.

For many AWS enthusiasts and developers, this new Eucalyptus Console project on GitHub ( offers an opportunity to develop a personal, customizable AWS management console for specific demands and use cases. Composed of a client-side web interface written in Javascript, using Backbone.js and Rivets.js, and a server-side proxy written in Python, using Boto and Tornado, Eucalyptus Console is designed to deliver the maximum flexibility when it comes to adding new features and adopting other latest web technologies.


Check out the link below for getting the Eucalyptus Console VM image up and running on Vagrant:

If you would rather install Eucalyptus Console from scratch on a CentOS 6 image/machine, check out the link below instead:


[Cloud Application] Run Eucalyptus UI Tester on your Mac using Vagrant

Eucalyptus User Console

Initially, this blog was written to be a technical blog that describes the instructions on how to run Eucalyptus UI Tester (se34euca) on your Mac using Vagrant and Virtual Box. However, writing this blog has made me reiterate the benefits on running or developing applications on a virtual machine.

Background: Automated Tester As An Application (ATAAA)

When developing software, there is a need for having an automated test suite readily available; with the click of a button, a developer should be able to run a sequence of automated tests to perform a speedy sanity check on the code that is being worked on.

Traditionally, a couple of in-house machines would be dedicated to serve as automated testers, shared by all developers. In such setup, it would require the developers to interact with the tester machines over VPN, which can get quite hectic sometimes — especially for those developers who like to hang out at a local coffee shop.

Now, with Vagrant and Virtual Box, you can have your own personal automated tester as a “cloud application” running on a laptop. In this scenario, when the code is ready for testing, you can quickly run a set of automated tests on your laptop by launching a virtual image that has been pre-configured to be the automated tester for the project/software. When finished, the virtual instance can be killed immediately to free up the resources on the laptop.

Screen shot 2013-07-09 at 9.46.04 PM

Benefits of Running Applications on a Virtual Instance

As mentioned in the introduction, while preparing this Eucalyptus UI Tester to run as a cloud application, I rediscovered the appreciation for using virtual machines as part of the software development environment. The fact that the application runs on a virtual image brings the following benefits: contain-ability, snapshot-ability, and portability of the application.

1. Contain-ability

Running the application on a virtual instance means that no matter how messy dependencies the application requires, they all get to be installed on a contained virtual environment. This means that you get to keep your precious laptop clean and tidy, protecting it from all those unwanted unstable, experimental packages.

2. Snapshot-ability

When working with a virtual instance, at some point, you should be able to stabilize the application, polish it up to be a known state, and take a snapshot of the virtual image in order to freeze up the moment. Once the snapshot is taken and preserved, you have the ability to bring the application back to the such known state at any time. It’s just like having a time machine.


3. Portability

When working with a team or a community, the portability of the application on a virtual image might be the most appealing benefit of all. Once you polish up the application to run nicely on a virtual image, then the promise is that it will also run smoothly on any other virtual machines out there — including on your fellow developers’ laptops as well as on the massive server farms in a data center, or in the cloud somewhere. Truly your application becomes “write once, run everywhere.”

Screen shot 2013-07-09 at 9.47.51 PM

Running Eucalyptus UI Tester on Your Mac Laptop via Vagrant

If you would like to run Eucalyptus UI Tester from scratch, follow the steps below:

1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps


2. Installing Eucalyptus UI Tester on CentOS 6 image via Vagrant

If you would like to run Eucalyptus UI Tester from the pre-baked Vagrant image, follow the steps below:

1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps


3. Running PreBaked Eucalyptus UI Tester Image using Vagrant

, and see 4. Creating a New Vagrant Package Image if you are interested in creating a new image via Vagrant.


1. Installing Vagrant and Virtual Box on Mac OS X in 5 Steps

2. Installing Eucalyptus UI Tester on CentOS 6 image via Vagrant

3. Running PreBaked Eucalyptus UI Tester Image using Vagrant

4. Creating a New Vagrant Package Image


DevOps Culture — Fail Fast on Eucalyptus

At a meetup event down in San Diego, California, Eucalyptus had a chance to meet Sander van Zoest (@svanzoest), the VP of technology at OneHealth (, who is also the organizer of the San Diego DevOps group ( Sander and his team at OneHealth have been using Eucalyptus cloud for some time. Asked why OneHealth runs Eucalyptus in-house, Sander had some interesting stories to say about dealing with health-related data and the company’s DevOps engineering culture.


Due to the strict regulations on Protected Health Information (PHI), OneHealth needs to take extra strong measures if they are to provide the services on AWS; Sander spent a good amount of time explaining to us how demanding it is to satisfy the regulations. Such barriers make things complicated to push any personal identifiable health information to the cloud.

For the AWS case, the very specific barrier was that AWS provides no legal protection when storing sensitive data in the cloud storage space. For instance, it is required by HIPAA and HITECH regulations that One Health needs to be able to promise a 72-hour response time to inform their customers about the breach of the data, should it ever happen, and provide an ETA to identify and patch the security hole that caused the breach.

Sander points out that at the moment, AWS does not guarantee such protections/services. For this reason, OneHealth’s production environment is deployed at Rackspace’s co-location since it provides HIPAA Business Associate Addendums. However, it is noted that given the evolving nature of the public cloud, it is very “cloudy” to predict how things are going to change in the near future. The recent announcement by AWS on CloudHSM ( — although it doesn’t cover the legal protection — is a good indicator showing AWS’s interest in providing secure storage service as moving forward.


What this uncertain, “cloudy” future means for engineering at OneHealth is employing a variety of infrastructure environments to take advantage of each platform while staying flexible. It becomes essential to design OneHealth’s services and applications to be deployable on bare-metal systems at Rackspace (production environment), AWS (sandbox/staging environment), Eucalyptus (in-house continuous integration and testing environment), and engineers’ laptops using Vagrant (development and testing environment).  (

Under such heterogeneous systems, from its production down to the engineer’s laptop, the development environment — the OS, dependencies, configurations, etc — needs to be kept uniform via virtualization and automation, allowing seamless pushing of new code from the laptop up to the production. For handling the life cycle of machines and VM instances, the engineers at OneHealth are big fans of Chef (, which makes the configuration management portable on any infrastructure platforms. For virtual machines, the instance images are prepared via debian preseed files while leveraging a open source tool VeeWee (


At OneHealth, the philosophy of DevOps is deeply embedded in every aspect of its development and operation. The concept of DevOps was not new to many engineers who brought in the ideas of “Infrastructure as Code” and “Commit Often and Fail Fast” from previous companies such as and Joost.

Speaking of DevOps culture, one fun fact Sander mentioned — which goes against intuition for many traditional IT shops — was that the operation team at OneHealth likes to take down the instances and rebuild them regularly. The recycling of the instances ensures the “freshness” of the deployed services and applications. The operation engineers should be more concerned if an instance’s uptime was longer than, say, 30 days because it meant that the content of the instance was outdated, possibly containing unfixed bugs or security issues. If the deployment setup was doing what it was supposed to be doing, then it should have killed the outdated instance and brought up a new instance with the latest updates.

The same goes for the development environment. It would be much better to refresh the dev environment instances with frequent relaunching and reconstructing than having the developers working on a stale dev environment, which turns out to be more harmful for the development. Plus, this destroy-and-rebuild enforcement encourages the developers to consistently check in the code to a version-controlled code repository, allowing early detection of conflicts in code.

All of these procedures, bringing together datacenter automation and configuration management, are part of a very new movement in software development now labeled as “DevOps”. The DevOps folks often joke around and say even a few years ago, the terminology didn’t even exist, but now, DevOps has become the most sought-after practice in IT. All thanks to the wide spread of cloud computing, giving birth to the programmable infrastructure.


Introducing Metaleuca


What is Metaleuca?

Metaleuca is a bare-metal provision management system that interacts with open-source software Cobbler via EC2-like CLI.

Using Metaleuca, users can communicate with Cobbler to self-provision a group of bare-metal machines to boot up with new, fresh OS images. The main appeal of Metaleuca is that it allows users to manage the bare-metal machines like EC2’s virtual instances via the command-lines that feel much like ec2-tools, or euca2ools.



Metaleuca Command-Line Tools

Metaleuca consists of a set of command-lines that mirror some of the command-lines in ec2-tool or euca2ools. The list below shows a number of the core command-lines used in Metaleuca:

  • metaleuca-describe-profiles – Describe all the profiles provided in Cobbler
  • metaleuca-describe-systems – Describe all the bare-metal systems registered in Cobbler
  • metaleuca-reboot-system – Reboot the selected bare-metal system
  • metaleuca-run-instances – Initiate the provision sequence on the selected bare-metal systems
  • metaleuca-describe-instances – Describe the statuses of the provisioned bare-metal systems
  • metaleuca-terminate-instances – Terminate the bare-metal systems, returning them back to the resource pool

Metaleuca Configuration

Prior to installing Metaleuca, it is required that you have already configured Cobbler to provision a group of bare-metal machines in your datacenter. If you are new to Cobbler, please visit the Cobbler’s homepage for more information on how to set up Cobbler.

Once you have Cobbler running in the datacenter, you will need to install Metaleuca on a Ubuntu machine, or virtual machine. The installation guide for Metaleuca is provided at:

Metaleuca Walkthrough

Now that you have Metaleuca configured in the datacenter, let’s go over a scenario where you will want to launch two bare-metal instances with fresh CentOS 6.3 images.

First, you might want to use the command “metaleuca-describe-systems” to survey all the available systems registered in Cobbler.

Picture 3.png 03-30-32-133

Metaleuca allows users to directly select which bare-metal machines to provision — by using the machines’ IPs. However, those who are familiar with AWS will contest that this approach is not how virtual machines are provisioned on EC2; rather than specifying the IPs, the AWS users are to provide the number of instances to launch. For this reason, here, we will cover the EC2’s approach to provision the instances.

In Metaleuca, first you will need to find out which ‘profile‘ is set to install the CentOS 6.3 image. In Cobbler, a profile maps to a preconfigured ‘kickstart‘ file that contains the netboot instructions on which OS to install when a machine boots up and initiates PXEBOOT. In other words, you may compare the profiles on Cobbler to the instance images on EC2. In Metaleuca, you can display the available profiles using the command ‘metaleuca-describe-profiles‘:

Picture 7

Let’s say that the profile “qa-centos6u3-x86_64-striped-drives” is what we want to use.

Next, you will want to determine which “system-group” you want the machines to be selected from. In Metaleuca, the bare metal machines can be grouped into different resource pools. For instance, in our QA system at Eucalyptus, which utilizes Metaleuca, we partitioned the machines in the datacenter into 6 groups: qa00, qa01, dev00, dev01, test00, and test01. Such grouping allows us to provide semantics on the machine pools based on their usages, in which we created the resource allocation policy for the users, who are mainly developers and QA engineers. The command “metaleuca-describe-system-groups” displays all the machines and the groups accordingly:

Picture 8

However, keep in mind that not all machines in the list will be available to be provisioned; some of the machines might be in use by other users. Thus, you will want to run the command “metaleuca-describe-system-groups -f” to discover which machines are free to use. Fortunately, at this moment, among those 6 system-groups mentioned above, the group ‘test01′ has 2 machines available, which are labeled as “FREED” in the screen shot below:

Picture 6

Once you figure out the profile and the system-group availability information, you are ready to provision the bare-metal machines. The command “metaleuca-run-instances” takes input of how many instances you want to launch, which profile to use, which group to call from, and finally, the user string to mark the machines:

./metaleuca-run-instances -n 2 -g test01 -p qa-centos6u3-x86_64-striped-drives -u kyo_machines_for_demo

Picture 10

And, similar to ec2-tools and euca2ools, you can monitor the progress of the provisioned machines using the command “metaleuca-describe-instances -u“:

Picture 11

Notice that the instances are at the “pending” state at the moment. Soon, in about 8 minutes after launching, the instances will be shown as “running”:

Picture 12

At that point, you may ssh into the bare metal machines to verify that they are up and running with the fresh CentOS 6.3 OS installed.

Later, the command “metaleuca-describe-system-user -u” comes in handy when you want to find out which machines are provisioned under your name:

Picture 13

When you are done with the machines, you may “free” the machines so that they can return to the resource pool; the command for that case is “metaleuca-terminate-instances -u“:Picture 14

When running the command “metaleuca-describe-instance -u“, you will notice that the machines have been successfully freed:

Picture 15

Metaleuca as an Open Source Project


Metaleuca was evolved out of the internal usage — the development and test environment for engineers — at Eucalyptus, and is now available as an open source project on Github under the Apache license. The goal of the project is to complete the integration of Metaleuca into the Eucalyptus system so that it can be served as a “bare-metal only” zone in Eucalyptus. Your contribution is much appreciated!

Check out the project at:


Beyond Continuous Integration: Locking Steps with Dev, QA, and Release

Continuous integration: the practice of frequently integrating one’s new or changed code with the existing code repository [wikipedia]

In this blog we will talk about how the continuous integration process was put in place for the new component, Eucalyptus User Console, in order to collaborate the efforts among the dev, QA and release teams throughout the development cycle of Eucalyptus 3.2.


Eucalyptus User Console is a newly introduced component in Eucalyptus, whose main goal is to provide an easy-to-use, intuitive browser-based interface to the cloud users, thus assisting in the dev/test cloud deployments among IT organizations and enterprises. Eucalyptus User Console consists of two components: javascript-based client-side application and Tornado-based user console proxy server.

Early Involvement

The first phase of the development was to come up with a quick prototype to demonstrate how the user console would work under the given initial design of the architecture (see the Eucalyptus Console components layout diagram above). As soon as the prototype was evaluated and its feasibility was verified, the release team started creating the packages for two major Linux OS platforms: Ubuntu and Centos/RHEL.

The early involvement of the release team turned out to be the best help any developers or QA engineers could ask for; since the very beginning stage of the development, the release team was able to provide invaluable information that served as guardrail for the fast-moving development. Such information included advising on how the files should be named and organized and identifying which dependencies should or should not be used in order to meet the requirements for various Linux distributions. Dealing with such issues at the later stage of the development would have been undoubtedly a major pain in the back-end.


Further more, the release team was able to ensure that the development of the new user console would never go off the track against the Linux distro requirements by setting up the automated daily package-building process using Jenkins — which utilizes the VM resources from our Release cloud that runs on Eucalyptus.

Keeping Up With Eucalyptus

Setting up the automated process to build the packages would allow the release team to keep an eye on the progress of the user console’s development in terms of the ability to build the packages according to the constraints set by the Linux distributions. However, it would not guarantee whether the newly built packages contain the version of the user console that works with the current, up-to-date Eucalyptus cloud that was also in development.

Thus, the challenge was to ensure that the latest built user console packages work with the latest built Eucalyptus throughout the development.

In order to solve this issue, the QA team created a testunit that automatically installs the latest user console packages on a newly built Eucalyptus. Then, the testunit was added to the main test sequences used by the Eucalyptus 3.2 development in our automated QA system, making the installation of the latest user console packages accessible by all developers at Eucalyptus.

This setup encouraged a failure in the user console package installation to be seen by any developers throughout the development, thus allowing the failure to be detected fast and reported with quickness.

Screen shot 2012-12-10 at 5.50.02 AM

The testunit ui_setup can be seen in action above in the table which displays the results of the test sequence ran by the automated QA system.. Check out the link below for more details of this testunit:

Circle of Trust

As the user console evolved out of its prototype state and took the form of a more product-like shape, the QA team was working in parallel, figuring out how to set up the automated testing process for the user console. The blog here talks in detail about how Selenium was used to create the automated web-browser testing tools, se34euca.


In the mid-stage of the development, as the features of the user console started functioning in reasonably stable manners, 3 automated tests were added — incrementally — to ensure that the working state of the user console throughout the development.

Screen shot 2012-12-10 at 6.41.28 AMThose 3 tests are:

  1. user_console_view_page_test
  2. user_console_generate_keypair_test
  3. user_console_launch_instance_test

These automated tests were to ask the 3 simple questions below on a daily basis:

  1. Can the user log in and see all the landing pages on the latest user console?
  2. Can the user generate a new keypair using the latest user console?
  3. Can the user launch a VM instance using the latest user console?

Of course, it would be possible, and desirable, to ask more questions in a more complicated fashion. However, during the rapid development phase, asking those 3 simple questions on a daily basis, turned out to be sufficient, and effective, to understand whether something terrible had happened to the user console or not.


The goal of these automated tests at this stage of the development was not to detect every little defect in the product. Not too soon at the moment.

The main purpose is rather to serve as an indicator for the developers, QA engineers, and release engineers to assure ourselves that the change that went in the code earlier today did not ruin the delicate trust among the three groups, meaning that the build, installation, and configuration procedures are still in tact. Having such assurance in check by mechanical means has made the three groups extremely effective in discovering issues during the development since it allowed each member to narrow down exactly what was responsible for the defects in a finely reduced time frame, which was in hours, rather than days or weeks.

Guardrail For Development

Having the automated package build process and the automated installation/configuration process in place at the early stage of the development was proven to be extremely useful; rather than agreeing on the written procedures, the dev, QA, and release team materialized such agreements into the actual implementation, and put them into work by using various automated mechanics that run on a daily basis. Therefore, throughout the development, we were able to witness and assure ourselves that we were making progress in accordance with the plan and our self-imposed restrictions.

Check out the Eucalyptus Open QA webpage to see the continuous integration at Eucalyptus in action:

Eucalyptus Open QA (beta) –

TCP Dumpster: Monitoring TCP for Eucalyptus User Console

This is the part III of the Eucalyptus Open QA blog series that cover various topics on the quality assurance process for Eucalyptus’s new user console.Eucalyptus User Console

On this blog, we would like to share the information on how we monitors the traffic on the user console proxy, using the Linux command ‘tcpdump‘ and its rendering application ‘tcpdumpster‘, to derive and understand the behaviors of users when interacting with the user console.


Eucalyptus user console consists of two components: javascript-based client application and Tornado-based user console proxy. When logged in, the client-side application, which runs on a user’s web-browser, polls the user’s cloud resource data at a certain interval, and the user console proxy, located in between the cloud and the users, relays the requests originated from the client applications.


Recalling from the first blog of the series, our challenging question was, when 100+ users are logged into the Eucalyptus user consoles at the same time, would the user console proxy be able to withstand the traffic that was generated by those 100+ users? Plus, how do we ensure the user experience under such heavy load?

The answer to the questions above was provided in details here.

The short answer is to generate 100+ user traffic using the automated open-source web-browser testing tool, Selenium, while manually evaluating the user experience on the user console.

However, prior to answering the questions above, first we needed to establish a way to quickly, yet effectively monitor the traffic between the clients and the proxy in order to make observations on the patterns and behaviors of the traffic.

TCP Dump

tcpdump‘ is a standard tool for monitoring the TCP traffic on Linux. For instance, if the user console proxy was running on the port 8888 on the machine, monitoring the traffic on the port 8888 can be as simple as running the command below on the Linux terminal at

tcpdump port 8888

Then, this command will “dump” out the information on every packet that crosses the port 8888 on the machine However, the information generated by this command is just too overwhelming; such information would fly by on the terminal screen as soon as the user consoles start interacting with the proxy. There had to be a better way to render the output of the command ‘tcpdump‘.

 TCP Dumpster

At Eucalyptus, using the automated QA system, a new, up-to-date Eucalyptus system is constantly installed and torn down within a day or two life span (check out here to see the Eucalyptus QA system in action). For this reason, we needed to come up with a quick way to set up the monitoring application on the machine where the proxy was installed. Plus, we would like to have all necessary monitoring information displayed on a single HTML page for a quick glance, thus making it easier for the observer to apply intuition on understanding the big picture. As a result, ‘tcpdumpster‘ was born.

Picture 96

The application ‘tcpdumpster‘ runs on the same machine where the proxy is installed. It runs the Linux command “tcpdump port 8888” and parses its output into a list file. This list tracks 8 attributes of the TCP traffic:

  • Unique connections, based on IP
  • Unique connections, based on Port
  • Connection count, per second
  • Connection count, averaged over a minute
  • Connection count, in total
  • Packet length, per second
  • Packet length, averaged over a minute
  • Packet length, in total

With those 8 attributes displayed on a single HTML page, which can be accessed via:

, we were able to make some interesting observations on the behaviors of the traffic as the user console starts interacting with the proxy.

TCP Dumpster Examples

The graph below is showing the traffic pattern for 7 minutes, generated by a user logged in to the user console.

Picture 18

Notice the first peak that represents the log-in of the user, followed by the periodic peaks that show the polling of the cloud resource data, and user actions can be seen in the blobs among the peaks.

The graph below is showing the traffic pattern as more selenium-based automated scripts are activated to simulate a large amount of users.

Picture 46

The first block shows when 1 and 2 Selenium scripts are active, and the second block shows when 6 and 12 Selenium scripts are active (check out here to learn how Selenium was used). When graphed for averaged over a minute, the differences between the stages become more visible:Picture 47

When graphed all together, along with the connection data, they look below:

Picture 45

tcpdumpster‘ turns out to be very useful when validating if a newly written selenium script is behaving correctly. The graph below shows the selenium script that launches a new instance, waits until the instance is running, then terminates the instance, waits a few minutes, and repeats:

Picture 81

And, of course, ‘tcpdumpster‘ is very handy when you are running a longterm test; it allows me to set up the test, go to sleep, and wait up the next day to check out the results. The graph below shows how the proxy was able to withstand the constant ‘refresh’ operations from multiple connections for longer than 5 hours:

Picture 94

Now, can you guess what is going on in the graph below?

Picture 105

Check out the GitHub link below and try out ‘tcpdumpster‘ on your own Eucalyptus user console proxy to find out for yourself:

Cloud App. Design: Create a Flexible Automated Web-UI Testing Tool using Selenium

In this article, I will go over the technical details on how in Eucalyptus, we used Selenium to simulate a large number of cloud user workload in order to ensure the quality of user experience in the new Eucalyptus user console.

As covered in my previous blog, Eucalyptus is coming out with a new user console, that is browser-based and intuitive to use, thus playing a key role in promoting the cloud adoption among IT organizations and enterprises.  But, the challenge was to ensure that this brand-new user console would be ready for handling the real-world workload when released out in the wild. The answer to this challenge was to simulate the activities of 150 cloud users using Selenium, an open source tool for automating web application testing.

How to Automate an Online User

The first step was to download Selenium IDE for Firefox. Selenium IDE is a must-to-have GUI tool for automating clicks and input-submits on a web application. After installing Selenium IDE on your computer, you can start Selenium IDE from Firefox’s Tools menu:

When started, Selenium IDE opens up its own separate window:

Notice the red dot on the top-right corner of Selenium IDE. When clicked, Selenium IDE will start recording all the activities you perform on the web-browser — every link you click and every input you type on the browser will be recorded as command-lines on Selenium IDE as seen below:

What Selenium IDE allows you to do is to replay the recorded activities, such as clicking and typing, on the browser in the exact same order that they were performed.


But, soon you will notice that when replayed on Selenium IDE, it tends to fly through all the clicks in lighting speed so that the replayed activities often result in failures — the browser and web application cannot keep up with the speed of the clicks performed by Selenium IDE.

In order to prevent such cases, you will need to manually step through the record activities and insert various “pause-and-check” points using ‘waitForElementPresent’ command. For instance, when there is a command ‘click link=Delete’, I would put ‘waitForElementPresent link=Delete’ command prior to the click command to ensure that the page will fully loaded and the link ‘Delete’ is indeed present on the page before allowing Selenium IDE to execute the command ‘click link=Delete’. Later I learned that for every ‘click’ command, it is always a good habit to throw in the ‘waitForElementPresent’ command.

After verified that the recorded action is repeatable via Selenium IDE at its full speed, the next step is to export the action into a Selenium Python WebDriver format:

The result of the export above is a script file that describes the recorded Selenium action in a Python’s unittest format:

Once have the script exported, you can run the recorded action on a remote Selenium server without having to open up a web-browser. In other words, now you can simulate an online user doing the exact same recorded action on a web-browser by simply running the Python script generated by Selenium IDE.

Remote Selenium Server Configuration

Before running the script, you will need to configure a machine to run a remote Selenium server, which will behave like a web-client. The steps are on a Ubuntu machine, you will execute the following commands:

sudo apt-get -y update
sudo apt-get -y install default-jre
sudo apt-get -y install xvfb
sudo apt-get -y install firefox
sudo apt-get -y install python-pip
pip install selenium
Xvfb :0 -ac 2> /dev/null &
nohup java -jar selenium-server-standalone-2.25.0.jar &
export DISPLAY=:0

After running the commands above, you will have a Ubuntu machine capable of running the exported Python Selenium script, which then, simulates an online user opening up a Firefox browser and performing the recorded clicks.

Creating a Flexible, Reusable Testing Tool

Now, your task is to produce many exported Python Selenium scripts for all activities on the web application that will be used as building blocks for creating different user behaviors and workflows.

The first collection of Python Selenium scripts I produced was to visit every single landing page on the Eucalyptus user console. The second collection of Python Selenium scripts was to create cloud resources under the default setting. Having those two sets of Python Selenium scripts allowed me to construct complicated user interactions on the web application. For instance, with a bit of shuffling of the scripts, I could build up a user scenario where the online user would visit the keypair page, create a new keypair, visit the dashboard page, visit the security group page, create a new security group, revisit the dashboard page, and so on.

The next task was to consolidate all the scripts into one library file, getting rid of static values in the variables and breaking down the actions in the scripts into functions. Having a such unified library enables test-writers to stitch and arrange these functions together to construct whole new user scenarios as needed.

When examining the Eucalyptus user console test framework se34euca, you will see that the main library file ‘’ contains the functions that are exported from Selenium IDE, where each function describes a very specific action to perform on the web console. The files ‘testcase_*.py’ list the arrangements of those functions to form simple, or complex user behaviors. Finally, the files ‘runtest_*.py’ are the executables of those test cases that take input of the target web console environment.

Cloud Application

Now that you have a way to convert a Ubuntu machine into a Selenium server and have the Selenium test framework checked into a GitHub repository, you have a way to launch the Selenium test as a cloud instance — using se34euca as an example, the steps are:

Step 1. Launch a cloud instance on a Ubuntu image.
Step 2. Convert the Ubuntu image into a Selenium server by running the configuration commands above, or running the installer in se34euca.
Step 3. Git clone se34euca.
Step 4. Run the test case of your choice.
Step 5. Terminate the instance when the test is finished.

Of course, you can easily automate the step 2, 3, and 4 to wrap the entire process into a single scripted operation. Then, with a help of a cloud infrastructure, such as AWS or Eucalyptus, simulating 150 user can be as simple as launching 150 instances to run the script on each instance by feeding the parameter as user-data.

Code Reference

For those who are interested in creating a framework for testing your own web application, please feel free to check out the Eucalyptus user console test framework se34euca at:

for a reference, and leave a comment if you have any questions or suggestions.


Get every new post delivered to your Inbox.

%d bloggers like this: