Kyo Lee

Open-Source Cloud Blog

Simulate 150 Cloud User Activities Using Open Source Tools

For the 3.2 release in this December, Eucalyptus is coming out with an intuitive, easy-to-use cloud user console, which aims to support the on-premise dev/test cloud adoption among IT organizations and enterprises.


This easy-to-use Eucalyptus User Console is consisted of two main components: a browser-side javascript application, written in JQuery, and a proxy server that utilizes Python Boto to relay requests to Eucalyptus Cloud, which is written in Python Tornado, an open source version of the scalable, non-blocking web server developed by Facebook.

The target scale for the initial version of the user console is set to handle 150 simultaneous user activities under a single user console proxy.

Now, the challenge is how to simulate these 150 users to ensure that the user consoles and the proxy are able to withstand the workload of 150 active cloud users; more importantly how to ensure that such workload is not jeopardizing the user experience on the console.

One obviously answer is to find 150 people, train them thoroughly, and ask them to participate in the load testing. After all, 150 is doable.

However, what’s not doable is that having those 150 people to repeat the process over and over during the entire life cycle of the development until the release.

Then, the most “realistic” answer is to simulate those 150 people using machines. It turns out that the machines are really good at repeating the same things over and over, and they tend to behave in a very predictable manner when tuned properly.

At Eucalyptus, we use Selenium, open source web testing automation tools, to simulate the actual user interactions on the user console.

The steps are first, use Selenium IDE on Firefox to write an automation script that completes a single path of cloud user workflow — for instance, one simple user workflow is to log into the console, create a new keypair, and log out, and another workflow to log in, create a new volume, and log out. Second, repeat the first step above for all possible use cases to ensure that all, or most, of the functionality on the console are covered, allowing all use cases to be automatically executable via Selenium IDE. Third, export those automated IDE scripts to Selenium Python WebDriver format, which allows the automated scripts to run on a remote server without needing to actually opening up a browser. Finally, create a wrapper for each exported script so that each test case can be execute as a command-line tool on Linux.

The link below contains the collection of automated Selenium WebDriver test scripts, command-line tools, and their installer for testing the Eucalyptus User Console:

Se34Euca (Selenium34Eucalyptus) – https://github.com/eucalyptus/se34euca

With Se34Euca, you can instantaneously convert any machine — or virtual machine if you are already a cloud geek 😉 — into a Eucalyptus cloud user simulator.

The steps are, on a Ubuntu image, run the commands below to install and setup Se34Euca:

sudo apt-get -y install git-core

git clone git://github.com/eucalyptus/se34euca.git

cd ./se34euca/script/

./installer_se34euca.py

Then, running the actual test can be as simple as:

export DISPLAY=:0

./runtest_view_page.py -i 192.168.51.6 -p 8888 -a ui-test-acct-00 -u user00 -w mypassword1 -t view_all_page_in_loop

The command line above will simulate a cloud user clicking through every single landing page on the user console within 2 second, then taking a rest for 5 seconds, and repeating the frantic, yet controlled clicking again and again and again.

However, funny enough, it turned out that the automated script’s ability to click through all pages on the user console within 2 seconds was well beyond the capability of a human user. The graph below renders the normal behavior of an actual human user. The X-axis in the graph shows the total length of TCP packets seen in a second on the user console proxy server machine via tcpdump.  Notice the peak in the beginning as the user logging in, and a group of little ripples that mark the user clicking buttons or viewing different pages in a 7-minute period:

And, the graph below shows the difference in the actual user behavior and the automated script behavior simulated by a single instance of Se34Euca. Notice the super-human strength of the automated script — the first half of the graph below is showing the same 7-minute period shown in the graph above. According to the graph below, the automated script is able to generate 10 times workload than a human user.

This discovery turns out to be good news; the fact that one Se34Euca instance can generate 10x human user workload, all I need to do is to launch 15 instances of Se34Euca to simulate 150 users. So, I provisioned 3 Ubuntu machines and launched 5 instances of Se34Euca on each machine:

The first fifth of the graph above covers the same period as the second graph above. What you are looking at is 15 instances of Se34Euca clicking through every single page on the Eucalyptus User Console for about two hours, starting at 21:00 mark.

When computed for averaging the packet length per second over 60 second observation period, the graph looks below:

The graph above is showing that when 150 users simultaneously logging in to the user consoles, the average packet transmission throughput rate seen on the wire is about 750Kb per second. Assuming that the user console proxy server is hooked on 1 Gig link, the throughput of 750Kb per second is certainly “doable” by all means. 😉

Then, how do we ensure the user experience of the console?

Simple. While the user console proxy server is being slammed by 150 click-monkeys, I’m opening up my own browser to verify that my interaction with the console is smooth as usual. 🙂

On my next blog, I will cover more details on the exact setup of the Eucalyptus User Console load-testing, including the selenium scripts and monitoring setup, and dig deeper into the analysis of the data. Please, stay tuned 😉

Meanwhile, feel free to check out the blog below if you would like to preview the Eucalyptus User Console for yourself:

http://coderslike.us/2012/11/11/installing-the-eucalyptus-console-from-source-and-packages/

Advertisements

Allow Everyone to Launch Your Web-Service: Open Source Web-Service using GIT and AWS

Open Source is the future of software.

The statement above is self-evident at this point of history — I mean, have you checked out www.microsoft.com/opensource recently?

Then, what does open-source mean when it comes to web-services?

Open Source has been a well-understood concept among the web-application community due to the neat feature “View Source” that came with almost all web-browsers. Copy-and-pasting a chunk of HTML code from one website to another was a de facto standard way of developing websites. Whether admitted it or not, web-application development is deeply rooted in open source culture.

But, where should this open source culture be heading to? Should we keep maintain the culture of copy-and-pasting others’ code with no telling? Or, rather try to embrace this open source culture and explore the possibilities that arise from sharing code?

Then, it occurred to me that, “What if we could share the entire code for web-services?”

Imagine if anyone could launch web-services such as E-bay, Craigslist, Amazon, Angie’s list, Yelp, Instagram, Twitter, etc on their own instances on AWS. Anyone could instantly replicate all the functionality and services provided by such known web-services at its micro scale. If the big name web-services are the department stores in downtown, these open-source web-services on AWS instances are the mom-and-pop stores for the local community, or they could be the black market since these services could form and disappear in a rather unpredictable manner — similar to the life cycle of AWS instances. 😉

Think about the scenario where a person can launch his/her own e-bay online auction service on an AWS instance for a school fund-raising event for a month. The person might also want to launch an instance of Craigslist for the same event, or Instagram and Twitter to keep people engaged during this period. Then, once the event is over, all the services can go away as if nothing ever happened.

Welcome to the era of Cloud Applications.

To test out the concept, I used the application “Open QA“, a web-service that is designed to display the test results of the development progress at Eucalyptus. The goal is to allow any Eucalyptus community member to launch the same service on AWS instances.

First, the web-service has to be broken down into two bodies, the service layer and the data. Then, each body of the web-service needs to be available in the open, allowing anyone to download the code and convert the AWS instances to Open QA web-service instances.

This is where GIT comes in very nicely. For public repositories, GIT will allow you create as many branches as you want, which can serve as read-only code repositories for anyone on the Internet without hassle.

The service layer part of Open QA is available at:

https://github.com/eucalyptus/open-qa

The data for Open QA is available at:

https://github.com/eucalyptus/open-qa-frontend-data

Upon launching an instance on AWS, a user can simply convert the raw Amazon Linux instance into the Open QA web-service by executing 4 commands below:

sudo yum -y install git
git clone git://github.com/eucalyptus/open-qa.git
cd ./open-qa/script/
./open_qa-installer.py -t amazon -e <your_email>

At this point, the AWS instance is running a HTTPD service with Open QA PHP code in its /var/www/html directory.

However, with only the service layer installed, the instance is missing the data, thus has no content to display.

Notice that the data for Open QA is available at a different GitHub repository. The user can supply the service with the actual Open QA data by following 5 commands below:

cd ~
git clone git://github.com/eucalyptus/open-qa-frontend-data.git
cd ./open-qa-frontend-data
sudo tar -zxvf ./cache_storage_for_open_qa.tar.gz
sudo cp -r ./cache_storage_for_open_qa/* /var/www/html/webcache/.

Now, the instance is completely converted to run the Open QA web-service with the up-to-date test results.

The data for Open QA is, for now, updated at every 12:01am — pushed to the public GIT repository from the QA server at Eucalyptus HQ. In order to keep the data in sync, the AWS instances are scheduled to periodically pull the latest data via “git pull” on the data directory and copy over the new data to /var/www/html/webcache directory, which can be set as a cron job via “sudo crontab -e” on the instances.

This proof-of-concept case with Open QA application is to demonstrate how GIT and AWS can be used to make an open-source web-service to be launched by anyone with the AWS account. The separation of the service layer and the data for the web-service allows the update to take place independently; for some other web-applications, it might be desirable that the user supplies his/her own customized data, thus only utilizing the service layer.

If anyone ever dreams of launching 10,000 instances of a web-application in the cloud, it must be done through the open source scenario above where the application pulls the needed service and data at its own pace and for its own purpose, which is to serve locally.

10 Steps to Euca Monkey

Euca Monkey is an easy-to-deploy test tool designed for performing stress-test on Eucalyptus Cloud. The tool repeatedly generates and tears down 6 types of cloud user resources: running instances, volumes, snapshots, security groups, keypairs, and IP addresses. As the resources are being populated and released, the cloud is actively queried to validate such resources are indeed being allocated correctly per request. Then, the tool renders the progress of the stress testing using Gnuplot, an open source graphing tool, and displays the graphs as webservice in real time.

Euca Monkey uses cloud-resource-populator — which utilizes Eutester, which is based on Boto, thus making the tool AWS-API-compatible  — to populate and release resources from Eucalyptus as a user. The input for cloud-resource-populator looks below:

[USER INFO]
account: ui-test-acct-23
user: user-23
password: mypassword23
[RESOURCES]
running instances: 2
volumes:  2
snapshots: 1
security groups: 10
keypairs: 3
ip addresses: 2
[ITERATIONS]
iterations: 200

With the given input above, cloud-resource-populator will generate the resources as specified in the [RESOURCES] section as the user ‘user-23′ under the account ‘ui-test-acct-23′. When viewed from Eucalyptus user console, it will look as below:

As soon as the resources are populated according to the specification, cloud-resource-populator will immediately send requests to the cloud to release all the allocated resources, which makes the console view look as below:

And, as you would have guessed, the process of populating and releasing of the resources is repeated for [ITERATIONS] times.

The most appealing feature of Euca Monkey is that it launches a webservice to render the progress of the stress-testing in real time.

The graph above is showing the input values of a few iterations of the resource population and tear-down process. When tracing the running instance line, which is in red, this graph is telling us that 20 instance were started on the first request, then those 20 instances were all terminated on the second mark, thus brought the count down to 0. And, 20 instances were started again, then terminated, and so on. Such operations were repeated 7 times on the graph above. And, also notice that there are 4 other resources being populated and released 7 times as well.

When the cloud is behaving nicely, the actual resources should be populated and released in match with the input values in the graph above, thus resulting in the output graph as below:

While the first graph shown above renders the input values to the tool cloud-resource-populator, this graph is showing the actual values reported by the cloud. The fact that these two graphs look the same means “Yay Cloud!!”

However, occasionally, during the development of Eucalyptus, you would see the graph like below on a rainy day in Santa Barbara:

The graph above reveals an interesting state of the cloud. Notice that the “running instance” line went from the “nice” behavior pattern to a flat line. It means that the cloud was able to launch and terminate 20 instances during the first phase, but somehow it got stuck to the state where it was not able to release instances, nor launch more instances, thus stuck with 18 running instances. But, notice that other resources were making the usual progress as before, except the security groups. It turned out what we were witnessing was the deadlock case in Eucalyptus Cluster Controller, which occurred around the 8th hour of the stress-testing. R.I.P CC. 😦

The purpose of stress-testing is to push the limit of the system to the point where malfunctions and faulty behaviors of the system can be observed. Such stress testing is crucial for the development of a distributed system like Eucalyptus; many unknowns and bugs are constantly introduced to the system as new features from various components are being integrated. Thus, having an easily deployable stress-testing tool with visualization support, such as Euca Monkey, yields tremendous benefits for the developers in an agile development environment since the tool aims to ensure the system’s stability and reliability throughout the rapid development cycle.

If you would like to take Euca Monkey for a spin, feel free to check out the GitHub link below. On a fresh Centos 6 machine or VM, it will take 10 simple steps to launch your own monkey.

https://github.com/eucalyptus/euca-monkey

Other Resources:

Eutester – https://github.com/eucalyptus/eutester

Boto – https://github.com/boto/boto

Eucalyptus QA – https://github.com/eucalyptus-qa

Open QA for Eucalyptus – https://github.com/eucalyptus/open-qa

OPEN QA for EUCALYPTUS

Open Source Software Project… What is it good for?

It’s free! It’s for the community! It’s the future of software! It’s better than closed! Everyone is doing it! It drives innovation! It’s the way of Steve Jobs (huh?)… And, it’s free!

I mean, seriously.

All I hear is nothing but the presuppositions on the greatness of open source — how it will greatly benefit you and your great organization.

But, can someone tell me how an open source project will benefit itself for being out in the open?

Yes, open source is free. Yes, you can download it and use it without paying a dime, which is fantastic by anyone’s standard.

Then, you will soon realize, “Wait, that sounds just too good to be true. There must be something they get in return for giving it away for free.”

Yes, it is absolutely true, and it comes down to this one crucial benefit:

QUALITY.

It turns out, among those freeloaders, there exist these rare kinds who defy intuition and want to give it back for some reason — there are those who want to share their abilities to fix things, the abilities to break things, the abilities to point out flaws, the abilities to compliment beauty, the abilities to talk smack, and the abilities to appreciate what others have done.

When the project is out in the open, feedback from these enthusiasts comes in various forms, in very chaotic ways, which is overwhelming at first. However, every single interaction with these folks contributes to one significant attribute of the open source project — it pushes the quality of the software.

Eucalyptus has been striving to achieve an incomparable goal, that is to be the most tested, thus the most stable, in turn, the most dependable cloud infrastructure in the open. It has been a long and tough road for us to march on through the ups-and-downs of tech-industry turmoil. However, the commitment toward this goal has been unwavering.

In part of the effort to become the most dependable open source cloud infrastructure, Eucalyptus is welcoming all community members to participate in the quality assurance process of Eucalyptus development.

OPEN QA for Eucalyptus Wiki Page:

https://github.com/eucalyptus/open-qa/wiki

OPEN QA Website (Beta Version):

http://ec2-50-112-61-121.us-west-2.compute.amazonaws.com/open_qa.php

Every community member is invited to check out the Open QA website on a daily basis and provide feedback and criticism on the development process of Eucalyptus.

You will find Eucalyptus developers engaging in active conversations on the IRC channels: #eucalyptus, #eucalyptus-devel, and #eucalyptus-qa on irc.freenode.net. Or, you may also prefer posting your thoughts on a forum via: https://engage.eucalyptus.com Or, please feel free to directly file bugs on Eucalyptus via JIRA at: https://eucalyptus.atlassian.net/browse/EUCA

But, in whichever way you decide to engage with us, please never hesitate to:

Live long and prosper \\//

— Open QA for Eucalyptus —

The Kids Are All Right

Metallica, Outside Lands, San Francisco

I Love The Internet Machine

Loving the Interweb.

Implication of Fragmentation in Linux

At Eucalyptus we have been proud to say,

“Eucalyptus Cloud runs on almost all major Linux distros: Ubuntu, Debian, CentOS, Red Hat, Fedora, etc.. You name it, we will support it!”

The crowd erupts in cheers, with occasional tears of joy. There will be a parade later.

Yes, it sounds wonderfully majestic as it should be.

When developing open-source software, you need to support all major, stable Linux platforms so that the software can reach out to every single open-source enthusiast who is often loyal to a certain flavor of Linux.

But, what does all this mean to software developers?

A nightmare.

A nightmare accompanied with a horrible migraine.

In the ideal software utopia, all Linux platforms behave in an identical manner; you should be able to run things on Fedora in the same exact way you run those things on Ubuntu. After all, they are all “Linux”, aren’t they?

Welcome to the harsh reality called “fragmentation” in the software world.

It is true that all those distros feel like Linux; they share the same core — Linux kernel — and provide the same level of abstraction which can be described as “Linux experience.”

But, in reality, no two Linux distros are never the same.

The biggest problem with this inconvenient truth is that no one knows for sure what the exact differences are when going from one distro to another distro.

Let’s say we use CentOS as a default Linux distro for developing and testing software. If every function and feature works well on CentOS, can we safely assume that the software will also behave nicely with other Linux distros, such as Red Hat, Ubuntu, and Debian?

The answers to this simple question always fall somewhere in between “Maybe”, “It depends”, “Possibly yes”, “Theoretically it should”, “What is Linux?”, and “NO!”

The only assured way to discover the correct answer is to run tests on the software under all distros.

At present Eucalyptus officially supports three main Linux distros — Ubuntu, Redhat, and CentOS — and is working on adding two more distros: Debian and Fedora. Every time a new distro is added, we are to repeat the entire set of test suite for the new distro. If running a whole set of test suite takes X amount of resources, supporting 5 distros would mean 5X resources. Even further, per distro, we support two of the latest versions — for instance, for CentOS, Eucalyptus supports its version 5 and 6. This additional requirement brings up the total amount testing resources to be 10X.

When translated to the operation cost, if running a complete set of test suite under one Linux environment takes one day, due to the fragmentation of Linux distros, we need to add nine more days of testing in order to completely cover all corner cases in various Linux distros.

The real world implication of this nightmare is that whenever a little tweak goes into Eucalyptus, it might take up to 10 days, in the worst case scenario, to ensure ourselves that this seemingly innocent tweak will not bring down the house under some other Linux distros in strange, unpredictable ways.

In school we are taught to celebrate diversity, but they often forget to emphasize the beauty in simplicity. Handling the issue of the Linux fragmentation remains to be one of many challenges that Eucalyptus has to overcome.

photo(1)

Celebrating Higgs Boson Discovery in America

Celebrating Higgs Boson Discovery in America 😉

Eucalyptus 3.2 Coming Soon!

Had a little fun with iMovie during Eucalyptus 3.2 Design meeting week.

Quality Flow in Eucalyptus

When being interviewed at Eucalyptus, one is often asked, “when do you stop testing software?” This is not a trick question. As a matter of fact, this is not even a question; there is only one answer, and everyone knows it.

You never stop testing.

In Eucalyptus, we take the answer above very much literally.

Introduction

Eucalyptus has always displayed strong interest in creating an innovative, state-of-the-art software development workflow that supports four value-driven development practices: continuous integration, continuous testing, extreme automation, and open development. Based on these core principals, Eucalyptus engineering has designed an advanced quality control process, backed by a versatile automated QA system, whose function is to maintain the quality standard in Eucalyptus that is persistent and unchallengeable.

The graph below shows the overview of Eucalyptus software development workflow:

Graph 1. Quality Flow of Eucalyptus Software Development

Overview

Eucalyptus quality control process ensures that there exists monotonic increase in quality in the code as traveling through quality assurance check-points, known as Quality Gates, represented as green bars in Graph 1. Each quality gate is a set of automated operations performed by Eucalyptus’s automated QA system, which manages testing of Eucalyptus in a completely automated fashion — including bare-metal provisioning, installation and configuration of Eucalyptus, sequencing of automated tests, and providing feedback via various medium.

Eucalyptus quality control process is not just a quality assurance process; it is fully integrated into Eucalyptus software development. The process combines continuous integration and continuous testing into a unified workflow, which is powered by extreme automation via the QA system. The goal is to get rid of mundane tasks for developers so that they can focus on tasks that they are good at doing — developing a cloud infrastructure system of the highest quality.

Advantages

Guarantees in Quality

The branch #master (https://github.com/eucalyptus/eucalyptus) is the face of Eucalyptus open development model; every Eucalyptus engineer and community member checks out from the branch #master for development, experiment, or contribution. Thus, it is absolutely crucial that at all time, the quality of the branch #master is set to meet the standard of a functional cloud infrastructure system. In other words, at any given moment, a person should be able to check out from the branch #master and immediately start developing on top of the existing system without wasting time wondering if the system has any major problems that may influence further development. It is Eucalyptus’s responsibility to provide such guarantees in quality in the open branch #master. It is such guarantees in quality that encourage the community members to actively engage with Eucalyptus since it “defrictionalizes” contributions from merging into the main body of the code.

Commit Often and Fail Fast

In the early days, without formal guidance put in place, Eucalyptus has experienced a fair share of growing pain where development branches have drifted far, far away. The only known, effective solution to the “drifting development branch” problem is to have developers commit and check out the code as often as possible (continuous integration). However, the challenge in this practice is on how to minimize the effort that goes into this checking-in and checking-out process. The more tedious and complicated this process is, the more a developer is inclined to delay the process, causing his or her branch to be drifted far, far away. On the other hand, overly simplified commit-and-checkout process will severely jeopardize the quality of the code since a such process would likely permit unchecked mistakes to sneak into the main body of the code.

Eucalyptus quality control process attempts to address this issue by automating the entire “code-commit to quality-validation” process with a click of a button (extreme automation). As seen on TV, a developer can “just click it and forget it” when committing the code – that is until he or she gets a notification email indicating whether the commit has passed or failed to merge into the branch #master. The goal is to encourage developers to commit frequently by providing an easy mechanism that automatically performs quality verification on the code and returns fast feedback.

Procedures

In Eucalyptus software development workflow, a commit from a development branch to the branch #master is a two-step process. As seen in Graph 1., there is no direct path that leads to #master. Every commit must go through the aggregation branch called #testing. When the code on a development branch – from the far left stage on the graph – is committed to #testing, leaving the hand of a developer, the automated quality assurance process – represented as “Quality Gate 2” in the graph – will determine whether the code is allowed to move to #master. If the code fails to meet the quality standard enforced by Quality Gate 2, the branch #testing will be reverted to remove the history of the failed code. The purpose of this action is to bring the state of the branch #testing back to the last known stable quality. This action allows other developers to continuously commit to #testing even after previous commits in #testing fail to enter the branch #master.

The automated tests triggered by Quality Gate 2 is set to consume about 20 QA machines running tests for 5 hours. The cost of operation for Quality Gate 2 is very high. Therefore, there is a need for setting up a pre-commit check-up process — represented as “Quality Gate 1” in the graph above — in order to perform quick spot-checks on development branches. Such preliminary check-up process aims to minimize failures caused by preventable mistakes when committing to #testing.

Challenges

1. Scale and Resources

Ideally, every commit to the branch #testing must be examined and validated independently. However, in reality, with the limited resources and the frequency of commits, it becomes impossible to process all the commits on time if examined one at a time. Let’s say, in order to perform all the necessary automated tests kicked off by Quality Gate 2, it takes 20 QA machines running tests for 5 hours. Given this resource requirement, processing 5 commits individually would take 100 machines running tests for 5 hours in parallel, or the entire day if those commits end up completing for the resources. It is not difficult to see how fast the wait-line in the queue will grow for pending commits.

The current solution to this problem is batch-processing of commits at a scheduled time. There will be 2 or 3 times a day when Quality Gate 2 opens up to provide the opportunities for pending commits to “try out” for the branch #master. All the commits will be bundled together and go through the automated quality assurance testing as a group. If it passes, then all the commits will enter the branch #master. But, if it fails, then all the commits are rejected together, notifying each developer who submitted the commit accordingly. By examining the test-results report, attached to the notification email, the developers should communicate with each other to determine exactly what went wrong in the system and learn whose commit was responsible for the failure, which will allow the other, non-responsible developers to retry the commit.

2. Quality of Automated Tests

Eucalyptus quality control process counts on an extensive range of automated tests that cover all the essential functionality of a cloud infrastructure system. Those automated tests are designed to capture the accurate view of the target system. At the end of testing, there must be a clear “passed” or “failed” signal for each automated test with a high level of reliability. Any automated test that lacks such reliability provides no value in Eucalyptus quality control process since it would introduce noise when trying to examine the overall quality of the system.

To make the matters more complex, in agile-like development, today’s feature tests are tomorrow’s regression tests. In other words, in a such fast-paced development cycle, the automated tests written for testing new features in this release cycle will be used for detecting regressions in the next release cycle. In such continuous testing environment, combined with agile development, there exists a close relationship between the quality of the features and the quality of the automated tests; poorly written tests for a feature would result in a poorly maintained feature since such tests would fail to detect regressions as the development progresses. Thus, in order to develop good quality features, Eucalyptus quality control process must provide rapid ways to:

  • Ease the maintenance of existing automated tests (for detecting regressions)
  • Ramp up the maturity time of newly created automated tests (for validating new features)

Eucalyptus’s approach to these challenges is to develop a sophisticated testing framework, called “eutester“(https://github.com/eucalyptus/eutester), in order to enhance the automated test writing and maintenance experience.

3. Extremeness in Extreme Automation

Eucalyptus quality control process is glued together by complete automation. The entire process is put on auto-pilot that is guided by various automated pieces and modules working in accordance. Each independent component should have the ability to make decisions based on its own intelligence, meaning that it knows how to configure itself, run tasks, detect failures, examine the condition of the failures, recover from the failures, gather information, communicate with other components, and etc.. A collection of such intelligent components is essential to keep the quality process in motion with minimal interruptions. Now, the challenge is, how do we make it think like a QA engineer?

FYI. Eucalyptus is currently hiring talented Quality Assurance engineers who share the same vision and will power to attack these challenges (http://www.eucalyptus.com/careers/jobs).

%d bloggers like this: