Kyo Lee

Open-Source Cloud Blog

Tag: QA

Beyond Continuous Integration: Locking Steps with Dev, QA, and Release

Continuous integration: the practice of frequently integrating one’s new or changed code with the existing code repository [wikipedia]

In this blog we will talk about how the continuous integration process was put in place for the new component, Eucalyptus User Console, in order to collaborate the efforts among the dev, QA and release teams throughout the development cycle of Eucalyptus 3.2.

Backgrounduserconsoleconponentview

Eucalyptus User Console is a newly introduced component in Eucalyptus, whose main goal is to provide an easy-to-use, intuitive browser-based interface to the cloud users, thus assisting in the dev/test cloud deployments among IT organizations and enterprises. Eucalyptus User Console consists of two components: javascript-based client-side application and Tornado-based user console proxy server.

Early Involvement

The first phase of the development was to come up with a quick prototype to demonstrate how the user console would work under the given initial design of the architecture (see the Eucalyptus Console components layout diagram above). As soon as the prototype was evaluated and its feasibility was verified, the release team started creating the packages for two major Linux OS platforms: Ubuntu and Centos/RHEL.

The early involvement of the release team turned out to be the best help any developers or QA engineers could ask for; since the very beginning stage of the development, the release team was able to provide invaluable information that served as guardrail for the fast-moving development. Such information included advising on how the files should be named and organized and identifying which dependencies should or should not be used in order to meet the requirements for various Linux distributions. Dealing with such issues at the later stage of the development would have been undoubtedly a major pain in the back-end.

jenkins_logo

Further more, the release team was able to ensure that the development of the new user console would never go off the track against the Linux distro requirements by setting up the automated daily package-building process using Jenkins — which utilizes the VM resources from our Release cloud that runs on Eucalyptus.

Keeping Up With Eucalyptus

Setting up the automated process to build the packages would allow the release team to keep an eye on the progress of the user console’s development in terms of the ability to build the packages according to the constraints set by the Linux distributions. However, it would not guarantee whether the newly built packages contain the version of the user console that works with the current, up-to-date Eucalyptus cloud that was also in development.

Thus, the challenge was to ensure that the latest built user console packages work with the latest built Eucalyptus throughout the development.

In order to solve this issue, the QA team created a testunit that automatically installs the latest user console packages on a newly built Eucalyptus. Then, the testunit was added to the main test sequences used by the Eucalyptus 3.2 development in our automated QA system, making the installation of the latest user console packages accessible by all developers at Eucalyptus.

This setup encouraged a failure in the user console package installation to be seen by any developers throughout the development, thus allowing the failure to be detected fast and reported with quickness.

Screen shot 2012-12-10 at 5.50.02 AM

The testunit ui_setup can be seen in action above in the table which displays the results of the test sequence ran by the automated QA system.. Check out the link below for more details of this testunit:

https://github.com/eucalyptus-qa/ui_setup

Circle of Trust

As the user console evolved out of its prototype state and took the form of a more product-like shape, the QA team was working in parallel, figuring out how to set up the automated testing process for the user console. The blog here talks in detail about how Selenium was used to create the automated web-browser testing tools, se34euca.

big-logo

In the mid-stage of the development, as the features of the user console started functioning in reasonably stable manners, 3 automated tests were added — incrementally — to ensure that the working state of the user console throughout the development.

Screen shot 2012-12-10 at 6.41.28 AMThose 3 tests are:

  1. user_console_view_page_testhttps://github.com/eucalyptus-qa/user_console_view_page_test
  2. user_console_generate_keypair_testhttps://github.com/eucalyptus-qa/user_console_generate_keypair_test
  3. user_console_launch_instance_test https://github.com/eucalyptus-qa/user_console_launch_instance_test

These automated tests were to ask the 3 simple questions below on a daily basis:

  1. Can the user log in and see all the landing pages on the latest user console?
  2. Can the user generate a new keypair using the latest user console?
  3. Can the user launch a VM instance using the latest user console?

Of course, it would be possible, and desirable, to ask more questions in a more complicated fashion. However, during the rapid development phase, asking those 3 simple questions on a daily basis, turned out to be sufficient, and effective, to understand whether something terrible had happened to the user console or not.

traffic_light

The goal of these automated tests at this stage of the development was not to detect every little defect in the product. Not too soon at the moment.

The main purpose is rather to serve as an indicator for the developers, QA engineers, and release engineers to assure ourselves that the change that went in the code earlier today did not ruin the delicate trust among the three groups, meaning that the build, installation, and configuration procedures are still in tact. Having such assurance in check by mechanical means has made the three groups extremely effective in discovering issues during the development since it allowed each member to narrow down exactly what was responsible for the defects in a finely reduced time frame, which was in hours, rather than days or weeks.

Guardrail For Development

Having the automated package build process and the automated installation/configuration process in place at the early stage of the development was proven to be extremely useful; rather than agreeing on the written procedures, the dev, QA, and release team materialized such agreements into the actual implementation, and put them into work by using various automated mechanics that run on a daily basis. Therefore, throughout the development, we were able to witness and assure ourselves that we were making progress in accordance with the plan and our self-imposed restrictions.

Check out the Eucalyptus Open QA webpage to see the continuous integration at Eucalyptus in action:

Eucalyptus Open QA (beta) – http://ec2-50-112-61-121.us-west-2.compute.amazonaws.com/open_qa.php

Advertisements

Quality Flow in Eucalyptus

When being interviewed at Eucalyptus, one is often asked, “when do you stop testing software?” This is not a trick question. As a matter of fact, this is not even a question; there is only one answer, and everyone knows it.

You never stop testing.

In Eucalyptus, we take the answer above very much literally.

Introduction

Eucalyptus has always displayed strong interest in creating an innovative, state-of-the-art software development workflow that supports four value-driven development practices: continuous integration, continuous testing, extreme automation, and open development. Based on these core principals, Eucalyptus engineering has designed an advanced quality control process, backed by a versatile automated QA system, whose function is to maintain the quality standard in Eucalyptus that is persistent and unchallengeable.

The graph below shows the overview of Eucalyptus software development workflow:

Graph 1. Quality Flow of Eucalyptus Software Development

Overview

Eucalyptus quality control process ensures that there exists monotonic increase in quality in the code as traveling through quality assurance check-points, known as Quality Gates, represented as green bars in Graph 1. Each quality gate is a set of automated operations performed by Eucalyptus’s automated QA system, which manages testing of Eucalyptus in a completely automated fashion — including bare-metal provisioning, installation and configuration of Eucalyptus, sequencing of automated tests, and providing feedback via various medium.

Eucalyptus quality control process is not just a quality assurance process; it is fully integrated into Eucalyptus software development. The process combines continuous integration and continuous testing into a unified workflow, which is powered by extreme automation via the QA system. The goal is to get rid of mundane tasks for developers so that they can focus on tasks that they are good at doing — developing a cloud infrastructure system of the highest quality.

Advantages

Guarantees in Quality

The branch #master (https://github.com/eucalyptus/eucalyptus) is the face of Eucalyptus open development model; every Eucalyptus engineer and community member checks out from the branch #master for development, experiment, or contribution. Thus, it is absolutely crucial that at all time, the quality of the branch #master is set to meet the standard of a functional cloud infrastructure system. In other words, at any given moment, a person should be able to check out from the branch #master and immediately start developing on top of the existing system without wasting time wondering if the system has any major problems that may influence further development. It is Eucalyptus’s responsibility to provide such guarantees in quality in the open branch #master. It is such guarantees in quality that encourage the community members to actively engage with Eucalyptus since it “defrictionalizes” contributions from merging into the main body of the code.

Commit Often and Fail Fast

In the early days, without formal guidance put in place, Eucalyptus has experienced a fair share of growing pain where development branches have drifted far, far away. The only known, effective solution to the “drifting development branch” problem is to have developers commit and check out the code as often as possible (continuous integration). However, the challenge in this practice is on how to minimize the effort that goes into this checking-in and checking-out process. The more tedious and complicated this process is, the more a developer is inclined to delay the process, causing his or her branch to be drifted far, far away. On the other hand, overly simplified commit-and-checkout process will severely jeopardize the quality of the code since a such process would likely permit unchecked mistakes to sneak into the main body of the code.

Eucalyptus quality control process attempts to address this issue by automating the entire “code-commit to quality-validation” process with a click of a button (extreme automation). As seen on TV, a developer can “just click it and forget it” when committing the code – that is until he or she gets a notification email indicating whether the commit has passed or failed to merge into the branch #master. The goal is to encourage developers to commit frequently by providing an easy mechanism that automatically performs quality verification on the code and returns fast feedback.

Procedures

In Eucalyptus software development workflow, a commit from a development branch to the branch #master is a two-step process. As seen in Graph 1., there is no direct path that leads to #master. Every commit must go through the aggregation branch called #testing. When the code on a development branch – from the far left stage on the graph – is committed to #testing, leaving the hand of a developer, the automated quality assurance process – represented as “Quality Gate 2” in the graph – will determine whether the code is allowed to move to #master. If the code fails to meet the quality standard enforced by Quality Gate 2, the branch #testing will be reverted to remove the history of the failed code. The purpose of this action is to bring the state of the branch #testing back to the last known stable quality. This action allows other developers to continuously commit to #testing even after previous commits in #testing fail to enter the branch #master.

The automated tests triggered by Quality Gate 2 is set to consume about 20 QA machines running tests for 5 hours. The cost of operation for Quality Gate 2 is very high. Therefore, there is a need for setting up a pre-commit check-up process — represented as “Quality Gate 1” in the graph above — in order to perform quick spot-checks on development branches. Such preliminary check-up process aims to minimize failures caused by preventable mistakes when committing to #testing.

Challenges

1. Scale and Resources

Ideally, every commit to the branch #testing must be examined and validated independently. However, in reality, with the limited resources and the frequency of commits, it becomes impossible to process all the commits on time if examined one at a time. Let’s say, in order to perform all the necessary automated tests kicked off by Quality Gate 2, it takes 20 QA machines running tests for 5 hours. Given this resource requirement, processing 5 commits individually would take 100 machines running tests for 5 hours in parallel, or the entire day if those commits end up completing for the resources. It is not difficult to see how fast the wait-line in the queue will grow for pending commits.

The current solution to this problem is batch-processing of commits at a scheduled time. There will be 2 or 3 times a day when Quality Gate 2 opens up to provide the opportunities for pending commits to “try out” for the branch #master. All the commits will be bundled together and go through the automated quality assurance testing as a group. If it passes, then all the commits will enter the branch #master. But, if it fails, then all the commits are rejected together, notifying each developer who submitted the commit accordingly. By examining the test-results report, attached to the notification email, the developers should communicate with each other to determine exactly what went wrong in the system and learn whose commit was responsible for the failure, which will allow the other, non-responsible developers to retry the commit.

2. Quality of Automated Tests

Eucalyptus quality control process counts on an extensive range of automated tests that cover all the essential functionality of a cloud infrastructure system. Those automated tests are designed to capture the accurate view of the target system. At the end of testing, there must be a clear “passed” or “failed” signal for each automated test with a high level of reliability. Any automated test that lacks such reliability provides no value in Eucalyptus quality control process since it would introduce noise when trying to examine the overall quality of the system.

To make the matters more complex, in agile-like development, today’s feature tests are tomorrow’s regression tests. In other words, in a such fast-paced development cycle, the automated tests written for testing new features in this release cycle will be used for detecting regressions in the next release cycle. In such continuous testing environment, combined with agile development, there exists a close relationship between the quality of the features and the quality of the automated tests; poorly written tests for a feature would result in a poorly maintained feature since such tests would fail to detect regressions as the development progresses. Thus, in order to develop good quality features, Eucalyptus quality control process must provide rapid ways to:

  • Ease the maintenance of existing automated tests (for detecting regressions)
  • Ramp up the maturity time of newly created automated tests (for validating new features)

Eucalyptus’s approach to these challenges is to develop a sophisticated testing framework, called “eutester“(https://github.com/eucalyptus/eutester), in order to enhance the automated test writing and maintenance experience.

3. Extremeness in Extreme Automation

Eucalyptus quality control process is glued together by complete automation. The entire process is put on auto-pilot that is guided by various automated pieces and modules working in accordance. Each independent component should have the ability to make decisions based on its own intelligence, meaning that it knows how to configure itself, run tasks, detect failures, examine the condition of the failures, recover from the failures, gather information, communicate with other components, and etc.. A collection of such intelligent components is essential to keep the quality process in motion with minimal interruptions. Now, the challenge is, how do we make it think like a QA engineer?

FYI. Eucalyptus is currently hiring talented Quality Assurance engineers who share the same vision and will power to attack these challenges (http://www.eucalyptus.com/careers/jobs).

%d bloggers like this: