Kyo Lee

Open-Source Cloud Blog

Tag: selenium IDE

Cloud App. Design: Create a Flexible Automated Web-UI Testing Tool using Selenium

In this article, I will go over the technical details on how in Eucalyptus, we used Selenium to simulate a large number of cloud user workload in order to ensure the quality of user experience in the new Eucalyptus user console.

As covered in my previous blog, Eucalyptus is coming out with a new user console, that is browser-based and intuitive to use, thus playing a key role in promoting the cloud adoption among IT organizations and enterprises.ย  But, the challenge was to ensure that this brand-new user console would be ready for handling the real-world workload when released out in the wild. The answer to this challenge was to simulate the activities of 150 cloud users using Selenium, an open source tool for automating web application testing.

How to Automate an Online User

The first step was to download Selenium IDE for Firefox. Selenium IDE is a must-to-have GUI tool for automating clicks and input-submits on a web application. After installing Selenium IDE on your computer, you can start Selenium IDE from Firefox’s Tools menu:

When started, Selenium IDE opens up its own separate window:

Notice the red dot on the top-right corner of Selenium IDE. When clicked, Selenium IDE will start recording all the activities you perform on the web-browser — every link you click and every input you type on the browser will be recorded as command-lines on Selenium IDE as seen below:

What Selenium IDE allows you to do is to replay the recorded activities, such as clicking and typing, on the browser in the exact same order that they were performed.

TIP.

But, soon you will notice that when replayed on Selenium IDE, it tends to fly through all the clicks in lighting speed so that the replayed activities often result in failures — the browser and web application cannot keep up with the speed of the clicks performed by Selenium IDE.

In order to prevent such cases, you will need to manually step through the record activities and insert various “pause-and-check” points using ‘waitForElementPresent’ command. For instance, when there is a command ‘click link=Delete’, I would put ‘waitForElementPresent link=Delete’ command prior to the click command to ensure that the page will fully loaded and the link ‘Delete’ is indeed present on the page before allowing Selenium IDE to execute the command ‘click link=Delete’. Later I learned that for every ‘click’ command, it is always a good habit to throw in the ‘waitForElementPresent’ command.

After verified that the recorded action is repeatable via Selenium IDE at its full speed, the next step is to export the action into a Selenium Python WebDriver format:

The result of the export above is a script file that describes the recorded Selenium action in a Python’s unittest format:

Once have the script exported, you can run the recorded action on a remote Selenium server without having to open up a web-browser. In other words, now you can simulate an online user doing the exact same recorded action on a web-browser by simply running the Python script generated by Selenium IDE.

Remote Selenium Server Configuration

Before running the script, you will need to configure a machine to run a remote Selenium server, which will behave like a web-client. The steps are on a Ubuntu machine, you will execute the following commands:

sudo apt-get -y update
sudo apt-get -y install default-jre
sudo apt-get -y install xvfb
sudo apt-get -y install firefox
sudo apt-get -y install python-pip
pip install selenium
Xvfb :0 -ac 2> /dev/null &
nohup java -jar selenium-server-standalone-2.25.0.jar &
export DISPLAY=:0


After running the commands above, you will have a Ubuntu machine capable of running the exported Python Selenium script, which then, simulates an online user opening up a Firefox browser and performing the recorded clicks.

Creating a Flexible, Reusable Testing Tool

Now, your task is to produce many exported Python Selenium scripts for all activities on the web application that will be used as building blocks for creating different user behaviors and workflows.

The first collection of Python Selenium scripts I produced was to visit every single landing page on the Eucalyptus user console. The second collection of Python Selenium scripts was to create cloud resources under the default setting. Having those two sets of Python Selenium scripts allowed me to construct complicated user interactions on the web application. For instance, with a bit of shuffling of the scripts, I could build up a user scenario where the online user would visit the keypair page, create a new keypair, visit the dashboard page, visit the security group page, create a new security group, revisit the dashboard page, and so on.

The next task was to consolidate all the scripts into one library file, getting rid of static values in the variables and breaking down the actions in the scripts into functions. Having a such unified library enables test-writers to stitch and arrange these functions together to construct whole new user scenarios as needed.

When examining the Eucalyptus user console test framework se34euca, you will see that the main library file ‘lib_euca_ui_test.py’ contains the functions that are exported from Selenium IDE, where each function describes a very specific action to perform on the web console. The files ‘testcase_*.py’ list the arrangements of those functions to form simple, or complex user behaviors. Finally, the files ‘runtest_*.py’ are the executables of those test cases that take input of the target web console environment.

Cloud Application

Now that you have a way to convert a Ubuntu machine into a Selenium server and have the Selenium test framework checked into a GitHub repository, you have a way to launch the Selenium test as a cloud instance — using se34euca as an example, the steps are:

Step 1. Launch a cloud instance on a Ubuntu image.
Step 2. Convert the Ubuntu image into a Selenium server by running the configuration commands above, or running the installer in se34euca.
Step 3. Git clone se34euca.
Step 4. Run the test case of your choice.
Step 5. Terminate the instance when the test is finished.

Of course, you can easily automate the step 2, 3, and 4 to wrap the entire process into a single scripted operation. Then, with a help of a cloud infrastructure, such as AWS or Eucalyptus, simulating 150 user can be as simple as launching 150 instances to run the script on each instance by feeding the parameter as user-data.

Code Reference

For those who are interested in creating a framework for testing your own web application, please feel free to check out the Eucalyptus user console test framework se34euca at:

https://github.com/eucalyptus/se34euca

for a reference, and leave a comment if you have any questions or suggestions.

Advertisements

Simulate 150 Cloud User Activities Using Open Source Tools

For the 3.2 release in this December, Eucalyptus is coming out with an intuitive, easy-to-use cloud user console, which aims to support the on-premise dev/test cloud adoption among IT organizations and enterprises.


This easy-to-use Eucalyptus User Console is consisted of two main components: a browser-side javascript application, written in JQuery, and a proxy server that utilizes Python Boto to relay requests to Eucalyptus Cloud, which is written inย Python Tornado, an open source version of the scalable, non-blocking web server developed by Facebook.

The target scale for the initial version of the user console is set to handle 150 simultaneous user activities under a single user console proxy.

Now, the challenge is how to simulate these 150 users to ensure that the user consoles and the proxy are able to withstand the workload of 150 active cloud users; more importantly how to ensure that such workload is not jeopardizing the user experience on the console.

One obviously answer is to find 150 people, train them thoroughly, and ask them to participate in the load testing. After all, 150 is doable.

However, what’s not doable is that having those 150 people to repeat the process over and over during the entire life cycle of the development until the release.

Then, the most “realistic” answer is to simulate those 150 people using machines. It turns out that the machines are really good at repeating the same things over and over, and they tend to behave in a very predictable manner when tuned properly.

At Eucalyptus, we use Selenium, open source web testing automation tools, to simulate the actual user interactions on the user console.

The steps are first, use Selenium IDE on Firefox to write an automation script that completes a single path of cloud user workflow — for instance, one simple user workflow is to log into the console, create a new keypair, and log out, and another workflow to log in, create a new volume, and log out. Second, repeat the first step above for all possible use cases to ensure that all, or most, of the functionality on the console are covered, allowing all use cases to be automatically executable via Selenium IDE. Third, export those automated IDE scripts to Selenium Python WebDriver format, which allows the automated scripts to run on a remote server without needing to actually opening up a browser. Finally, create a wrapper for each exported script so that each test case can be execute as a command-line tool on Linux.

The link below contains the collection of automated Selenium WebDriver test scripts, command-line tools, and their installer for testing the Eucalyptus User Console:

Se34Euca (Selenium34Eucalyptus) – https://github.com/eucalyptus/se34euca

With Se34Euca, you can instantaneously convert any machine — or virtual machine if you are already a cloud geek ๐Ÿ˜‰ — into a Eucalyptus cloud user simulator.

The steps are, on a Ubuntu image, run the commands below to install and setup Se34Euca:

sudo apt-get -y install git-core

git clone git://github.com/eucalyptus/se34euca.git

cd ./se34euca/script/

./installer_se34euca.py

Then, running the actual test can be as simple as:

export DISPLAY=:0

./runtest_view_page.py -i 192.168.51.6 -p 8888 -a ui-test-acct-00 -u user00 -w mypassword1 -t view_all_page_in_loop

The command line above will simulate a cloud user clicking through every single landing page on the user console within 2 second, then taking a rest for 5 seconds, and repeating the frantic, yet controlled clicking again and again and again.

However, funny enough, it turned out that the automated script’s ability to click through all pages on the user console within 2 seconds was well beyond the capability of a human user. The graph below renders the normal behavior of an actual human user. The X-axis in the graph shows the total length of TCP packets seen in a second on the user console proxy server machine via tcpdump.ย  Notice the peak in the beginning as the user logging in, and a group of little ripples that mark the user clicking buttons or viewing different pages in a 7-minute period:

And, the graph below shows the difference in the actual user behavior and the automated script behavior simulated by a single instance of Se34Euca. Notice the super-human strength of the automated script — the first half of the graph below is showing the same 7-minute period shown in the graph above. According to the graph below, the automated script is able to generate 10 times workload than a human user.

This discovery turns out to be good news; the fact that one Se34Euca instance can generate 10x human user workload, all I need to do is to launch 15 instances of Se34Euca to simulate 150 users. So, I provisioned 3 Ubuntu machines and launched 5 instances of Se34Euca on each machine:

The first fifth of the graph above covers the same period as the second graph above. What you are looking at is 15 instances of Se34Euca clicking through every single page on the Eucalyptus User Console for about two hours, starting at 21:00 mark.

When computed for averaging the packet length per second over 60 second observation period, the graph looks below:

The graph above is showing that when 150 users simultaneously logging in to the user consoles, the average packet transmission throughput rate seen on the wire is about 750Kb per second. Assuming that the user console proxy server is hooked on 1 Gig link, the throughput of 750Kb per second is certainly “doable” by all means. ๐Ÿ˜‰

Then, how do we ensure the user experience of the console?

Simple. While the user console proxy server is being slammed by 150 click-monkeys, I’m opening up my own browser to verify that my interaction with the console is smooth as usual. ๐Ÿ™‚

On my next blog, I will cover more details on the exact setup of the Eucalyptus User Console load-testing, including the selenium scripts and monitoring setup, and dig deeper into the analysis of the data. Please, stay tuned ๐Ÿ˜‰

Meanwhile, feel free to check out the blog below if you would like to preview the Eucalyptus User Console for yourself:

http://coderslike.us/2012/11/11/installing-the-eucalyptus-console-from-source-and-packages/

%d bloggers like this: