Simulate 150 Cloud User Activities Using Open Source Tools
For the 3.2 release in this December, Eucalyptus is coming out with an intuitive, easy-to-use cloud user console, which aims to support the on-premise dev/test cloud adoption among IT organizations and enterprises.
The target scale for the initial version of the user console is set to handle 150 simultaneous user activities under a single user console proxy.
Now, the challenge is how to simulate these 150 users to ensure that the user consoles and the proxy are able to withstand the workload of 150 active cloud users; more importantly how to ensure that such workload is not jeopardizing the user experience on the console.
One obviously answer is to find 150 people, train them thoroughly, and ask them to participate in the load testing. After all, 150 is doable.
However, what’s not doable is that having those 150 people to repeat the process over and over during the entire life cycle of the development until the release.
Then, the most “realistic” answer is to simulate those 150 people using machines. It turns out that the machines are really good at repeating the same things over and over, and they tend to behave in a very predictable manner when tuned properly.
At Eucalyptus, we use Selenium, open source web testing automation tools, to simulate the actual user interactions on the user console.
The steps are first, use Selenium IDE on Firefox to write an automation script that completes a single path of cloud user workflow — for instance, one simple user workflow is to log into the console, create a new keypair, and log out, and another workflow to log in, create a new volume, and log out. Second, repeat the first step above for all possible use cases to ensure that all, or most, of the functionality on the console are covered, allowing all use cases to be automatically executable via Selenium IDE. Third, export those automated IDE scripts to Selenium Python WebDriver format, which allows the automated scripts to run on a remote server without needing to actually opening up a browser. Finally, create a wrapper for each exported script so that each test case can be execute as a command-line tool on Linux.
The link below contains the collection of automated Selenium WebDriver test scripts, command-line tools, and their installer for testing the Eucalyptus User Console:
Se34Euca (Selenium34Eucalyptus) – https://github.com/eucalyptus/se34euca
With Se34Euca, you can instantaneously convert any machine — or virtual machine if you are already a cloud geek ;) — into a Eucalyptus cloud user simulator.
The steps are, on a Ubuntu image, run the commands below to install and setup Se34Euca:
sudo apt-get -y install git-core
git clone git://github.com/eucalyptus/se34euca.git
Then, running the actual test can be as simple as:
./runtest_view_page.py -i 192.168.51.6 -p 8888 -a ui-test-acct-00 -u user00 -w mypassword1 -t view_all_page_in_loop
The command line above will simulate a cloud user clicking through every single landing page on the user console within 2 second, then taking a rest for 5 seconds, and repeating the frantic, yet controlled clicking again and again and again.
However, funny enough, it turned out that the automated script’s ability to click through all pages on the user console within 2 seconds was well beyond the capability of a human user. The graph below renders the normal behavior of an actual human user. The X-axis in the graph shows the total length of TCP packets seen in a second on the user console proxy server machine via tcpdump. Notice the peak in the beginning as the user logging in, and a group of little ripples that mark the user clicking buttons or viewing different pages in a 7-minute period:
And, the graph below shows the difference in the actual user behavior and the automated script behavior simulated by a single instance of Se34Euca. Notice the super-human strength of the automated script — the first half of the graph below is showing the same 7-minute period shown in the graph above. According to the graph below, the automated script is able to generate 10 times workload than a human user.
This discovery turns out to be good news; the fact that one Se34Euca instance can generate 10x human user workload, all I need to do is to launch 15 instances of Se34Euca to simulate 150 users. So, I provisioned 3 Ubuntu machines and launched 5 instances of Se34Euca on each machine:
The first fifth of the graph above covers the same period as the second graph above. What you are looking at is 15 instances of Se34Euca clicking through every single page on the Eucalyptus User Console for about two hours, starting at 21:00 mark.
When computed for averaging the packet length per second over 60 second observation period, the graph looks below:
The graph above is showing that when 150 users simultaneously logging in to the user consoles, the average packet transmission throughput rate seen on the wire is about 750Kb per second. Assuming that the user console proxy server is hooked on 1 Gig link, the throughput of 750Kb per second is certainly “doable” by all means. ;)
Then, how do we ensure the user experience of the console?
Simple. While the user console proxy server is being slammed by 150 click-monkeys, I’m opening up my own browser to verify that my interaction with the console is smooth as usual. :)
On my next blog, I will cover more details on the exact setup of the Eucalyptus User Console load-testing, including the selenium scripts and monitoring setup, and dig deeper into the analysis of the data. Please, stay tuned ;)
Meanwhile, feel free to check out the blog below if you would like to preview the Eucalyptus User Console for yourself: