10 Steps to Euca Monkey
Euca Monkey is an easy-to-deploy test tool designed for performing stress-test on Eucalyptus Cloud. The tool repeatedly generates and tears down 6 types of cloud user resources: running instances, volumes, snapshots, security groups, keypairs, and IP addresses. As the resources are being populated and released, the cloud is actively queried to validate such resources are indeed being allocated correctly per request. Then, the tool renders the progress of the stress testing using Gnuplot, an open source graphing tool, and displays the graphs as webservice in real time.
Euca Monkey uses cloud-resource-populator — which utilizes Eutester, which is based on Boto, thus making the tool AWS-API-compatible — to populate and release resources from Eucalyptus as a user. The input for cloud-resource-populator looks below:
[USER INFO] account: ui-test-acct-23 user: user-23 password: mypassword23 [RESOURCES] running instances: 2 volumes: 2 snapshots: 1 security groups: 10 keypairs: 3 ip addresses: 2 [ITERATIONS] iterations: 200
With the given input above, cloud-resource-populator will generate the resources as specified in the [RESOURCES] section as the user 'user-23' under the account 'ui-test-acct-23'. When viewed from Eucalyptus user console, it will look as below:
As soon as the resources are populated according to the specification, cloud-resource-populator will immediately send requests to the cloud to release all the allocated resources, which makes the console view look as below:
And, as you would have guessed, the process of populating and releasing of the resources is repeated for [ITERATIONS] times.
The most appealing feature of Euca Monkey is that it launches a webservice to render the progress of the stress-testing in real time.
The graph above is showing the input values of a few iterations of the resource population and tear-down process. When tracing the running instance line, which is in red, this graph is telling us that 20 instance were started on the first request, then those 20 instances were all terminated on the second mark, thus brought the count down to 0. And, 20 instances were started again, then terminated, and so on. Such operations were repeated 7 times on the graph above. And, also notice that there are 4 other resources being populated and released 7 times as well.
When the cloud is behaving nicely, the actual resources should be populated and released in match with the input values in the graph above, thus resulting in the output graph as below:
While the first graph shown above renders the input values to the tool cloud-resource-populator, this graph is showing the actual values reported by the cloud. The fact that these two graphs look the same means "Yay Cloud!!"
However, occasionally, during the development of Eucalyptus, you would see the graph like below on a rainy day in Santa Barbara:
The graph above reveals an interesting state of the cloud. Notice that the "running instance" line went from the "nice" behavior pattern to a flat line. It means that the cloud was able to launch and terminate 20 instances during the first phase, but somehow it got stuck to the state where it was not able to release instances, nor launch more instances, thus stuck with 18 running instances. But, notice that other resources were making the usual progress as before, except the security groups. It turned out what we were witnessing was the deadlock case in Eucalyptus Cluster Controller, which occurred around the 8th hour of the stress-testing. R.I.P CC. :(
The purpose of stress-testing is to push the limit of the system to the point where malfunctions and faulty behaviors of the system can be observed. Such stress testing is crucial for the development of a distributed system like Eucalyptus; many unknowns and bugs are constantly introduced to the system as new features from various components are being integrated. Thus, having an easily deployable stress-testing tool with visualization support, such as Euca Monkey, yields tremendous benefits for the developers in an agile development environment since the tool aims to ensure the system's stability and reliability throughout the rapid development cycle.
If you would like to take Euca Monkey for a spin, feel free to check out the GitHub link below. On a fresh Centos 6 machine or VM, it will take 10 simple steps to launch your own monkey.
Eutester - https://github.com/eucalyptus/eutester
Boto - https://github.com/boto/boto
Eucalyptus QA - https://github.com/eucalyptus-qa
Open QA for Eucalyptus - https://github.com/eucalyptus/open-qa