Kyo Lee

Open-Source Cloud Blog

Tag: monitoring

TCP Dumpster: Monitoring TCP for Eucalyptus User Console

This is the part III of the Eucalyptus Open QA blog series that cover various topics on the quality assurance process for Eucalyptus’s new user console.Eucalyptus User Console

On this blog, we would like to share the information on how we monitors the traffic on the user console proxy, using the Linux command ‘tcpdump‘ and its rendering application ‘tcpdumpster‘, to derive and understand the behaviors of users when interacting with the user console.

Background

Eucalyptus user console consists of two components: javascript-based client application and Tornado-based user console proxy. When logged in, the client-side application, which runs on a user’s web-browser, polls the user’s cloud resource data at a certain interval, and the user console proxy, located in between the cloud and the users, relays the requests originated from the client applications.

userconsoleconponentview

Recalling from the first blog of the series, our challenging question was, when 100+ users are logged into the Eucalyptus user consoles at the same time, would the user console proxy be able to withstand the traffic that was generated by those 100+ users? Plus, how do we ensure the user experience under such heavy load?

The answer to the questions above was provided in details here.

The short answer is to generate 100+ user traffic using the automated open-source web-browser testing tool, Selenium, while manually evaluating the user experience on the user console.

However, prior to answering the questions above, first we needed to establish a way to quickly, yet effectively monitor the traffic between the clients and the proxy in order to make observations on the patterns and behaviors of the traffic.

TCP Dump

tcpdump‘ is a standard tool for monitoring the TCP traffic on Linux. For instance, if the user console proxy was running on the port 8888 on the machine 192.168.51.6, monitoring the traffic on the port 8888 can be as simple as running the command below on the Linux terminal at 192.168.51.6:

tcpdump port 8888

Then, this command will “dump” out the information on every packet that crosses the port 8888 on the machine 192.168.51.6. However, the information generated by this command is just too overwhelming; such information would fly by on the terminal screen as soon as the user consoles start interacting with the proxy. There had to be a better way to render the output of the command ‘tcpdump‘.

 TCP Dumpster

At Eucalyptus, using the automated QA system, a new, up-to-date Eucalyptus system is constantly installed and torn down within a day or two life span (check out here to see the Eucalyptus QA system in action). For this reason, we needed to come up with a quick way to set up the monitoring application on the machine where the proxy was installed. Plus, we would like to have all necessary monitoring information displayed on a single HTML page for a quick glance, thus making it easier for the observer to apply intuition on understanding the big picture. As a result, ‘tcpdumpster‘ was born.

Picture 96

The application ‘tcpdumpster‘ runs on the same machine where the proxy is installed. It runs the Linux command “tcpdump port 8888” and parses its output into a list file. This list tracks 8 attributes of the TCP traffic:

  • Unique connections, based on IP
  • Unique connections, based on Port
  • Connection count, per second
  • Connection count, averaged over a minute
  • Connection count, in total
  • Packet length, per second
  • Packet length, averaged over a minute
  • Packet length, in total

With those 8 attributes displayed on a single HTML page, which can be accessed via:

http://192.168.51.6/tcpdumpster.php

, we were able to make some interesting observations on the behaviors of the traffic as the user console starts interacting with the proxy.

TCP Dumpster Examples

The graph below is showing the traffic pattern for 7 minutes, generated by a user logged in to the user console.

Picture 18

Notice the first peak that represents the log-in of the user, followed by the periodic peaks that show the polling of the cloud resource data, and user actions can be seen in the blobs among the peaks.

The graph below is showing the traffic pattern as more selenium-based automated scripts are activated to simulate a large amount of users.

Picture 46

The first block shows when 1 and 2 Selenium scripts are active, and the second block shows when 6 and 12 Selenium scripts are active (check out here to learn how Selenium was used). When graphed for averaged over a minute, the differences between the stages become more visible:Picture 47

When graphed all together, along with the connection data, they look below:

Picture 45

tcpdumpster‘ turns out to be very useful when validating if a newly written selenium script is behaving correctly. The graph below shows the selenium script that launches a new instance, waits until the instance is running, then terminates the instance, waits a few minutes, and repeats:

Picture 81

And, of course, ‘tcpdumpster‘ is very handy when you are running a longterm test; it allows me to set up the test, go to sleep, and wait up the next day to check out the results. The graph below shows how the proxy was able to withstand the constant ‘refresh’ operations from multiple connections for longer than 5 hours:

Picture 94

Now, can you guess what is going on in the graph below?

Picture 105

Check out the GitHub link below and try out ‘tcpdumpster‘ on your own Eucalyptus user console proxy to find out for yourself:

https://github.com/eucalyptus/tcpdumpster

Advertisements

Pigeons on a Euca: Eucalyptus Cloud Monitoring Mobile App via Twitter

Being a system administrator is the easiest job in the building when the system is working; no one questions your presence nor existence. Your tasks are highly under-appreciated during the time of peace, yet you do not mind for such vanity since you’d rather be reading blogs and watching youTubes in serenity. Every once in a while, an idiot cracks an ancient joke, “hey, aren’t you supposed be working?”. But, I’m working, you imbecile employ-of-the-month.

However, the curse begins once you step out of the building, entering the realm of unknowns, far-disconnected from the comfort of your Macbook and Wi-Fi. While staring at endless tail-lights on a freeway, having a long walk on the beach, or being queued behind 7 shopping carts at a grocery market, your mind begins to wonder, “how’s my system doing?”

Since the day 1, you have set up numerous layers of email-notification alarms, but it’s never enough; “getting the e-mails” only means “it’s too late”. Always there is an urge of logging in. But you can’t. You are cut off. You are trapped; the lady in front of you just pulled out a checkbook while the sign clearly says “credit or cash only.” You begin to panic. You compulsively refresh emails on your smartphone, but no answers. Silence is deafening. No news is never the good news. The only exit is when the system whispers in your ear: “Have no worries. Everything is working… For now.”

Now, your concerning days are over. In the midst of the 3G wilderness, the application Pigeons on a Euca will deliver you peace and tranquilly that are comparable to those of a laptop on VPN. Of course, that is if you are equipped with a smartphone at all times and the system is running Eucalyptus Cloud.

The trick is to run a periodic Cloud monitoring app via Twitter.

Instead of being passively notified by emails when there arise problems in the system, you can set up the application Pigeons on a Euca that runs a small script that actively “tweets” the status of the cloud for you and your co-sys.admins to follow.

Screenshot of Pigeons on a Euca on iPhone

Here are the requirements:

  • Have a twitter account opened for this application.
  • Have a machine, or a virtual machine, running Linux with network capability.
  • Have the cloud admin’s credentials.
  • Have a smartphone with Twitter Client App installed.

Current (Beta) Features**:

  • In every 1 min, it tweets the status-change on running instances*.
  • In every 10 min, it tweets the number of currently running instances in the cloud*.
  • In every 10 min, it tweets the number of newly-launched instances in the cloud*.
  • In every 10 min, it tweets each availability zone information

* These features rely on the new version of euca2ools (v 2.0)

** The application is highly configurable so that more reporting can easily be added when needed.

Instructions on How to Set up the Application Pigeons on a Euca

First, you need to open a new twitter account.

1. Go to twitter.com and sign up for a new account; if you already have an account with Twitter, you are going to need a new email address to create a new twitter account for this application.

2. As soon as the new twitter account is open, check the “Protect my Tweets” box on the Tweet Privacy section so that your tweets do not accidently get broadcast to the public channel.

3. Apply for a developer account at dev.twitter.com.

4. After opening the twitter developer account, you need to create an application at dev.twitter.com.

5. Fill out the application details form. There is no requirement on what you put in the name and description boxes. For the website box, you may specify any working web URL of your choice; it won’t matter for this application.

6. After creating the application, on the “settings” page, change the access level to be “Read and Write

7. Click the button “Change this Twitter application’s settings” at the bottom of the page to apply the change. It might take a few minutes for the change to be applied.

8. On the “Details” page, verify that the access level is changed to “Read and write“. After seeing that the change has taken place, click on the button “Create my access token” at the bottom of the page.

9. Go to the page “OAuth tool” and verify the consumer key and access token are generated. You will need these keys to configure the application Pigeons on a Euca later.

10. At this point, your twitter account is configured to receive script-generated tweets from the application Pigeons on a Euca.

Second, after the twitter account is ready, you need to set up the machine where the application Pigeons on a Euca will be running on.

1. Install a perl module “Net::Twitter::Lite” on your Linux box.

You may install the module from source by visting the link:

http://search.cpan.org/~mmims/Net-Twitter-Lite-0.10004/lib/Net/Twitter/Lite.pm

Or, for UBUNTU distributions, such as Lucid, you may simply add the line:

deb http://ubuntu.mirror.cambrium.nl/ubuntu/ lucid main universe

to “/etc/apt/source.list”.

Then, install the perl module twitter-lite-perl by using the commands:

apt-get update
apt-get install libnet-twitter-lite-perl

2. Install the latest version of “euca2ools” (v 2.0)

Visit the website (http://open.eucalyptus.com/downloads) for detailed instructions on how to install the latest euca2ools.

For UBUNTU distributions, such as Lucid, you add the line:

deb http://downloads.eucalyptus.com/software/euca2ools/2.0/ubuntu lucid universe

to “/etc/apt/source.list”

Then, install euca2ools by using the commands:

apt-get update
apt-get install euca2ools

Third, after all the necessary modules and tools are installed on the machine, now you can finish setting up the application on the machine.

1. Download the tarball pigeons_on_a_euca.tar.gz from the project repository:

https://projects.eucalyptus.com/redmine/projects/pigeons-on-a-euca/files

or

https://github.com/eucalyptus/pigeons_on_a_euca

2. Untar the tarball at a directory of your choice:

tar zxvf pigeons_on_a_euca.tar.gz

3. On the “my applications” page on dev.twitter.com, copy the lines in the “OAuth tool” section, as shown in the Step 9. of the first instruction set.

4. Change the lines in the file “./pigeon_on_a_euca/pigeon_cage/key/o_auth_settings.key” with your account’s actual values.

5. Perform a quick check to validate the setups so far by running the commands:

cd ./pigeon_on_a_euca/pigeon_cage

perl ./tweet_it_away.pl ./tweets/mytest.tweet

6. Check the twitter account to verify that the line “this is a test” was tweeted. Also notice the lock sign on the tweet that indicates the security level is private.

7. After verifying the test tweet, go to the directory “./pigeons_on_a_euca/credentials” and store your Eucalyptus cloud’s admin credentials.

8. Verify that you can talk to your Eucalyptus cloud via the admin credentials by running the commands:

cd ./pigeons_on_a_euca/credentials

source eucarc

euca-describe-availability-zones verbose

9. At this point, the application is all set to run. Do a quick check by running the main script:

perl ./activate_the_pigeons.pl

10. Check the twitter account to verify that the status of instances running on the cloud are being tweeted.

11. To run the main script in the background, do:

nohup perl ./activate_the_pigeons.pl > ./stdout.log >> ./stderr.log &

12. To monitor the run:

tail -f stdout.log

Last, install any Twitter Client App on your smartphone and follow the account that you created above.

Now you have an mobile application that keeps you updated with the status of the cloud.

Warning: The amount of tweets generated by the application might be overwhelming; at its maximum rate, it will upload 350 tweets per hour. It is recommended that you and your co-workers open a separate twitter account exclusively for receiving tweets from this application.

And, if you decide to modify the script, please be aware of the hourly limit of the tweet updates, which is set to be 350 tweets per hour. Carefully limit your tweets so that the application maintains consistent tweet-ability.

Thank you for your interest in the application, and feel free to contribute and share.

Kyo

%d bloggers like this: