Kyo Lee

Open-Source Cloud Blog

Tag: Linux

Docker Strange Dev: How I learned to stop worrying and love the VM

A couple of months ago, my laptop died. I remember the day I brought it home for the first time. Still in college. I made a big decision—despite of my marginal finances—to go with the top model. “It should last at least five years,” I convinced myself. Seven years later, I got the news from the certified Apple repair person, “Well, you should just buy a new laptop than trying to fix this one.” So I did.

Now, I have a new laptop, which is currently placed on a pedestal. Watching how fast applications open and close on this laptop makes me cry. The future has arrived. I tell myself, “No! I will not f*** this one up this time! I will never install any software on this laptop ever!” Of course, I am in denial. This pure awesomeness will eventually decay. I predict the rate of decay exponentially accelerated by the number of software installed on the laptop (Disclaimer: No supporting data exists for this assertion—other than my paranoia).

I begin searching for the answer to the quest: Install no software. So I try Docker.

Docker_logo

The noticeable difference between using Docker, which runs Linux container, and using the virtual machine (VM) is speed.

Both options provide the ability to isolate software’s running environment from the base operating system. Thus both deliver the desirable paradigm: Build once and run everywhere. And both address my concern—that the base operation system must remain minimal and untouched. However, Docker stands out from the classic VM approach by allowing the cloud application—which is a virtual image with desired software installed—to run within milliseconds. Compare that to the minutes of time it takes to boot a VM instance.

Speed_Racer

What significance does this difference make?

Docker removes a big chunk of mental roadblock for developers. Thanks to Docker’s superior response time, developers can barely identify the perceptual distinction between running applications on a virtual image (a virtual container, to be precise) and running applications on the base OS. In addition to its responsiveness, Docker appeals to the developers by obsoleting the tedious procedures of booting the VM instance and managing the instance’s life cycle. Unlike previous VM-centric approach, Docker embraces the application-centric design principal. Docker’s command line interface (CLI) below asks us two simple questions:

1. Which image to use?

2. What command to run?

docker run [options] <image> <command>

With Docker, I can type single-line commands to run cloud applications, similar to other Linux commands. Then I will have my databases servers running. I will have my web servers running. I will have my API servers running. And I will have my file servers running. All these are done by using single-line commands. The entire web-stack can now be running on my laptop within seconds. The best part of all this? When I’m done, there will be no trace left on my laptop—as if they never existed. The pure awesomeness prevails.

docker-whaleeuca_new_logo

Running Eucalyptus Console on Docker

 

For those who want to check out Eucalyptus Console to access Amazon Web Services, here are the steps to launch Eucalyptus Console using Docker on OS X.

Step 1. Install Docker on your laptop

Here is a great link that walks you through how to install Docker on OS X:

http://docs.docker.com/installation/mac/

Step 2. Pull Eucalyptus Console Docker image repository

Run the command below to pull Eucalyptus Console Docker images (it will take some time to download about 1.5 G image files):

docker pull kyolee310/eucaconsole

Run the command below to verify that the eucaconsole images have been pulled:

docker images

Screen Shot 2014-09-14 at 10.52.11 PMFor those who want to build the images from scratch, here is the link to the Dockerfile used:

https://github.com/eucalyptus/dockereuca

Step 3.  Update Docker VM’s clock

When running Docker on OS X, make sure that OS X’s clock is synchronized properly. A skewed clock can cause problems for some applications on Docker. In order to fix this issue, you will need to log into the Docker VM and synchronize the clock manually.

You can SSH into Docker VM using the command below:

boot2docker ssh

Once logged in, run the command below to sync the clock:

sudo ntpclient -s -h pool.ntp.org

Run the command below to verify that the clock has been sync’ed

date

One more patching work to do is to create an empty “/etc/localtime” file so that you can link your OS X’s localtime file to Docker VM’s localtime file at runtime:

sudo touch /etc/localtime

Exit the SSH session:

exit

This issue is being tracked here:

https://github.com/boot2docker/boot2docker/issues/476

Screen Shot 2014-09-14 at 10.53.04 PM

Step 4. Launch Eucalyptus Console via Docker

Run the command below to launch Eucalyptus Console on Docker:

docker run -i -t -v /etc/localtime:/etc/localtime:ro -p 8888:8888 kyolee310/eucaconsole:package-4.0 bash

It’s a shame, but running a live “bash” session is not Docker’s way of doing things, but excuse me for the moment until I figure out how to run Eucalyptus Console properly without using the “service” command.

The command above will open a bash shell session for the eucaconsole image, then run the command below to launch Eucalyptus Console:

service eucaconsole start

Step 5.  Open Eucalyptus Console on a browser

Run the command below to find out the IP of Docker VM:

boot2docker ip

, whose output would look like:

    The VM’s Host only interface IP address is: 192.168.59.103

Using the IP above, access Eucalyptus Console at port 8888:

ex. http://192.168.59.103:8888/

Screen Shot 2014-09-14 at 8.54.09 PM

In order to access AWS, you will need to obtain your AWS access key and secret key. Here is the link by AWS on howto:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html

Step 6. Log into AWS using Eucalyptus Console

Screen Shot 2014-09-14 at 8.54.28 PM

Advertisements

Implication of Fragmentation in Linux

At Eucalyptus we have been proud to say,

“Eucalyptus Cloud runs on almost all major Linux distros: Ubuntu, Debian, CentOS, Red Hat, Fedora, etc.. You name it, we will support it!”

The crowd erupts in cheers, with occasional tears of joy. There will be a parade later.

Yes, it sounds wonderfully majestic as it should be.

When developing open-source software, you need to support all major, stable Linux platforms so that the software can reach out to every single open-source enthusiast who is often loyal to a certain flavor of Linux.

But, what does all this mean to software developers?

A nightmare.

A nightmare accompanied with a horrible migraine.

In the ideal software utopia, all Linux platforms behave in an identical manner; you should be able to run things on Fedora in the same exact way you run those things on Ubuntu. After all, they are all “Linux”, aren’t they?

Welcome to the harsh reality called “fragmentation” in the software world.

It is true that all those distros feel like Linux; they share the same core — Linux kernel — and provide the same level of abstraction which can be described as “Linux experience.”

But, in reality, no two Linux distros are never the same.

The biggest problem with this inconvenient truth is that no one knows for sure what the exact differences are when going from one distro to another distro.

Let’s say we use CentOS as a default Linux distro for developing and testing software. If every function and feature works well on CentOS, can we safely assume that the software will also behave nicely with other Linux distros, such as Red Hat, Ubuntu, and Debian?

The answers to this simple question always fall somewhere in between “Maybe”, “It depends”, “Possibly yes”, “Theoretically it should”, “What is Linux?”, and “NO!”

The only assured way to discover the correct answer is to run tests on the software under all distros.

At present Eucalyptus officially supports three main Linux distros — Ubuntu, Redhat, and CentOS — and is working on adding two more distros: Debian and Fedora. Every time a new distro is added, we are to repeat the entire set of test suite for the new distro. If running a whole set of test suite takes X amount of resources, supporting 5 distros would mean 5X resources. Even further, per distro, we support two of the latest versions — for instance, for CentOS, Eucalyptus supports its version 5 and 6. This additional requirement brings up the total amount testing resources to be 10X.

When translated to the operation cost, if running a complete set of test suite under one Linux environment takes one day, due to the fragmentation of Linux distros, we need to add nine more days of testing in order to completely cover all corner cases in various Linux distros.

The real world implication of this nightmare is that whenever a little tweak goes into Eucalyptus, it might take up to 10 days, in the worst case scenario, to ensure ourselves that this seemingly innocent tweak will not bring down the house under some other Linux distros in strange, unpredictable ways.

In school we are taught to celebrate diversity, but they often forget to emphasize the beauty in simplicity. Handling the issue of the Linux fragmentation remains to be one of many challenges that Eucalyptus has to overcome.

%d bloggers like this: