Kyo Lee

Open-Source Cloud Blog

Category: web-service

DevOps Culture — Fail Fast on Eucalyptus

sandiegozoo
At a meetup event down in San Diego, California, Eucalyptus had a chance to meet Sander van Zoest (@svanzoest), the VP of technology at OneHealth (http://www.onehealth.com/), who is also the organizer of the San Diego DevOps group (http://www.meetup.com/sddevops/). Sander and his team at OneHealth have been using Eucalyptus cloud for some time. Asked why OneHealth runs Eucalyptus in-house, Sander had some interesting stories to say about dealing with health-related data and the company’s DevOps engineering culture.

onehealth

Due to the strict regulations on Protected Health Information (PHI), OneHealth needs to take extra strong measures if they are to provide the services on AWS; Sander spent a good amount of time explaining to us how demanding it is to satisfy the regulations. Such barriers make things complicated to push any personal identifiable health information to the cloud.

For the AWS case, the very specific barrier was that AWS provides no legal protection when storing sensitive data in the cloud storage space. For instance, it is required by HIPAA and HITECH regulations that One Health needs to be able to promise a 72-hour response time to inform their customers about the breach of the data, should it ever happen, and provide an ETA to identify and patch the security hole that caused the breach.

Sander points out that at the moment, AWS does not guarantee such protections/services. For this reason, OneHealth’s production environment is deployed at Rackspace’s co-location since it provides HIPAA Business Associate Addendums. However, it is noted that given the evolving nature of the public cloud, it is very “cloudy” to predict how things are going to change in the near future. The recent announcement by AWS on CloudHSM (http://aws.typepad.com/aws/2013/03/aws-cloud-hsm-secure-key-storage-and-cryptographic-operations.html) — although it doesn’t cover the legal protection — is a good indicator showing AWS’s interest in providing secure storage service as moving forward.

cloudy2

What this uncertain, “cloudy” future means for engineering at OneHealth is employing a variety of infrastructure environments to take advantage of each platform while staying flexible. It becomes essential to design OneHealth’s services and applications to be deployable on bare-metal systems at Rackspace (production environment), AWS (sandbox/staging environment), Eucalyptus (in-house continuous integration and testing environment), and engineers’ laptops using Vagrant (development and testing environment).  (http://www.vagrantup.com/).

Under such heterogeneous systems, from its production down to the engineer’s laptop, the development environment — the OS, dependencies, configurations, etc — needs to be kept uniform via virtualization and automation, allowing seamless pushing of new code from the laptop up to the production. For handling the life cycle of machines and VM instances, the engineers at OneHealth are big fans of Chef (http://www.opscode.com/chef/), which makes the configuration management portable on any infrastructure platforms. For virtual machines, the instance images are prepared via debian preseed files while leveraging a open source tool VeeWee (https://github.com/jedi4ever/veewee).

chefchef_icon

At OneHealth, the philosophy of DevOps is deeply embedded in every aspect of its development and operation. The concept of DevOps was not new to many engineers who brought in the ideas of “Infrastructure as Code” and “Commit Often and Fail Fast” from previous companies such as MP3.com and Joost.

Speaking of DevOps culture, one fun fact Sander mentioned — which goes against intuition for many traditional IT shops — was that the operation team at OneHealth likes to take down the instances and rebuild them regularly. The recycling of the instances ensures the “freshness” of the deployed services and applications. The operation engineers should be more concerned if an instance’s uptime was longer than, say, 30 days because it meant that the content of the instance was outdated, possibly containing unfixed bugs or security issues. If the deployment setup was doing what it was supposed to be doing, then it should have killed the outdated instance and brought up a new instance with the latest updates.

The same goes for the development environment. It would be much better to refresh the dev environment instances with frequent relaunching and reconstructing than having the developers working on a stale dev environment, which turns out to be more harmful for the development. Plus, this destroy-and-rebuild enforcement encourages the developers to consistently check in the code to a version-controlled code repository, allowing early detection of conflicts in code.

All of these procedures, bringing together datacenter automation and configuration management, are part of a very new movement in software development now labeled as “DevOps”. The DevOps folks often joke around and say even a few years ago, the terminology didn’t even exist, but now, DevOps has become the most sought-after practice in IT. All thanks to the wide spread of cloud computing, giving birth to the programmable infrastructure.

euca_new_logo

Advertisements

Allow Everyone to Launch Your Web-Service: Open Source Web-Service using GIT and AWS

Open Source is the future of software.

The statement above is self-evident at this point of history — I mean, have you checked out www.microsoft.com/opensource recently?

Then, what does open-source mean when it comes to web-services?

Open Source has been a well-understood concept among the web-application community due to the neat feature “View Source” that came with almost all web-browsers. Copy-and-pasting a chunk of HTML code from one website to another was a de facto standard way of developing websites. Whether admitted it or not, web-application development is deeply rooted in open source culture.

But, where should this open source culture be heading to? Should we keep maintain the culture of copy-and-pasting others’ code with no telling? Or, rather try to embrace this open source culture and explore the possibilities that arise from sharing code?

Then, it occurred to me that, “What if we could share the entire code for web-services?”

Imagine if anyone could launch web-services such as E-bay, Craigslist, Amazon, Angie’s list, Yelp, Instagram, Twitter, etc on their own instances on AWS. Anyone could instantly replicate all the functionality and services provided by such known web-services at its micro scale. If the big name web-services are the department stores in downtown, these open-source web-services on AWS instances are the mom-and-pop stores for the local community, or they could be the black market since these services could form and disappear in a rather unpredictable manner — similar to the life cycle of AWS instances. 😉

Think about the scenario where a person can launch his/her own e-bay online auction service on an AWS instance for a school fund-raising event for a month. The person might also want to launch an instance of Craigslist for the same event, or Instagram and Twitter to keep people engaged during this period. Then, once the event is over, all the services can go away as if nothing ever happened.

Welcome to the era of Cloud Applications.

To test out the concept, I used the application “Open QA“, a web-service that is designed to display the test results of the development progress at Eucalyptus. The goal is to allow any Eucalyptus community member to launch the same service on AWS instances.

First, the web-service has to be broken down into two bodies, the service layer and the data. Then, each body of the web-service needs to be available in the open, allowing anyone to download the code and convert the AWS instances to Open QA web-service instances.

This is where GIT comes in very nicely. For public repositories, GIT will allow you create as many branches as you want, which can serve as read-only code repositories for anyone on the Internet without hassle.

The service layer part of Open QA is available at:

https://github.com/eucalyptus/open-qa

The data for Open QA is available at:

https://github.com/eucalyptus/open-qa-frontend-data

Upon launching an instance on AWS, a user can simply convert the raw Amazon Linux instance into the Open QA web-service by executing 4 commands below:

sudo yum -y install git
git clone git://github.com/eucalyptus/open-qa.git
cd ./open-qa/script/
./open_qa-installer.py -t amazon -e <your_email>

At this point, the AWS instance is running a HTTPD service with Open QA PHP code in its /var/www/html directory.

However, with only the service layer installed, the instance is missing the data, thus has no content to display.

Notice that the data for Open QA is available at a different GitHub repository. The user can supply the service with the actual Open QA data by following 5 commands below:

cd ~
git clone git://github.com/eucalyptus/open-qa-frontend-data.git
cd ./open-qa-frontend-data
sudo tar -zxvf ./cache_storage_for_open_qa.tar.gz
sudo cp -r ./cache_storage_for_open_qa/* /var/www/html/webcache/.

Now, the instance is completely converted to run the Open QA web-service with the up-to-date test results.

The data for Open QA is, for now, updated at every 12:01am — pushed to the public GIT repository from the QA server at Eucalyptus HQ. In order to keep the data in sync, the AWS instances are scheduled to periodically pull the latest data via “git pull” on the data directory and copy over the new data to /var/www/html/webcache directory, which can be set as a cron job via “sudo crontab -e” on the instances.

This proof-of-concept case with Open QA application is to demonstrate how GIT and AWS can be used to make an open-source web-service to be launched by anyone with the AWS account. The separation of the service layer and the data for the web-service allows the update to take place independently; for some other web-applications, it might be desirable that the user supplies his/her own customized data, thus only utilizing the service layer.

If anyone ever dreams of launching 10,000 instances of a web-application in the cloud, it must be done through the open source scenario above where the application pulls the needed service and data at its own pace and for its own purpose, which is to serve locally.

%d bloggers like this: