Allow Everyone to Launch Your Web-Service: Open Source Web-Service using GIT and AWS

by kyolee310

Open Source is the future of software.

The statement above is self-evident at this point of history — I mean, have you checked out www.microsoft.com/opensource recently?

Then, what does open-source mean when it comes to web-services?

Open Source has been a well-understood concept among the web-application community due to the neat feature “View Source” that came with almost all web-browsers. Copy-and-pasting a chunk of HTML code from one website to another was a de facto standard way of developing websites. Whether admitted it or not, web-application development is deeply rooted in open source culture.

But, where should this open source culture be heading to? Should we keep maintain the culture of copy-and-pasting others’ code with no telling? Or, rather try to embrace this open source culture and explore the possibilities that arise from sharing code?

Then, it occurred to me that, “What if we could share the entire code for web-services?”

Imagine if anyone could launch web-services such as E-bay, Craigslist, Amazon, Angie’s list, Yelp, Instagram, Twitter, etc on their own instances on AWS. Anyone could instantly replicate all the functionality and services provided by such known web-services at its micro scale. If the big name web-services are the department stores in downtown, these open-source web-services on AWS instances are the mom-and-pop stores for the local community, or they could be the black market since these services could form and disappear in a rather unpredictable manner — similar to the life cycle of AWS instances.😉

Think about the scenario where a person can launch his/her own e-bay online auction service on an AWS instance for a school fund-raising event for a month. The person might also want to launch an instance of Craigslist for the same event, or Instagram and Twitter to keep people engaged during this period. Then, once the event is over, all the services can go away as if nothing ever happened.

Welcome to the era of Cloud Applications.

To test out the concept, I used the application “Open QA“, a web-service that is designed to display the test results of the development progress at Eucalyptus. The goal is to allow any Eucalyptus community member to launch the same service on AWS instances.

First, the web-service has to be broken down into two bodies, the service layer and the data. Then, each body of the web-service needs to be available in the open, allowing anyone to download the code and convert the AWS instances to Open QA web-service instances.

This is where GIT comes in very nicely. For public repositories, GIT will allow you create as many branches as you want, which can serve as read-only code repositories for anyone on the Internet without hassle.

The service layer part of Open QA is available at:

https://github.com/eucalyptus/open-qa

The data for Open QA is available at:

https://github.com/eucalyptus/open-qa-frontend-data

Upon launching an instance on AWS, a user can simply convert the raw Amazon Linux instance into the Open QA web-service by executing 4 commands below:

sudo yum -y install git
git clone git://github.com/eucalyptus/open-qa.git
cd ./open-qa/script/
./open_qa-installer.py -t amazon -e <your_email>

At this point, the AWS instance is running a HTTPD service with Open QA PHP code in its /var/www/html directory.

However, with only the service layer installed, the instance is missing the data, thus has no content to display.

Notice that the data for Open QA is available at a different GitHub repository. The user can supply the service with the actual Open QA data by following 5 commands below:

cd ~
git clone git://github.com/eucalyptus/open-qa-frontend-data.git
cd ./open-qa-frontend-data
sudo tar -zxvf ./cache_storage_for_open_qa.tar.gz
sudo cp -r ./cache_storage_for_open_qa/* /var/www/html/webcache/.

Now, the instance is completely converted to run the Open QA web-service with the up-to-date test results.

The data for Open QA is, for now, updated at every 12:01am — pushed to the public GIT repository from the QA server at Eucalyptus HQ. In order to keep the data in sync, the AWS instances are scheduled to periodically pull the latest data via “git pull” on the data directory and copy over the new data to /var/www/html/webcache directory, which can be set as a cron job via “sudo crontab -e” on the instances.

This proof-of-concept case with Open QA application is to demonstrate how GIT and AWS can be used to make an open-source web-service to be launched by anyone with the AWS account. The separation of the service layer and the data for the web-service allows the update to take place independently; for some other web-applications, it might be desirable that the user supplies his/her own customized data, thus only utilizing the service layer.

If anyone ever dreams of launching 10,000 instances of a web-application in the cloud, it must be done through the open source scenario above where the application pulls the needed service and data at its own pace and for its own purpose, which is to serve locally.