Creating a Micro Cloud Foundry

On a clean MacBook Pro running Snow Leopard (10.6.8), I  downloaded an ISO to install 64-bit Ubuntu Server 10.4.3, then I installed VMware Fusion 4.1.1. I launched Fusion and started the process to create a new virtual machine:

• Introduction: I clicked the “continue without disk” button.
• Installation Media: I chose the disk image for Ubuntu Server.
• Operating System: I left the defaults of Linux and Ubuntu 64-bit.
• Linux Easy Install: I accepted the default user and specified a password (‘swordfish’).
• Tools Download: I downloaded the tools.
• Finish: I accepted the defaults (1 GB RAM, 20 GB hard disk, NAT networking).
• Save As: I named the virtual machine ‘My Cloud’.
• A few minutes later the server started and gave a login prompt.

It seems that Fusion’s easy install of Ubuntu does not get the keyboard properly mapped, so I entered the following command:

sudo dpkg-reconfigure console-setup

I selected the ‘Dell 101-key PC’ keyboard (pressing the <D> key repeatedly since the arrow keys didn’t work), and accepted the other default settings. I then entered the following commands (based in part on the instructions here) to update and install a few needed packages:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install vim
sudo apt-get install openssh-server
sudo apt-get install curl

Next, I entered the command to setup this Ubuntu server as a VMware Cloud Application Platform (VCAP):

bash < <(curl -s -k -B \
 https://raw.github.com/cloudfoundry/vcap/master/dev_setup/bin/vcap_dev_setup)

This process took some time (an hour?) and when it finished it showed a message describing how to start Cloud Foundry:

~/cloudfoundry/vcap/dev_setup/bin/vcap_dev start

We now have the confusion of dealing with two machines (one real and one virtual) and so need to be careful to identify which one we mean in the subsequent instructions:

• Client: A MacBook Pro is my development machine, and a “client” with respect to “the cloud.” For this project, I interact with the client primarily using Terminal (using “a shell on the client”), but also using a web browser (Safari, Firefox, or Chrome) and the Finder.
• Server: VMware Cloud Application Platform (VCAP) runs on 64-bit Ubuntu in a Fusion virtual machine on the Mac, and provides “the cloud services” on a “Micro Cloud.” I interact with the server primarily using ssh in a Terminal session (“a shell on the server”), but occasionally using the console visible in Fusion. This micro cloud offers the same developer experience that a public cloud (such as appfog.com) should provide, so you can practice your deployment without making your application public.

In order to interact with the server using ssh in a Terminal session, we need to know the server’s IP address. On the server console, I entered the following command:

ifconfig eth0 | grep inet

The first line gives the IPv4 address (in my case, 192.168.10.128). On the client, I then entered the following command:

sudo vim /etc/hosts

to edit the client’s hosts file and add the following line:

192.168.10.128   mycloud

You should, of course, use the IP address you got from the server. Then, from a client shell, I entered the following to get a shell on the server:

ssh mycloud

I entered ‘yes’ to add the RSA key and then entered the password I defined when I built the server (‘swordfish’). To simplify my server commands, I added an alias using the following commands on the server:

echo "alias vcap='~/cloudfoundry/vcap/dev_setup/bin/vcap_dev \$1'" >> ~/.bashrc
source ~/.bashrc

On the server, I entered the following command:

vcap start

This gave me some debugging information, and ended with a list of services that all show as “RUNNING.”

Targeting the Micro Cloud

From the client we interact with a cloud using vmc, the command-line interface in a client shell. When we were using the public cloud, the “target” was api.cloudfoundry.com; with the micro cloud we use api.vcap.me as the target. VMware has registered the domain vcap.me and configured the DNS servers to map that domain (and all its subdomains) to 127.0.0.1 (or localhost). Of course, the micro cloud is not at 127.0.0.1 from the perspective of the client, so we need to establish a tunnel from port 80 on the client to port 80 on the server. First, we need to ensure that port 80 is not in use. On my Mac, I opened System Preferences, selected Sharing, and then made sure that ‘Web Sharing’ is unchecked. With port 80 available, I entered the following command in a client shell (replacing USER with the Ubuntu user defined during the initial setup):

sudo ssh -L 80:mycloud:80 USER@mycloud -N

This command establishes a tunnel from port 80 on the client to port 80 on the server, and the shell hangs until the tunnel is terminated (e.g., with <Ctrl>+<C>). To show that the cloud is there and responding, I entered http://api.vcap.me/info in a client web browser and observed the JSON data returned. In another client shell, I entered the following vmc commands (you should provide your own email!):

vmc target api.vcap.me
vmc info
vmc add-user --email jfoster@vmware.com --passwd swordfish
vmc info

With the micro cloud up and running, I tried deploying my trivial Ruby application (described here) using the following client commands:

cd ~/env
vmc push

In response to the questions, I named the application ‘jfoster-env’ and asked for 2 instances; otherwise I accepted the defaults. At this point (see notes below) I was able to see the application in a web browser at http://jfoster-env.vcap.me/ and http://jfoster-env.vcap.me/env and I tried the various vmc commands at the end of this post. The behavior was generally consistent with the public cloud.

Note 1: The first time I tried a push the response was “Error: Application [jfoster-env] failed to start, logs information below. Error 306: Error retrieving file ‘/logs/err.log.” I believe that this means that the application failed to start in time, and that when the system went to look for an error log, there wasn’t one. The application actually was running and responded to HTTP requests. Also, ‘vmc apps’ and ‘vmc stats jfoster-env’ showed the application running and ‘vmc restart jfoster-env’ reported success.

Note 2: The response to ‘vmc logs jfoster-env’ included “Error 306: Error retrieving file ‘logs/startup.log.” I found someone who said, “The file ‘logs/startup.log’ isn’t always generated, so the error returned by vmc can be a red herring.”

Conclusion

We have built an Ubuntu server and configured it as a private micro cloud. We used it as a target for our application and were able to use it in place of the public cloud. Next we will investigate adding a new runtime and framework to Cloud Foundry.

I started by registering for an account because I wanted to try deploying on VMware’s cloud (this step is not necessary for our later deployment on a private cloud). The confirmation email took several hours to come, and I’ve read of others who waited for days, so if you register don’t be surprised if it takes a while to hear back.

On a clean install of Snow Leopard (Mac 10.6), I started by creating a very simple Sinatra application in Ruby. (Of course, the goal of this project is to deploy Smalltalk to the cloud, but we will start with things that Cloud Foundry understands, and go from there!) In a Terminal, I entered the following commands (providing my password when prompted):

sudo gem install mime-types
sudo gem install rubyzip
sudo gem install uuidtools
sudo gem install sinatra
mkdir env; cd env

In ~/env/ I created a file, env.rb, consisting of the following text:

require 'rubygems'
require 'sinatra'

get '/' do
    host = ENV['VMC_APP_HOST']
    port = ENV['VMC_APP_PORT']
    "<h1>XXXXX Hello from the Cloud! via: #{host}:#{port}</h1>"
end

get '/env' do
    res = ''
    ENV.each do |k, v|
        res << "#{k}: #{v}<br/>"
    end
    res
end

Then, from the Terminal, I entered the following command:

ruby env.rb

This told me that Sinatra was running a web server on port 4567, so I opened a web browser on http://localhost:4567/env and verified that my application ran and returned the expected values. I used <Ctrl>+<C> in the Terminal to terminate the application.

The next task was to install the command-line tools (vmc) used to interact with Cloud Foundry. In the Terminal I entered the following command:

sudo gem install vmc --pre

The ‘–pre’ option tells ruby to install the pre-release version of vmc. Next, I specified the target cloud for deployment, I logged in using the email address and the password I got when I registered (see above), and I pushed the application to the cloud:

vmc target api.cloudfoundry.com
vmc login
vmc push

The push command prompts a series of questions. I accepted the default (Y) to deploy from the current directory. I gave a unique name for the application (‘jfoster-env’). I accepted the default to confirm that this is a Sinatra application and it can be deployed in 128M of RAM. I specified two (2) instances so I could see the application run on two machines. Finally, I accepted the default to confirm that I did not need any services and I did not need to save the configuration. The tools then proceeded to confirm that the application was pushed and started successfully. In a web browser, I went to the designated location, http://jfoster-env.cloudfoundry.com/, and found that my application was indeed running. I refreshed the page a few times and confirmed that the application was running on two machines (showing different internal IP addresses and ports).

I tried a number of commands to explore what was available through the vmc command-line tools:

vmc help
vmc user
vmc target
vmc info
vmc runtimes
vmc frameworks
vmc apps
vmc instances jfoster-env +2
vmc apps
vmc instances jfoster-env -2
vmc stats jfoster-env
vmc logs jfoster-env
vmc files jfoster-env
vmc files jfoster-env app
vmc files jfoster-env logs
vmc env jfoster-env
vmc stop jfoster-env
vmc apps
vmc delete jfoster-env
vmc apps
vmc logout

In all of this, we have been interacting with the VMware public cloud. The next blog post will explore creating and using a private development cloud.

For this discussion we will consider the following example: from the collection of [A-H] we want to get the subset from begin to end, or [D-F].

Of course, one could iterate over the entire collection and test each object, but this will be inefficient for large collections. At the other extreme would be to have an end-of-day job that create a collection for that date and add items to the day’s collection. This would give very quick lookup, but would have a cost in storage.

A typical attempt to solve this problem would be to create an index on the start date and the stop date and then build a result set of items that start before the end point and stop after the begin point. The problem with this approach is that it likely creates two intermediate results, [A-F] and [D-H], that together are larger than the original collection (before returning the intersection of those intermediate results). In GemStone, this can generally be done in such a way that the intermediate results hold the object IDs (or OOPs), but the objects are not actually read from disk. This is typically a good solution, but others are possible.

Decades ago my good friend, Carl Zimmerman, came up with another approach that uses one index and avoids the large intermediate results. The following is adapted from his design to work with GemStone’s indexing system.

Each object has an ‘indexableDateRange’ instance variable (with an equality index defined) that encodes the duration (in days) and the start point (a Date after 1900) as a SmallInteger. The value is set as follows:

  indexableDateRange := start isNil
    ifTrue: [0]
    ifFalse: [stop isNil
      ifTrue: [start asDays]
      ifFalse: [(stop subtractDate: start) + 1 * 100000 + start asDays]].

Note a few things about this expression. If the value is zero, then the activity has not started. If the value is less than 100000, then the activity is ongoing. If the value is greater than 100000, then the activity has finished. Note that this design handles dates from 1901 to 2173. Handling dates outside that range or other units, such as minutes, is left as an exercise for the reader (or an opportunity for consulting!).

Let’s start by creating some test data. The following creates 1000 objects that have a start date from 2000 through most of 2009 and durations of up to 10 days (or still in progress).

| set random origin |
set := UserGlobals at: #'James' put: IdentitySet new.
set createEqualityIndexOn: #'key' withLastElementClass: SmallInteger.
System commitTransaction.
random := Random new.
origin := Date newDay: 1 monthNumber: 1 year: 2000.
1000 timesRepeat: [
  | start stop key |
  start := origin addDays: (random integerBetween: 1 and: 3650).
  stop := start addDays: (random integerBetween: 0 and: 10).
  stop = start ifTrue: [stop := nil].
  key := start isNil
    ifTrue: [0]
    ifFalse: [stop isNil
      ifTrue: [start asDays]
      ifFalse: [(stop subtractDate: start) + 1 * 100000 + start asDays]].
 set add: key -> (Array with: start with: stop).
].
System commitTransaction.

The following demonstrates a “brute-force” search that tests every object:

| start stop set resultSet |
set := UserGlobals at: #'James'.
start := (Date newDay: 1 monthNumber: 1 year: 2005) asDays.
stop := start + 6.
resultSet := set select: [:each |
  each value first asDays <= stop and: [each value last isNil or: [start <= each value last asDays]].
].
resultSet

The following code demonstrates the Zimmerman search algorithm:

| start stop set resultSet duration |
set := UserGlobals at: #'James'.
start := (Date newDay: 1 monthNumber: 1 year: 2005) asDays.
stop := start + 6.
resultSet := IdentitySet new.
resultSet addAll: (set select: {:each | each.key <= stop}).    "Items that are in-progress"
duration := 100000.    "start at one-day duration"
[true] whileTrue: [
  | low high stream |
  low := duration + start - (duration // 100000).
  high := duration + stop.
  resultSet addAll: (set select: {:each | (each.key >= low) & (each.key <= high)}).
  duration := duration + 100000.
  stream := set selectAsStream: {:each | each.key >= duration}.
  stream atEnd ifTrue: [^resultSet].
  duration := stream next key // 100000 * 100000.
].
resultSet

This approach starts with all the events that started before the stop point and are still in progress. Then it iterates over the completed items starting with those that have a one-day duration. A range of keys is computed based on the start, stop, and current duration, and the equality index is used to get the matching items for that duration. For domains in which there are a limited number of durations (such as video rentals), there are relatively few intermediate result sets and none have any unneeded objects. Given GemStone’s efficient handling of IdentitySets and intersection operations, I suspect that this is not worth the trouble, but it is an interesting approach. This approach does fault in one extra object for each duration after 1, but since the needed information is actually in the indexing data structures, this could be avoided.

Anyway, it is something to think about (and measure).

The original no-cost license allowed for up to 1 GB shared page cache (SPC) and up to 4 GB for the total repository size. In a busy system where you could have at least 25% of the object space in memory (and had adequate memory for each of the Gems) doing an hourly MFC was important (to avoid having excess garbage) and not too expensive.

In some situations, however, the maintenance gem may be causing excess overhead. If you are running GLASS in the “cloud” (e.g., on SliceHost or some other virtual server), then the cost of RAM may provide a significant constraint on your SPC. Also, if your system is not used heavily, then there might not be enough garbage (primarily from expired sessions) to justify frequent MFC. Finally, since the no-cost license now allows for unlimited repository size, the consequence of running out of space is not so dire.

This is important because MFC is a comparatively heavy-weight operation and can take a lot of time. On some larger customer databases, it can take many days and the user experience often suffers. The size of the database is really not the most important factor; more important is the percent of the database that can fit in the SPC. If you are running in the cloud with  512 MB of RAM and are allowed only 1/8th of a CPU, then a 2 GB database in a 256 MB SPC could see a significant decrease in performance during an MFC.

To determine the necessary MFC frequency, take a look at the maintenance gem log. This will show the number of sessions expired each minute and the number of possible dead objects found during each MFC. You can go through the log and add up the possible dead during a 24-hour period. The next step is to go to the admingcgem log and look for an entry labeled “Starting doSweepWsUnion” with a timestamp shortly after the MFC completed. This should show a “PD size” that is approximately the number reported by the MFC. A few lines down will be a couple entries showing a count of objects removed from the possibleDead. Just before the next “Starting doSweepWsUnion” will be a final (and generally lower) “possible dead size”. You can take the total of these final sizes to get an estimate of the useful work done by the MFC.

In one case, we saw hourly MFCs take about 10 minutes each and over the course of a day the repository garbage collection process found 1.5 million dead objects. If each object is about 120 bytes, this is less than 200 MB. In this case it was worthwhile to switch to a daily MFC and do it during off-peak time when it would not affect the users.

To implement this change required an edit to $GEMSTONE/seaside/bin/runSeasideGems[30] to comment out the lines to start the maintenance gem. Instead, you can create a new script modeled on $GEMSTONE/seaside/bin/startMaintenance[30] that does not have an endless loop for expiring sessions and doing an MFC but does the work once then exits. You can then set up a cron job to call your new script.

I realize that I’m not providing all the details of the new setup here. The goal is more to provide an exploration of alternatives.

The first step to fix this is to find out which files are being served from Seaside. You can do this by looking at your web server logs or by looking at requests made by the browser (in Firefox, use Firebug). If you see SUDevelopmentLibrary being requested, then look for references to that class. Typically the class will be the argument to an #’addLibrary:’ message following a #’register:asApplicationAt:’ message. Remove the #’addLibrary:’ call and re-register your application.

Second, create operating system files containing the material previously served by Seaside. You can copy the text from the methods or you can obtain the files elsewhere. For example, to replace SUDevelopmentLibrary, go to the Scriptaculous downloads page. Note, however, that SUDevelopmentLibrary has an additional method, #’treePatchJs’, that is not included in the download. Typically these files would be placed in your web server’s /scripts directory.

Finally, add references to the static file(s) to your component. Typically this is done in #’updateRoot:’ on your component. For example:

	#('prototype.js' 'scriptaculous.js' 'treePatch.js') do: [:each |
		anHtmlRoot script
			beJavascript;
			url: '/scripts/scriptaculous/' , each;
			yourself.
	].

With this, we reduced by 95% the number of pages served by Seaside!

• Arriving in Edinburgh on Friday
• Around Edinburgh on Saturday
• Scottish Highlands Bus Tour on Sunday
• ESUG 2011 on Monday and Tuesday
• Social Event on Wednesday
• Web Frameworks Panel on Thursday
• WebTools Presentation on Friday

VMware vFabric GemStone/S  3.0.0 is now live on vmware.com.

Highlights of what’s new in the GemStone/S 64 Bit 3.0.0 release:

• Significant performance improvements and scalability features, including a redesigned Smalltalk virtual machine with just-in-time (JIT) compilation to native code, and parallelized, load-balancing garbage collection and repository scan operations.

• Internal redesign for compatibility and portability, including new features supporting the Seaside web framework, and improved support for the ANSI Smalltalk standard and for portability to other Smalltalk dialects.

• Various other features, such as Foreign Function Interface (FFI), simplifying the interface to third-party run-time libraries.

GA date: 07/14/2011

Download portal: http://downloads.vmware.com/d/info/datacenter_downloads/vmware_gemstone_s/3.0

Release Notes & documentation for GemStone/S:  http://community.gemstone.com/display/GSS64/GemStoneS+64+Documentation

No-cost License Key: http://seaside.gemstone.com/etc/

Seaside has the built-in ability to use HTTP Authentication to restrict an application to a specific user/password. The method WAAdmin class>>#’register:asApplicationAt:user:password:’ registers the application with WAAuthenticationFilter as a filter and provides a single user and password that must be provided in order to view the initial page. This provides some password security, but does not differentiate among allowed users (e.g., everyone will use the same ‘admin’ user name).

If you have an application that will always require authentication but should differentiate among multiple users, you can use WAAuthenticationFilter as well. For example, if you have a subclass of WAComponent named HelloComponent, you could provide a class-side initialization method as follows:

initialize
"HelloComponent initialize."
 | application filter |
 application := WAAdmin 
 register: self
 asApplicationAt: 'hello'.
 filter := WAAuthenticationFilter new
 authenticator: self;
 yourself.
 application addFilter: filter.

Like Seaside’s default authentication approach, this method adds a WAAuthenticationFilter as a filter, but provides an override to the default authenticator (here the component class itself). The authenticator must implement #’verifyPassword:forUser:’. The class side of HelloComponent could have something more sophisticated than the following:

verifyPassword: password forUser: username
 username = 'bert' ifTrue: [^password = 'ernie'].
 ^false.

At this point your render code can assume that a valid user has been properly authenticated and can vary the content based on the user. For example, the following instance-side method in HelloComponent would show what user was authenticated:

renderContentOn: html
 html text: 'Hello ' , self requestContext request user , '!'.

Now you have basic HTTP authentication for multiple users. Of course, you are now subject to the limitations of basic HTTP authentication, including the fact that logout is somewhat awkward. You can provide a logout link:

renderContentOn: html
 html text: 'Hello ' , self requestContext request user , '!'; break.
 html anchor
 callback: [self doLogout];
 with: 'Logout'.

The logout method informs the user how to complete the logout and then adds a field to let the verification method know to fail:

doLogout
	| url |
	self inform: 'When the login dialog appears, click the Cancel button then close your browser.'.
	url := self requestContext request url copy.
	url addField: 'logout'.
	self requestContext redirectTo: url.

The verification method would then check for the new field:

verifyPassword: password forUser: username
 (WACurrentRequestContext value request url queryFields includesKey: 'logout') ifTrue: [^false].
 username = 'bert' ifTrue: [^password = 'ernie'].
 ^false.

Charlie Meyer has an explanation of how he used a similar approach with LDAP authentication.

After doing the load, I found a few packaging issues:

• ‘Metacello-MC’ was marked dirty but had no changes. I reloaded it to clear the dirty flag.
• ‘Seaside2′ existed but had no repositories or versions. I unloaded the package.
• ‘Seaside-Core.gemstone’ was missing WAAuthConfiguration. I reloaded the package.
• ‘Seaside-Component’ was dirty but had no changes. I reloaded it to clear the dirty flag.
• ‘Seaside-Tools-Core’ was dirty but had no changes. I reloaded it to clear the dirty flag.
• ‘VB-Regex’ existed but was replaced by ‘Regex-Core’ and ‘Regex-Tests-Core’. I unloaded it.

With this, I have Seaside 3.0 loaded (but not running). There are a substantial number of changes between Seaside 2.8 and Seaside 3.0, so much work remains to be done–but at least we have the code loaded!

ConfigurationOfGLASS project updateProject.
ConfigurationOfGofer project updateProject.
ConfigurationOfGoferProjectLoader project updateProject.
ConfigurationOfGsCore project updateProject.
ConfigurationOfGsMisc project updateProject.
ConfigurationOfGsMonticello project updateProject.
ConfigurationOfGsOB project updateProject.
ConfigurationOfMetacello project updateProject.
System commitTransaction.

MCPlatformSupport autoMigrate: true.
ConfigurationOfMetacello project latestVersion load.
ConfigurationOfGoferProjectLoader project load: #stable.
MCPlatformSupport autoMigrate: false.
System commitTransaction.

MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
	[
		ConfigurationOfGLASS project load: #stable.
	] on: Warning do: [:ex | ex resume ].
].
System commitTransaction.

MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
	[
		Gofer project load: 'Seaside30' version: '3.0.5'.
	] on: Warning do: [:ex | ex resume ].
].
System commitTransaction.