Adding Smalltalk to Cloud Foundry

In the previous post we set up a simple Smalltalk application on the client that could be pushed to the cloud. We are using the AidaWeb One-Click for Pharo as the “framework” and CogVM as the “runtime,” though you should be able to modify this process to use other frameworks and Smalltalk dialects. Now we look at changes needed to Cloud Foundry to receive that Smalltalk application. Much of this will be similar to the earlier process of adding Perl as a runtime and framework. (Peter McLain was particularly helpful in figuring out how to deploy a new runtime and framework on Cloud Foundry.)

The interesting part will come at the end when we deal with staging and starting an application.

Preliminaries

Before starting, you should create a private cloud (following the instructions here), and set up a client environment with VMC that recognizes Aida as a framework (as described here). Start the server, create a tunnel, open shells on the server and on the client, create a user, and view the list of frameworks and runtimes.

Adding Smalltalk to Cloud Foundry involves modifying five files and creating nine files. All modifications are relatively minor, and most of the new files are generally boilerplate. From a shell on the server, navigate to the VCAP directory:

cd ~/cloudfoundry/vcap/

Minor Modifications to VCAP

Edit cloud_controller/app/models/app.rb to add CogVM and Aida to lines 26 and 27:

Runtimes = %w[cog ruby18 ruby19 java node php erlangR14B02 python26]
Frameworks = %w[aida sinatra rails3 java_web spring grails node php otp_rebar lift wsgi django unknown]

Then add the following three lines starting at line 541:

when "aida/1.0"
  self.framework = 'aida'
  self.runtime = 'cog'

Next we edit dev_setup/cookbooks/cloud_controller/attributes/default.rb to add one line at line 10:

default[:cloud_controller][:staging][:aida] = "aida.yml"

Next we edit
dev_setup/cookbooks/cloud_controller/templates/default/cloud_controller.yml.erb to add two lines starting at line 110:

cog:
  version: 3.9

Next we edit dev_setup/cookbooks/dea/attributes/default.rb to add cog as a runtime in line 3:

default[:dea][:runtimes] = ["cog", "ruby18", "ruby19", "nodejs", "java", "erlang", "php"]

The last modification to an existing file is to dev_setup/cookbooks/dea/templates/default/dea.yml.erb where we add the following lines starting at line 46:

<% if node[:dea][:runtimes].include?("cog") %>
  cog:
    executable: /opt/smalltalk/cog/CogVM
    version: 3.9
    version_flag: '-version'
    environment:
<% end %>

New Directories and Files

We will need several new directories:

mkdir dev_setup/cookbooks/cog/ 
mkdir dev_setup/cookbooks/cog/attributes/ 
mkdir dev_setup/cookbooks/cog/recipes/ 
mkdir staging/lib/vcap/staging/plugin/aida/

We need several new files that are fairly “boilerplate,” starting with dev_setup/cookbooks/cloud_controller/templates/default/aida.yml.erb:

---
name: "aida"
runtimes:
  - "cog":
      version: "3.9"
      description: "CogVM"
      executable: "/opt/smalltalk/cog/CogVM"
      default: true
      environment:
detection:
  - "aida.st"

Next we copy a blank README:

cp dev_setup/cookbooks/erlang/README.rdoc dev_setup/cookbooks/cog/README.rdoc

Next we create dev_setup/cookbooks/cog/attributes/default.rb:

include_attribute "deployment"
default[:cog][:version] = "3.9"
default[:cog][:source] = \
  "https://gforge.inria.fr/frs/download.php/29042/CogVM-Unix-13307.zip"
default[:cog][:path] = \
  File.join(node[:deployment][:home], "deploy", "cog")

Next is dev_setup/cookbooks/cog/metadata.rb:

maintainer "VMware"
maintainer_email "support@vmware.com"
license "Apache 2.0"
description "Installs/Configures CogVM"
long_description IO.read(File.join(File.dirname(__FILE__), 'README.rdoc'))
version "0.0.1"

Next is staging/lib/vcap/staging/plugin/manifests/aida.yml:

---
name: "aida"
runtimes:
  - cog:
      version: '3.9'
      description: 'CogVM for Smalltalk'
      executable: /opt/smalltalk/cog/CogVM
      default: true
app_servers:
detection:
  - "aida.st": '.'
staged_services:

Adding CogVM and Aida to Cloud Foundry

Next we create dev_setup/cookbooks/cog/recipes/default.rb. This is where CogVM and Aida are added to the deployment so they can be used later:

# 
# Cookbook Name:: cog 
# Recipe:: default 
# 
# Copyright 2012, VMware 
# 
Chef::Log.debug("Installing ia32-libs for CogVM")
package "ia32-libs"
if not File.exists?("/opt/smalltalk")
  Dir.mkdir "/opt/smalltalk"
end
bash "Install CogVM from https://gforge.inria.fr/frs/?group_id=1299" do
  code <<-EOH
mkdir /opt/smalltalk/cog; cd /opt/smalltalk/cog
curl https://gforge.inria.fr/frs/download.php/29042/CogVM-Unix-13307.zip \
  > CogVM.zip
unzip CogVM.zip; rm CogVM.zip
  EOH
  not_if do
    ::File.exists?("/opt/smalltalk/cog")
  end
end
bash "Install Aida http://www.aidaweb.si/download" do
  code <<-EOH
mkdir /opt/smalltalk/aida; cd /opt/smalltalk/aida
curl http://ftp.eranova.si/aida/Aida6.4-OneClickPharo1.3-2jan12.zip \
  > Aida.zip
unzip Aida.zip; rm Aida.zip
resources=AidaOneClickPharo.app/Contents/Resources
mv $resources/AidaOneClickPharo.image Aida.image
mv $resources/AidaOneClickPharo.changes Aida.changes
mv $resources/PharoV10.sources PharoV10.sources
rm -rf AidaOneClickPharo.app
  EOH
  not_if do
    ::File.exists?("/opt/smalltalk/aida")
  end
end

As you can see, we are installing the 32-bit libraries (since we are using a 32-bit version of CogVM and are running on a 64-bit version of Ubuntu). Then we download a current version of the CogVM and the Aida runtime (much like we did here for the client). We copy the desired files from the Aida one-click Pharo directory, and delete the rest.

Now we get to the interesting work of staging and starting the application!

Staging Plugin for Aida

As we discussed earlier, Smalltalk does not fit the Cloud Foundry model of having a clean separation between the application code, the framework, and the runtime. While the image-based model is a challenge, it should not be too difficult. The approach we are taking is to have a file (aida.st) that contains Smalltalk code to load the application into a provided image (this way there is less copied from the client to the server). Cloud Foundry provides a “staging” step that is invoked as part of “pushing” an application from the client to the server. The staging step takes the various pieces from the client, mixes them with pieces from the server, creates any additional files, and bundles them all into a “droplet” that can be rapidly deployed when you want to start additional instances of your application (possibly on distributed machines).

In Cloud Foundry, staging is done by a staging “plugin” that is called when an application is pushed or updated from the client. The minimal plugin consists of a executable file, stage, and any files it expects. One approach is to create a Ruby script that calls a subclass of StagingPlugin to do the work. We create staging/lib/vcap/staging/plugin/aida/stage:

#!/usr/bin/env ruby
require File.expand_path('../../common', __FILE__)
plugin_class = StagingPlugin.load_plugin_for('aida')
plugin_class.validate_arguments!
plugin_class.new(*ARGV).stage_application

With this, the interesting activity happens in staging/lib/vcap/staging/plugin/aida/plugin.rb:

class AidaPlugin < StagingPlugin
  include GemfileSupport
  def framework
    'aida'
  end
#
  def stage_application
    Dir.chdir(destination_directory) do
      create_app_directories
      copy_source_files
      do_staging
      create_startup_script
      create_stop_script
    end
  end
#
  def do_staging
    Dir.chdir("app") {|dir|
      `cp /opt/smalltalk/aida/Aida.changes Aida.changes`
      `cp /opt/smalltalk/aida/Aida.image Aida.image`
      `ln -s /opt/smalltalk/aida/PharoV10.sources PharoV10.sources`
      staging = <<EOF
| file contents |
AIDASite default stop.
Author fullName: 'CloudFoundry'.
file := FileStream readOnlyFileNamed: 'aida.st'.
contents := file contentsOfEntireFile.
Compiler evaluate: contents.
SmalltalkImage current snapshot: true andQuit: true.
EOF
      File.open("staging.st", "w") {|f| f.write(staging) }
      run = "/opt/smalltalk/cog/CogVM -vm-display-null -vm-sound-null " +
        "Aida.image staging.st"
      puts system(run)
      `chmod 400 Aida.image`
      main = <<EOF
AIDASite default
port: (SmalltalkImage current getSystemAttribute: 3) asNumber;
start.
EOF
      File.open("main.st", "w") {|f| f.write( main ) }
   }
  end
#
  def start_command
    "/opt/smalltalk/cog/CogVM " +
      "-vm-display-null -vm-sound-null " +
      "Aida.image main.st $VCAP_APP_PORT $@"
  end
#
  private
  def startup_script
    vars = environment_hash
    # PWD here is after we change to the 'app' directory.
    generate_startup_script(vars) do
      "# aida startup script"
    end
  end
#
  def stop_script
    vars = environment_hash
    generate_stop_script(vars)
  end
#
end

Most of the interesting work here is in the “staging” when Cloud Foundry takes the application from the client and creates a “droplet” that can later be deployed. We could, of course, wait till an instance is started to copy the extent and load the application code, but this means it would happen on each startup. Instead, we can copy the image, changes, and sources, launch Smalltalk, execute the developer’s aida.st script, and then save the image and make it read-only. This means that the only thing that needs to be done at launch time is change the listening port.

Deploying VCAP

With these modifications and additions to VCAP, we can update the cloud:

vcap stop; dev_setup/bin/vcap_dev_setup -d ~/cloudfoundry

Next we update the files that aren’t updated by the above (for whatever reason!?):

fromDir=`find ~/cloudfoundry/vcap/staging/ | grep pip_support.rb`
fromDir=`dirname $fromDir`
toDir=`find ~/cloudfoundry/.deployments/devbox/ | grep pip_support.rb`
toDir=`dirname $toDir`
cp -r $fromDir/aida $toDir
cp $fromDir/manifests/aida.yml $toDir/manifests/aida.yml

Then we can restart the cloud:

vcap start

Trying it Out!

Now from the client, get a list of the runtimes and frameworks:

vmc info; vmc runtimes; vmc frameworks

From the client, navigate to the application directory created here:

cd ~/cloud/aida/app

Make sure that there is a file named aida.st and that it does not have the code we added to test changing the port (since this is now handled on the server). Here is a sample file:

Author fullName: 'CloudFoundry'. "Developer's name"
WebDemoApp compile: 'introductionElement
  | e |
  e := WebElement new.
  e addText: self observee introduction.
  e addText: ''<p>Listening on port: '' , 
  session parent site port printString , ''</p>''.
  ^e'.

From this directory with only this file (and possibly a manifest.yml), push the application to your cloud:

vmc push

Just as with the Perl application, there are a series of questions. We can name the application ‘aida-env‘ and otherwise accept the defaults. When the application has staged and started, we can navigate to http://aida-env.vcap.me and (if all went well!) see AidaWeb in the cloud.

Conclusion

We have deployed a Pharo Smalltalk application to Cloud Foundry. While the above does work with a recent Cloud Foundry development setup, things are changing rapidly and I suspect that several parts of the above are either redundant or could be done better. But, following the “make it work, make it right, make it fast” principle, we now have it working.

A limitation of the Cloud Foundry architecture is that Smalltalk’s image based persistence is not supported. Instead, we should look to using an external database as a Cloud Foundry service. More on that later!