A question on the GLASS mailing list raised some interesting questions about garbage collection and persistent executable blocks. Some objects, such as instances of SortedCollection, may contain a reference to an executable block and this persistent reference may be difficult to update and may hold references to other objects that otherwise could be removed by the garbage collection process.

Simple vs. Complex Blocks

The first issue raised by the question is the distinction between a “simple block” and a “complex block.” In Smalltalk a block may reference variables outside the block and, if it does so, then the virtual machine needs to create a more complex data structure to capture the referenced objects (making it a “complex” block). If the block does not reference any variables outside its immediate scope (beyond the block arguments and block temporaries), then the virtual machine can typically use a much simpler implementation (making it a “simple” block).

Because of the extra overhead, complex blocks tend to take more room and be slower to execute. Thus, a common performance tuning activity is replacing complex blocks with simple blocks when possible, and this can occur in application code as well as vendor-supplied libraries. For example, in GemStone/S 64 Bit version 2.x, the implementation of SortedCollection>>#’addAll:’ is essentially the following:

aCollection do: [:each | self add: each].

The problem with this implementation is that the block references self, making it a complex block. In version 3.x the implementation changed (influenced in part by Paul Baumann’s work), and the code is essentially the following:

aCollection accompaniedBy: self do: [:me :each | me add: each].

The block in this code is a simple block since it does not reference self (or anything outside the block), and performance improved.

While this is all very interesting (and in some cases quite useful), I think that the underlying garbage collection problem is not really with simple vs. complex blocks, but with something else.

Class References in Blocks

A code block in a method will contain a reference to the method and thus to a class (or metaclass for a class-side method). Once the code block is persistent, there is a hard reference to the class (and its class variables, class instance variables, and pool variables). Changes to the class schema will create a new “version” of the class but the old version will remain and be referenced by the code block. Even if the old class version is removed from the ClassHistory and all instances of the old version are migrated to the new version, until the code block is replaced in the persistent object, there will be a hard reference to the old class version and its space cannot be reclaimed by the garbage collection process.

To see how this works consider the method SortedCollection>>#’_defaultBlock’:

| block |
block := SortedCollection new _defaultBlock. "anExecBlock2"
block _debugSourceForBlock. "'[ :a :b | a <= b ]'"
block method. "aGsNMethod"
block method isMethodForBlock. "true"
block method inClass. "SortedCollection"

Note that this situation exists whether or not the block is simple or complex and whether or not the block comes from an instance-side method or a class-side method. Thus, this is an orthogonal discussion or a unique concern.

Alternatively, note how an equivalent block can be created by evaluating a String, but does not contain a reference to a Class:

| block |
block := '[ :a :b | a <= b ]' evaluate. "anExecBlock"
block _debugSourceForBlock. "'[ :a :b | a <= b ]'"
block method. "aGsNMethod"
block method isMethodForBlock. "true"
block method inClass. "nil"

So, as suggested one could strip the class reference with the following code:

^ aBlock _sourceString evaluate

Alternatively, one could avoid the private method (#’_sourceString’) by making the String explicit:

^ '[ :a :b | a <= b ]' evaluate

While this avoids the private method, it hides the code in a string making it much more difficult for tools to recognize that this is code (and catch compile errors, support senders/implementors, provide for refactorings, etc.). Further, each of these approaches will create a new instance each time it is called, creating unnecessary objects.

As Dale noted in the email chain referenced above, we could modify the compiler so that a block did not know the class in which it was created. This might improved the garbage collection situation, but would make it more difficult to debug code and track a block back to its source. And, of course, you would need to wait till that feature was added to the product.

One way to address these issues is to create a new class with no instance variables (so that the schema would never change) and use class-side methods to return the desired blocks. The method that would otherwise hold the code block would use a level of indirection to return the actual block. If a code block needed to change, you would create a new class-side method in the special class and edit the indirecting method (so that references to the old code block could be found and modified). The original method would remain as long as there were references to it.

At the cost of a level of indirection (and reduced encapsulation), this gives you actual source code that can be recognized by tools, a single instance, and no stray references from the class (since there is only one class version).

[Note: I found this in “Drafts” and decided to publish it even though the original question was a couple years ago.]


A video of my presentation on GemStone/S and SQL is available here. Unfortunately, the audio is not very strong.

GemStone/S has a configuration STN_TRAN_FULL_LOGGING that, if set to TRUE, causes all transactions to be recorded to the redo-log. In this mode a backup plus transaction logs can be used to recover if the extents are lost. If the configuration is set to FALSE, small transactions are still logged but large transactions cause a “checkpoint” in which all data is written to the extents before the transaction completes. In this mode recovery from a crash is possible if all the extents and the most recent transaction log are available (older logs are typically deleted automatically in this mode). The logs themselves are not sufficient to allow recovery from a backup in partial logging mode, so partial logging mode is inappropriate for production (but handy sometimes for development since old logs are typically deleted automatically).

In some applications or situations it would be nice to segregate transactions into those that should be logged for crash recovery and those that can be easily recreated if needed (and avoid the overhead of time and space in writing to the log). For example, it is common to do a bulk-load of new data before it is needed by the application. If the system crashes, we’d prefer to recover the live data quickly and reload the new data later. With Oracle one can specify a table as being NOLOGGING and then do a series of “Direct-Load INSERT” commands that bypass the rollback/replay log. A key limitation with this capability, however, is that it does not support referential integrity. Likewise, with SQL Server one can do a BULK INSERT that will, under some circumstances, have minimal logging. By default, this will leave the constraint on the table marked as not-trusted.

For a relational database, a foreign key is a value like any other (typically an integer or string) and if a JOIN fails to find a match then the matching data is NULL or the row is ignored. In GemStone/S, an object reference is guaranteed to be valid and there is no easy way to handle data that lacks referential integrity. Thus, if an object is visible in the database (e.g., from a bulk load), then any session can, in a transaction, create a reference to it. If the reference is in the transaction (redo) log, then the object must itself be in the transaction log. Therefore, because in GemStone/S referential integrity is required, not optional, the relational approach to avoiding the transaction log is not safe.

The relational solutions described above for reducing transaction log activity allow for a bulk load of data that can, eventually (following a full back, recreation of indexes, and reapplication of any constraints) be equivalent to all other data. This fits an application upgrade (bulk load) use case quite well.

Another possible use case is short-lived data that is to be shared by multiple sessions but will not be saved over the long-term. One example is transient data where a summary is to be kept but the raw data discarded after the analysis. Another example is information about logged-in sessions with in-progress activity (such as Seaside session state) that could be safely lost in a system crash (presumably the users would need to start over anyway). This use-case does not present the same referential integrity concerns and GemStone/S is developing a feature that is intended to address this use-case: a SymbolDictionary in Globals named #’NotTranloggedGlobals’. The idea is that any object referenced from this root would not be recorded in the transaction log and should not be referenced from any other path. At present this is still a work-in-progress and customers are advised to not use it.

Periodically the International Earth Rotation Service determines that Coordinated Universal Time needs to be adjusted to match the mean solar day. This is because the earth doesn’t rotate at a constant speed and in general takes slightly longer than 86,400 seconds. These adjustments have been done through adding (or, in theory, removing) a second every few years. The next leap second insertion is scheduled for June 30th, 2015 at 23:59:60 UTC. How does GemStone/S handle that?

GemStone gets time from the host OS (Unix Time) and is typically described as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970. This definition is correct so long as we assume that each and every day has exactly 86,400 seconds. If you take into account the 27 leap seconds that will have been added as of June 30, 2015, it is more accurate to say that Unix Time is the number of atomic seconds that have elapsed since 00:00:27 UTC, 1 January 1970. The way the adjustment occurs is that when there is a leap second, the Unix time counter takes two atomic seconds to advance by one Unix second.

The impact of this approach is that a record of when something happened will be correct, but the time between two events could be reported as being one or more seconds less than the actual time between the events. For example, Unix time (and GemStone) will report that there are five seconds between June 30 at 23:59:55 and July 1 at: 00:00:00 when in fact in 2015 there were six seconds between those two times.

Whether this matters is an application-level or domain-level problem. If keeping track of the time between events needs to be accurate (e.g., recording some rapid physics event), then relying on Unix time (or GemStone) will not be sufficient. If all that is needed is a timestamp and it is acceptable to have June 30, 2015 23:59:59 UTC last for 2000 milliseconds, then things should be fine.

A screencast of this blog post is here.

Because Smalltalk was the origin of much of today’s GUI (mouse, overlapping windows, drop-down menus), Smalltalk developers are understandably accustomed to a nice GUI IDE. GemStone/S is an excellent database and Smalltalk execution environment and includes a built-in command-line tool, Topaz, where you can execute Smalltalk code, but has no native GUI. In this blog post we continue a demonstration of GemStone.app on the Macintosh (started here) and show Jade, a GUI-based IDE available on Microsoft Windows.

We launch GemStone.app on the Macintosh, update the version list, download, then install and start a GLASS extent (image) that includes Monticello/Metacello tools. When the database is running we start a Topaz session and install Seaside 3.0 and Magritte 3 with the following script:

"based on https://code.google.com/p/glassdb/wiki/Seaside30Configuration"
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
  ConfigurationOfMetacello project updateProject.
  ConfigurationOfMetacello loadLatestVersion.
  Gofer project load: 'Seaside30' group: 'Seaside-Adaptors-Swazoo'.
"based on http://www.iam.unibe.ch/pipermail/smallwiki/2012-February/007188.html"
Gofer it
  squeaksource: 'MetacelloRepository';
  package: 'ConfigurationOfMagritte3';
MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [
  ConfigurationOfMagritte3 project stableVersion load.
WAGsSwazooAdaptor new start.

When Swazoo is running, we can go to http://localhost:8080 and see Seaside running locally. This demonstrates running Smalltalk code in Topaz, the command-line tool. Next we look at a GUI-based IDE that runs on a Microsoft Windows client platform.

Jade is available as a 14 MB zip download from http://seaside.gemtalksystems.com/jade/. It includes an executable, client libraries (DLLs) for various GemStone/S versions (ranging from 32-bit version 6.1 to the latest 64-bit version), and related files (including source code). Like most GemStone/S client GUI tools, it is built in another Smalltalk (in this case, Dolphin Smalltalk from Object Arts), but unlike these other tools, you can’t see the client Smalltalk (unless your load Jade source code into your own Dolphin development environment), so we avoid the the two-object-space confusion. Jade is intended to take you directly to GemStone/S, without going through Pharo, Squeak, VA, or VW Smalltalk.

Jade is also designed to work with the no-cost version of GemStone/S (unlike the VA/VW-based GBS tools), and performs well on a slow network (unlike GemTools).

When you unzip the download, you have a folder with various items. Jade.exe is the primary executable (containing the Dolphin VM and the image) and it relies on Microsoft’s C Runtime Library. There is a copy of the executable in Jade.jpg for sites where executables are stripped from zip files during the download process (simply rename the suffix and it will become executable). Contacts.exe is used sometimes in a training class. The bin directory contains the GCI client libraries and a DLL with various images used in the IDE. You can also see a directory containing source code for Jade.

Screen Shot 2013-10-01 at 10.56.18 AM

When you launch Jade, you get a login window that gives you a place to select the GemStone/S version (which GCI library we will use), and other information needed for a login. The Stone box contains two fields, one for the host/IP of the Stone machine (from the perspective of the Gem machine, so localhost is almost always sufficient), and the name of the Stone. In the screencast mentioned above our stone was named gs64stone1. The Gem box contains a number of fields. Most logins will use an RPC Gem (a Linked Gem is available only on 32-bit Windows) with a Guest-authenticated Gem (if your NetLDI was not started in guest mode (-g), then you will need to provide an OS user and password). An RPC Gem will be started by a NetLDI, so we need to identify the Gem machine (in my example the host is vienna and the NetLDI is listening on port 54120) and the command used to start the Gem (except in rare cases the command will be ‘gemnetobject’). You provide a GemStone User ID and Password (by default, ‘DataCurator’ and ‘swordfish’), and if you are going to use any Monticello features it would be good to identify the developer’s name (one word with CamelCase).

Screen Shot 2013-10-01 at 11.01.49 AM

If you get an error on login, we attempt to give as much explanation as possible. Typically, (1) there is no NetLDI on the host/port (see following example), (2) there is no stone with that name, (3) there is a version mismatch, or (4) you have given an unrecognized user ID or password.

Screen Shot 2013-10-01 at 11.13.38 AM

When you have a successful login, you will get a launcher that consists of several tabs. The Transcript serves the traditional (ANSI) Transcript function of showing output sent to the Transcript stream. The second tab shows information about your current session.

Screen Shot 2013-10-01 at 11.19.59 AM

The third tab shows information about the current logged-in sessions, including where the Gem is located, where the client GCI is located, and whether a Gem is holding the oldest commit record. If you have appropriate security, you can send a SigAbort to a session or even terminate it!

Screen Shot 2013-10-01 at 11.21.07 AM

The final tab is a Workspace. In this tab you can execute, print, and inspect Smalltalk code. You can also use the toolbar or menus to abort or commit and open other tools.

Screen Shot 2013-10-01 at 11.17.23 AM


One of the tools is a User Profile Browser that shows the various users defined in the database.


Screen Shot 2013-10-01 at 11.31.42 AM

Next is a Monticello Repository Browser that shows various repositories, packages, and versions.


Screen Shot 2013-10-01 at 11.32.47 AM

The Monticello Browser includes a tool to browse differences between packages.

Screen Shot 2013-10-01 at 11.33.36 AM


Much of your work will be done in a System Browser. This view shows the four SymbolDictionary instances in my SymbolList. UserGlobals is bold, indicating that it is the ‘home’ or default SymbolDictionary, but I have selected Globals and see a list of class categories, a partial list of classes, and in the lower section of the screen is a list of the non-class objects in Globals (note things like AllGroups and AllUsers).

Screen Shot 2013-10-01 at 11.41.42 AM


This screen shot shows us an example of the Packages view (which requires Monticello), and a method with a breakpoint (the red rectangle around a method).

Screen Shot 2013-10-01 at 11.43.54 AM

There are other tools, including a debugger, but I’ll leave that for your exploration (and/or another post/screencast).

Have fun and let me know if you have questions or feature requests.






A screencast of this post here.

GemStone/S 64 Bit has been available for the Macintosh for several years but you generally need to install and configure it much like you would do if you were on a Linux/Unix system. That is, there are a lot of command-line steps and system configurations that are needed. For someone who is used to the Macintosh’s consistent graphical user interface and who is not so familiar with configuring a Unix server, this tends to create a high barrier-to-entry. For a while I’ve been playing with an alternative that makes it easier to install and run GemStone/S 64 Bit on a Macintosh. This blog post will describe how to use this tool.

To start, download GemStoneApp.dmg, a disk image of a Cocoa application. Open the disk image to get to the virtual disk containing the application and a shortcut to your Applications folder. Copy the application to your local machine (typically, to the Applications folder, but it can be anywhere). At this point you can eject the virtual disk and delete the disk image. This will give you a 770 KB bundle that is a Cocoa application built with Xcode 5.

Screen Shot 2013-09-25 at 1.47.03 PM

You can launch the application in several ways. First, open Spotlight (Command + space) and type ‘gemstone’ (without the quotes). The application should be found and you can launch it by clicking on the name or just pressing Return. Second, you can use Launchpad, find the GemStone application, and then click on it. Finally, you can use the Finder to navigate to the folder holding the application (typically ‘/Applications/’) and launch it from there.

If the system presents a dialog reporting that the application could not be launched because it was not downloaded from the Mac App Store then you have a couple options. First, you can configure your security to allow applications from “identified developers.” This is done by launching System Preferences, selecting Security & Privacy, unlocking the page if needed (click on the padlock icon at the bottom left if it is closed), and then click the radio button for ‘Mac App Store and identified developers’ under the heading ‘Allow applications downloaded from:’. Once this is done, relaunch the application (as described in the previous paragraph) and confirm that you want to run it. Second, you can use the Finder to navigate to the directory holding the application (typically ‘/Applications/’) and then right-click or Control-click on the application and select the ‘Open’ menu item. This may ask you to confirm that you want to open the application. If you confirm once then it will not ask again.

Screen Shot 2013-09-25 at 2.03.42 PM

Once the application launches, make sure that the ‘Setup’ tab is selected. There are some setup steps that are typically done as root (using ‘sudo’ from a shell prompt) that we can do programatically if we have adequate authorization. These steps are done using a ‘Helper Tool’ that runs as root in the background and performs very limited actions. In our case, we need to set a couple kernel settings, kern.sysv.shmall and kern.sysv.shmmax, to allow for shared memory (this can be done manually, but is easier with the helper tool). Click the ‘More info’ button if you want to learn more, then click the ‘Authenticate…’ button, give your password, and then click the ‘Install Helper’ button. (You can click the ‘Remove’ button to remove the helper tool.)

Next you need to import the list of available versions by clicking the ‘Update’ button. When this finishes (it should only take a couple seconds), you will have a list of versions and their release dates. To install a version, click the checkbox next to the version name or (if you have already downloaded a zip file of the product tree from here or here), click on the ‘Unzip…’ button and select an existing zip file of a product tree. After the version is unzipped you can start to use it.

Screen Shot 2013-09-25 at 2.23.45 PM

Click on the Databases tab and click on the ‘+’ button to create a database. This will set up a directory structure, create a config file, and copy a base extent. You can change the version (if the database has not been used), edit the name of the stone, the NetLDI, and the shared page cache size. After you have made any changes you want, you can click the ‘Start’ button to start the stone (and related processes).

Screen Shot 2013-09-25 at 2.30.05 PM


In the Databases tab there are a series of sub-tabs, the first of which is ‘Data Files.’ Here you can select the extent(s) or tranlog(s) to see some information about them.

The second sub-tab gives you some backup and restore options. When the database is not running you can initialize a base extent (a copy of $GEMSTONE/bin/extent0.dbf) or a ‘GLASS’ extent (a copy of $GEMSTONE/bin/extent0.seaside.dbf), and you can restore from a backup. When the database is running you can make a backup.

The third sub-tab is ‘Process Logs’ and this gives you a list of log files associated with the GemStone processes. You can double-click a line (or select the line and click the ‘Open’ button), and the appropriate log file will open using the Macintosh Console application (used to view system logs).

The fourth sub-tab is ‘Archives’ and gives you some information about archived process logs (the text files described above) and the transaction logs. These can typically be deleted without impacting a running system (though you might want to keep transaction logs made following any backup if you have to do a restore).

The fifth sub-tab is ‘Statistics’ and shows a list of statmonitor files created while the system is running. If you double-click a line (or single-click and click the ‘Open’ button) then the application will launch ‘VSD’, a Visual Statistics Display tool that can be used to analyze the running system.

(If there is a sixth sub-tab, ‘Upgrade’, you should ignore it since it is disabled and does not do anything right now.)

After the Databases tab is a third tab, ‘GS List,’ that shows a list of the current processes. The port number for the NetLDI process might be useful.

(If there is a fourth tab, ‘Logins’, you should ignore it since it is disabled and does not do anything right now.)

Returning to the Databases tab, we have the ability to open a Finder window on the database directory (the button with a folder). Here you can use the Finder to explore the implementation details of the database. There is a ‘GemTools’ button that opens a text field with a Smalltalk expression that can be pasted into a GemTools session definition. Finally, there is a ‘Terminal’ button that can be used to open the Macintosh Terminal application. This starts a new Terminal application (this will be confusing if you already have one running) with the current working directory set along with various environment variables, including $GEMSTONE and $PATH. From this terminal window you can execute GemStone commands like ‘gslist’ and ‘topaz’.

Note that you can run multiple databases at one time and they can be different versions. When you are done you can click the ‘Stop’ button on the ‘Databases’ tab. When there are no databases running you can Quit the application (the application window can be minimized but not closed). Let me know if this is helpful and what further features you would like to see.


My post on a Cloud Foundry AMI was with a simple Ruby application. In a comment Andrew Spyker asked about a node.js application. I’ve done Javascript but not node.js, so thought I’d give it a try. I followed my earlier instructions to get the CF Micro instance started and then starting with the ‘Use the Server’ I did something different.

Using the webserver example here, I created two files in a new directory. The first file, example.js, contained the following:

var http = require('http');
var port = parseInt(process.env.PORT,10);
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(port, '');
console.log('Server running at' + port.toString() + '/');

The second file, package.json, contained the following:

  "name": "http-server",
  "version": "0.0.1",
  "author": "James Foster <github@jgfoster.net>",
  "description": "webserver demo from http://nodejs.org/",
  "dependencies" : [ ],
  "engines": {
    "node": ">=0.10"

(Bear in mind that this is my first node.js application, and I just spent a hour or so poking around on the web to get this far.)

From the command line I’m able to run it by entering the following:

export PORT=1337; node example.js

With this I can open a web browser on http://localhost:1337/ and see the greeting.

To push my trivial application to my EC2 instance, I did the following:

cf target http://api.<my-ip>.xip.io
cf login --password mySecret admin
# okay to ignore CFoundry::InvalidRelation error in next command
# (see https://github.com/cloudfoundry/cf/issues/9)
cf create-space development 
cf target --space development
cf map-domain --space development <my-ip>.xip.io
cf push --command "node example.js"

The interaction included giving the application a name (“hello”), accepting the defaults, and saving the configuration:

Name> hello
Instances> 1
1: 128M
2: 256M
3: 512M
4: 1G
Memory Limit> 256M
Creating hello... OK
1: hello
2: none
Subdomain> hello
2: none
Binding hello. to hello... OK
Create services for application?> n
Save configuration?> y
Saving to manifest.yml... OK
Uploading hello... OK
Preparing to start hello... OK
-----> Downloaded app package (4.0K)
-----> Resolving engine versions
 Using Node.js version: 0.10.17
 Using npm version: 1.2.30
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
 npm WARN package.json http-server@0.0.1 No repository field.
 npm WARN package.json http-server@0.0.1 No readme data.
 npm WARN package.json http-server@0.0.1 No repository field.
 npm WARN package.json http-server@0.0.1 No readme data.
 Dependencies installed
-----> Building runtime environment
-----> Uploading droplet (15M)
Checking status of app 'hello'...
 0 of 1 instances running (1 starting)
 0 of 1 instances running (1 starting)
 1 of 1 instances running (1 running)
Push successful! App 'hello' available at http://hello.

When I went to the URL provided, I saw the greeting.

I recently made an Amazon Machine Image (AMI) available (described here) and described how to use it. In this post I will describe the process of taking an existing AWS Instance with Cloud Foundry Micro (described here) and making it suitable for a public AMI. The primary challenge is that we need to create a boot disk with Cloud Foundry already installed but configured so that it can use a new password, domain, and local IP address. We also need to remove any identifying information before shutting down the VM and taking a snapshot.

I first use the following script (named setup.sh) to install Cloud Foundry using the cf_nise_installer. Note that I am setting a well-known IP, domain, and password; these will be changed later during the boot process. I am also deleting some files that might exist from a previous install attempt. Finally, I am creating a list of files that might contain data that must be modified during the boot process.

set -x
cd ~
sudo rm -rf \
  cf_nise_installer \
  /var/vcap/data/sys/log/* \
  domain password files ipv4 \
export INSTALLER_URL="https://github.com/yudai/cf_nise_installer.git"
export INSTALLER_BRANCH='master'
export CF_RELEASE_BRANCH='release-candidate'
export NISE_DOMAIN='cloud.mycloud.local'
export NISE_PASSWORD='swordfish'
bash < <(curl -s -k -B https://raw.github.com/yudai/cf_nise_installer/${INSTALLER_BRANCH:-master}/local/bootstrap.sh)
list="$(sudo find /vol-a/var/vcap -type f \( -iname '*.yml' -o -iname 'postgres_ctl' \) )"
echo $list > ~/files

We next create a startup script, /home/ubuntu/rc.local, and modify /etc/rc.local to call our new script (if you delete the authorized_keys and fail to add them back, then you can’t log in to your instance!).

# should be called from /etc/rc.local
echo "Starting /home/ubuntu/rc.local script to start Cloud Foundry"
# See if 'debug: true' is in user-data
X=$( curl http://instance-data/latest/user-data 2>/dev/null | grep 'debug:' )
X=$( echo $X | cut -d : -f 2 | tr -d ' ' )
if [ "$X" == "true" ]; then
  set -x
# Make sure that we have some keys (should be deleted before snapshot)
cd /home/ubuntu
if [ ! -f .ssh/authorized_keys ]; then
  curl http://instance-data/latest/meta-data/public-keys/0/openssh-key \
    2>/dev/null > .ssh/authorized_keys
  chown ubuntu:ubuntu .ssh/authorized_keys
  chmod 600 .ssh/authorized_keys
  echo "Updated authorized_keys"
# See if 'autoStart: false' is in user-data; if so, done!
X=$( curl http://instance-data/latest/user-data 2>/dev/null | grep 'autoStart:' )
X=$( echo $X | cut -d : -f 2 | tr -d ' ' )
if [ "$X" == "false" ]; then
  echo "user-data includes 'autoStart: false' so we will quit"
  exit 0
# Each instance should have a different password (but once set it does not change!)
if [ -e password ]; then
  # If we have been through this code before, then use the previous password
  OLD_PASSWORD=$( < password )
  # This is the first time through, so we replace the default password
  # User may specify a 'password:' line in the user data
  X=$( curl http://instance-data/latest/user-data 2>/dev/null | grep 'password:' )
  NEW_PASSWORD=$( echo $X | cut -d : -f 2 | tr -d ' ' )
  if [ "$NEW_PASSWORD" == "" ]; then
    # No user-provided one and no previously created one, so assign a random one
    # another script is $( openssl rand -base64 6 )
    NEW_PASSWORD=$( < /dev/urandom tr -dc A-Za-z0-9 | head -c8 )
  # Save new password so it can be reused next time
  echo "$NEW_PASSWORD" > password
# See if 'domain:' is in user-data
X=$( curl http://instance-data/latest/user-data 2>/dev/null | grep 'domain:' )
if [ "$X" != "" ]; then
  NEW_DOMAIN=$( echo $X | cut -d : -f 2 | tr -d ' ' )
  # get the public-hostname
  until [ "$NEW_DOMAIN" != "" ]; do
    X=$( curl http://instance-data/latest/meta-data/ 2>/dev/null | grep public-hostname )
    if [ "$X" != "" ]; then
      X=$( curl http://instance-data/latest/meta-data/public-hostname 2>/dev/null )
      if [ "$X" != "" ]; then
        NEW_DOMAIN=$( echo $X | cut -d "." -f 1 | cut -d "-" -f 2- | sed "s/\-/\./g" )
    if [ "$NEW_DOMAIN" == "" ]; then
      echo "`date`: public-hostname not yet available from meta-data"
      sleep 1
if [ -f domain ]; then
  OLD_DOMAIN=$( cat domain )
echo $NEW_DOMAIN > domain
# Each new instance will have a unique private IP address
NEW_IPV4=$( curl http://instance-data/latest/meta-data/local-ipv4 2>/dev/null )
if [ -f ipv4 ]; then
  OLD_IPV4=$( cat ipv4 )
echo $NEW_IPV4 > ipv4
# Find all the files that need to be edited
# (takes several seconds, so cache it for faster start next time)
if [ -f files ]; then
  list=$( cat files )
  list="$(find /var/vcap -type f \( -iname '*.yml' -o -iname 'postgres_ctl' \) )"
  echo $list > files
for f in $list
  if [ "$OLD_PASSWORD" != "$NEW_PASSWORD" ]; then
    if grep -q $OLD_PASSWORD $f; then
      sed -i "s/$OLD_PASSWORD/$NEW_PASSWORD/g" $f
      echo "Updated password in $f"
  if grep -q $OLD_DOMAIN $f; then
    sed -i "s/$OLD_DOMAIN/$NEW_DOMAIN/g" $f
    echo "Updated domain in $f"
  if grep -q "$OLD_IPV4" $f; then
    sed -i "s/$OLD_IPV4/$NEW_IPV4/g" $f
    echo "Updated IP in $f"
# start Cloud Foundry
(cd /home/ubuntu/cf_nise_installer; ./local/start_processes.sh)

Finally, I create a cleanup.sh script to remove identifying information and sanitize the system before shutdown. Note that once authorized_keys are removed from the .ssh directory you can’t log in unless during the boot process new authorized_keys are installed.

sudo rm -f /root/.ssh/* /home/*/.ssh/* /var/vcap/monit/monit.log
(cd /var/log; sudo rm -f *.log dmesg* debug messages syslog)
rm -f ~/.viminfo ~/.bash_history
echo '' | sudo tee /var/log/lastlog
history -c
# sudo shutdown now

After running this script I use the AWS tools to stop the instance (this means that the command history is empty). At this point I can use the AWS tools to create an AMI. When the AMI starts it executes /etc/rc.local which calls my new script, /home/ubuntu/rc.local. This script sets the local IPv4 value, along with the domain and password (using configured values if provided in the user-data). When the appropriate configuration info has been updated, then we start Cloud Foundry. Each new instance has its own IP, domain, and password, making it secure and unique.

If there is something wrong with the boot volume (such as missing authorized_keys so you can’t log in), then you need to start another EC2 instance (a micro is fine), attach the volume, do the fix-up, and release it.

# mount a new volume (in case some surgery is needed)
sudo mkdir -p 000 /vol-a
sudo mount /dev/sdf /vol-a
# ... do whatever fix-up is needed and unmount the volume
sudo umount -d /vol-a
sudo rmdir /vol-a

I have used /etc/rc.local to hook into the boot process. An alternate may be to use crontab with the ‘@reboot’ option.

If you have an instance-store (a disk that exists only while the instance is running), then you might want to set up some swap space on it. The following script will create a 4 GB file to be used as swap space.

# http://serverfault.com/questions/218750/why-dont-ec2-ubuntu-images-have-swap
sudo dd if=/dev/zero of=/mnt/swapfile bs=1M count=4096 &&
sudo chmod 600 /mnt/swapfile &&
sudo mkswap /mnt/swapfile &&
echo /mnt/swapfile none swap defaults 0 0 | sudo tee -a /etc/fstab &&
sudo swapon -a
cat /proc/swaps

That summarizes how I created a Cloud Foundry Micro public AMI.

Video of ESUG 2013 Presentation

My presentation “Smalltalk in the Cloud” was recorded and can be found at the link. Unfortunately, the audio was not very strong.

Update: Slides are here.