You are currently browsing the category archive for the ‘GemStone’ category.
A 5-minute screencast of this blog post can be found here.
Would you like to run GemStone/S 64 Bit on a Microsoft Windows machine? If so, then the Windows Subsystem for Linux makes that possible. Starting with a 64-bit version of Windows 10, follow these steps:
Install Ubuntu 18.04
- Run Windows PowerShell as Administrator and enter the following (on one line):
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux - Confirm a restart to complete the process.
- From the Windows Store application, install Ubuntu 18.04, pin it to the Start menu, and then launch it.
- When prompted, enter a new UNIX username and password.
Install GemStone
- Update your system and install the unzip utility:
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo apt-get install unzip - In the Linux shell, create some GemStone directories:
sudo mkdir /opt/gemstone
sudo chown $USER /opt/gemstone
cd /opt/gemstone
mkdir locks log - In a web browser, navigate to the GemStone product page and copy the link for the Linux download. Use wget to download the product (the following is supposed to be one line, but WordPress hides some; copy the whole thing and past into a text editor):
wget https://downloads.gemtalksystems.com/pub/GemStone64/3.4.3/GemStone64Bit3.4.3-x86_64.Linux.zip - Unzip the product:
unzip GemStone* - Create a symbolic link to the unzipped directory:
ln -s GemStone* product - Edit your profile to add/update environment variables:
vim ~/.profile # add the following two lines
export GEMSTONE=/opt/gemstone/product
export PATH=$GEMSTONE/bin:$PATH - Update your environment to include the above variables by doing one of the following:
- Enter the two lines from step 6 into your command shell;
source ~/.profile; or,- Close your shell and open a new shell.
- Edit the services file to add an entry for NetLDI:
sudo vim /etc/services # add the following line
gs64ldi 50377/tcp # GemStone/S 64 Bit 3.x - Copy in a new database:
cp $GEMSTONE/bin/extent0.dbf $GEMSTONE/data/
chmod +w $GEMSTONE/data/extent0.dbf
Start GemStone
Open a new Linux shell to get your environment variables and start GemStone:
gslist
startnetldi -g -a $USER
startstone
gslist
Install Jade into Windows
- Open a web browser on the Jade Releases.
- Download (Save) Jade.zip (one of the assets) to your Downloads folder.
- Open the Downloads folder and extract the download.
- Copy the entire Jade folder to
C:\Program Files (x86)or any handy directory (it doesn’t need to be anywhere in particular). - Pin
Jade.exeto the Start Menu. - Edit
C:\Windows\System32\drivers\etc\services(as a Windows administrator using a Windows text editor such as Notepad++) to add an entry for NetLDI:
gs64ldi 50377/tcp # GemStone/S 64 Bit 3.4.2
Log in to GemStone with Jade
- From the Start Menu, launch Jade. If Windows identifies Jade as an unrecognized app, click More info and then click the Run anyway button.
- Ensure that the version shown in Jade is a close match to the GemStone version, and click the Login button.
- Now you can explore your GemStone/S database that is running in Windows!
Stop GemStone
From a Linux shell you can stop GemStone:
gslist
stopstone gs64stone DataCurator swordfish
stopnetldi
gslist
A screencast of this blog post is here.
Because Smalltalk was the origin of much of today’s GUI (mouse, overlapping windows, drop-down menus), Smalltalk developers are understandably accustomed to a nice GUI IDE. GemStone/S is an excellent database and Smalltalk execution environment and includes a built-in command-line tool, Topaz, where you can execute Smalltalk code, but has no native GUI. In this blog post we continue a demonstration of GemStone.app on the Macintosh (started here) and show Jade, a GUI-based IDE available on Microsoft Windows.
We launch GemStone.app on the Macintosh, update the version list, download 3.1.0.4, then install and start a GLASS extent (image) that includes Monticello/Metacello tools. When the database is running we start a Topaz session and install Seaside 3.0 and Magritte 3 with the following script:
run "based on https://code.google.com/p/glassdb/wiki/Seaside30Configuration" MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [ ConfigurationOfMetacello project updateProject. ConfigurationOfMetacello loadLatestVersion. Gofer project load: 'Seaside30' group: 'Seaside-Adaptors-Swazoo'. ]. % errorCount ! commit run "based on http://www.iam.unibe.ch/pipermail/smallwiki/2012-February/007188.html" Gofer it squeaksource: 'MetacelloRepository'; package: 'ConfigurationOfMagritte3'; load. % errorCount ! commit run MCPlatformSupport commitOnAlmostOutOfMemoryDuring: [ ConfigurationOfMagritte3 project stableVersion load. ]. % errorCount ! commit run WAGsSwazooAdaptor new start. %
When Swazoo is running, we can go to http://localhost:8080 and see Seaside running locally. This demonstrates running Smalltalk code in Topaz, the command-line tool. Next we look at a GUI-based IDE that runs on a Microsoft Windows client platform.
Jade is available as a 14 MB zip download from http://seaside.gemtalksystems.com/jade/. It includes an executable, client libraries (DLLs) for various GemStone/S versions (ranging from 32-bit version 6.1 to the latest 64-bit version), and related files (including source code). Like most GemStone/S client GUI tools, it is built in another Smalltalk (in this case, Dolphin Smalltalk from Object Arts), but unlike these other tools, you can’t see the client Smalltalk (unless your load Jade source code into your own Dolphin development environment), so we avoid the the two-object-space confusion. Jade is intended to take you directly to GemStone/S, without going through Pharo, Squeak, VA, or VW Smalltalk.
Jade is also designed to work with the no-cost version of GemStone/S (unlike the VA/VW-based GBS tools), and performs well on a slow network (unlike GemTools).
When you unzip the download, you have a folder with various items. Jade.exe is the primary executable (containing the Dolphin VM and the image) and it relies on Microsoft’s C Runtime Library. There is a copy of the executable in Jade.jpg for sites where executables are stripped from zip files during the download process (simply rename the suffix and it will become executable). Contacts.exe is used sometimes in a training class. The bin directory contains the GCI client libraries and a DLL with various images used in the IDE. You can also see a directory containing source code for Jade.
When you launch Jade, you get a login window that gives you a place to select the GemStone/S version (which GCI library we will use), and other information needed for a login. The Stone box contains two fields, one for the host/IP of the Stone machine (from the perspective of the Gem machine, so localhost is almost always sufficient), and the name of the Stone. In the screencast mentioned above our stone was named gs64stone1. The Gem box contains a number of fields. Most logins will use an RPC Gem (a Linked Gem is available only on 32-bit Windows) with a Guest-authenticated Gem (if your NetLDI was not started in guest mode (-g), then you will need to provide an OS user and password). An RPC Gem will be started by a NetLDI, so we need to identify the Gem machine (in my example the host is vienna and the NetLDI is listening on port 54120) and the command used to start the Gem (except in rare cases the command will be ‘gemnetobject’). You provide a GemStone User ID and Password (by default, ‘DataCurator’ and ‘swordfish’), and if you are going to use any Monticello features it would be good to identify the developer’s name (one word with CamelCase).
If you get an error on login, we attempt to give as much explanation as possible. Typically, (1) there is no NetLDI on the host/port (see following example), (2) there is no stone with that name, (3) there is a version mismatch, or (4) you have given an unrecognized user ID or password.
When you have a successful login, you will get a launcher that consists of several tabs. The Transcript serves the traditional (ANSI) Transcript function of showing output sent to the Transcript stream. The second tab shows information about your current session.
The third tab shows information about the current logged-in sessions, including where the Gem is located, where the client GCI is located, and whether a Gem is holding the oldest commit record. If you have appropriate security, you can send a SigAbort to a session or even terminate it!
The final tab is a Workspace. In this tab you can execute, print, and inspect Smalltalk code. You can also use the toolbar or menus to abort or commit and open other tools.
One of the tools is a User Profile Browser that shows the various users defined in the database.
Next is a Monticello Repository Browser that shows various repositories, packages, and versions.
The Monticello Browser includes a tool to browse differences between packages.
Much of your work will be done in a System Browser. This view shows the four SymbolDictionary instances in my SymbolList. UserGlobals is bold, indicating that it is the ‘home’ or default SymbolDictionary, but I have selected Globals and see a list of class categories, a partial list of classes, and in the lower section of the screen is a list of the non-class objects in Globals (note things like AllGroups and AllUsers).
This screen shot shows us an example of the Packages view (which requires Monticello), and a method with a breakpoint (the red rectangle around a method).
There are other tools, including a debugger, but I’ll leave that for your exploration (and/or another post/screencast).
Have fun and let me know if you have questions or feature requests.
A screencast of this post here.
GemStone/S 64 Bit has been available for the Macintosh for several years but you generally need to install and configure it much like you would do if you were on a Linux/Unix system. That is, there are a lot of command-line steps and system configurations that are needed. For someone who is used to the Macintosh’s consistent graphical user interface and who is not so familiar with configuring a Unix server, this tends to create a high barrier-to-entry. For a while I’ve been playing with an alternative that makes it easier to install and run GemStone/S 64 Bit on a Macintosh. This blog post will describe how to use this tool.
To start, download GemStoneApp.dmg, a disk image of a Cocoa application. Open the disk image to get to the virtual disk containing the application and a shortcut to your Applications folder. Copy the application to your local machine (typically, to the Applications folder, but it can be anywhere). At this point you can eject the virtual disk and delete the disk image. This will give you a 770 KB bundle that is a Cocoa application built with Xcode 5.
You can launch the application in several ways. First, open Spotlight (Command + space) and type ‘gemstone’ (without the quotes). The application should be found and you can launch it by clicking on the name or just pressing Return. Second, you can use Launchpad, find the GemStone application, and then click on it. Finally, you can use the Finder to navigate to the folder holding the application (typically ‘/Applications/’) and launch it from there.
If the system presents a dialog reporting that the application could not be launched because it was not downloaded from the Mac App Store then you have a couple options. First, you can configure your security to allow applications from “identified developers.” This is done by launching System Preferences, selecting Security & Privacy, unlocking the page if needed (click on the padlock icon at the bottom left if it is closed), and then click the radio button for ‘Mac App Store and identified developers’ under the heading ‘Allow applications downloaded from:’. Once this is done, relaunch the application (as described in the previous paragraph) and confirm that you want to run it. Second, you can use the Finder to navigate to the directory holding the application (typically ‘/Applications/’) and then right-click or Control-click on the application and select the ‘Open’ menu item. This may ask you to confirm that you want to open the application. If you confirm once then it will not ask again.
Once the application launches, make sure that the ‘Setup’ tab is selected. There are some setup steps that are typically done as root (using ‘sudo’ from a shell prompt) that we can do programatically if we have adequate authorization. These steps are done using a ‘Helper Tool’ that runs as root in the background and performs very limited actions. In our case, we need to set a couple kernel settings, kern.sysv.shmall and kern.sysv.shmmax, to allow for shared memory (this can be done manually, but is easier with the helper tool). Click the ‘More info’ button if you want to learn more, then click the ‘Authenticate…’ button, give your password, and then click the ‘Install Helper’ button. (You can click the ‘Remove’ button to remove the helper tool.)
Next you need to import the list of available versions by clicking the ‘Update’ button. When this finishes (it should only take a couple seconds), you will have a list of versions and their release dates. To install a version, click the checkbox next to the version name or (if you have already downloaded a zip file of the product tree from here or here), click on the ‘Unzip…’ button and select an existing zip file of a product tree. After the version is unzipped you can start to use it.
Click on the Databases tab and click on the ‘+’ button to create a database. This will set up a directory structure, create a config file, and copy a base extent. You can change the version (if the database has not been used), edit the name of the stone, the NetLDI, and the shared page cache size. After you have made any changes you want, you can click the ‘Start’ button to start the stone (and related processes).
In the Databases tab there are a series of sub-tabs, the first of which is ‘Data Files.’ Here you can select the extent(s) or tranlog(s) to see some information about them.
The second sub-tab gives you some backup and restore options. When the database is not running you can initialize a base extent (a copy of $GEMSTONE/bin/extent0.dbf) or a ‘GLASS’ extent (a copy of $GEMSTONE/bin/extent0.seaside.dbf), and you can restore from a backup. When the database is running you can make a backup.
The third sub-tab is ‘Process Logs’ and this gives you a list of log files associated with the GemStone processes. You can double-click a line (or select the line and click the ‘Open’ button), and the appropriate log file will open using the Macintosh Console application (used to view system logs).
The fourth sub-tab is ‘Archives’ and gives you some information about archived process logs (the text files described above) and the transaction logs. These can typically be deleted without impacting a running system (though you might want to keep transaction logs made following any backup if you have to do a restore).
The fifth sub-tab is ‘Statistics’ and shows a list of statmonitor files created while the system is running. If you double-click a line (or single-click and click the ‘Open’ button) then the application will launch ‘VSD’, a Visual Statistics Display tool that can be used to analyze the running system.
(If there is a sixth sub-tab, ‘Upgrade’, you should ignore it since it is disabled and does not do anything right now.)
After the Databases tab is a third tab, ‘GS List,’ that shows a list of the current processes. The port number for the NetLDI process might be useful.
(If there is a fourth tab, ‘Logins’, you should ignore it since it is disabled and does not do anything right now.)
Returning to the Databases tab, we have the ability to open a Finder window on the database directory (the button with a folder). Here you can use the Finder to explore the implementation details of the database. There is a ‘GemTools’ button that opens a text field with a Smalltalk expression that can be pasted into a GemTools session definition. Finally, there is a ‘Terminal’ button that can be used to open the Macintosh Terminal application. This starts a new Terminal application (this will be confusing if you already have one running) with the current working directory set along with various environment variables, including $GEMSTONE and $PATH. From this terminal window you can execute GemStone commands like ‘gslist’ and ‘topaz’.
Note that you can run multiple databases at one time and they can be different versions. When you are done you can click the ‘Stop’ button on the ‘Databases’ tab. When there are no databases running you can Quit the application (the application window can be minimized but not closed). Let me know if this is helpful and what further features you would like to see.
If you want to just look at the code in a GemStone/S image, and maybe try out a simple Smalltalk expression, you can spin up a Heroku dyno running GemStone/S. See the buildpack for instructions on how to run Webtools. Webtools is itself still quite rough, but if you want to contribute or open an issue, feel free.
Of course, this isn’t very useful as a database or application server because the dyno has an ephemeral file system and can be restarted any time. But it does demonstrate that I’ve figured out how to create a Heroku buildpack, and that was my primary goal!
A GemStone/S 64 Bit system needs to have various maintenance tasks performed regularly. One of these is managing log file, backups, and transaction logs. I recently put together a basic Linux VM that has GemStone/S 64 Bit installed, but little more (the last link here). One of the non-Smalltalk things that I sometimes struggle with is figuring out which file can be deleted. The following demonstrates some bash scripting that seems to address a few of my questions. It starts by deleting all but the most recent two log files for pcmon, pagemanager, and symbolgem. It then does some garbage collection activity and a full backup. It then deletes all but the most recent two backups. Finally, if there are two backups in the last two days, it deletes all transaction logs more than two days old. This does not do as much error checking as should be done in a serious production environment (such as try restoring the backup, apply transaction logs, etc., before deleting things), but it demonstrates some of the things that can be done in bash to do date-specific cleanup.
#!/bin/bash
# cleanup log files
cd /opt/gemstone/log
(ls -t *pcmon.log | head -n 2; ls *pcmon.log) | \
sort | uniq -u | xargs rm
(ls -t *pagemanager.log | head -n 2; ls *pagemanager.log) | \
sort | uniq -u | xargs rm
(ls -t *symbolgem.log | head -n 2; ls *symbolgem.log) | \
sort | uniq -u | xargs rm
# do backup
topaz -l -T 50000 << EOF
output push backup.out
errorCount
send SystemRepository markForCollection
send SystemRepository reclaimAll
send SystemRepository startNewLog
run
SystemRepository fullBackupCompressedTo:
'/opt/gemstone/backups/backup-' ,
(DateTime now asSeconds // 60) printString.
%
logout
errorCount
exit
EOF
# cleanup backups
cd /opt/gemstone/backups
(ls -t backup* | head -n 2; ls backup*) | sort | uniq -u | xargs rm
if [ "2" -le "`find . -mtime -2 -name 'backup*' | wc -l`" ]; then
cd /opt/gemstone/data
find . -mtime +2 -name 'tranlog*' | xargs rm
fi
While there are a variety of ways of interacting with GemStone/S 64 Bit, the most basic way is to use Topaz, the command-line GemStone C Interface (GCI) client that has been part of GemStone since the beginning. While GemStone/S 64 Bit as a server is not available for Microsoft Windows, Topaz is available as a Windows client application that can be used to connect to a Unix/Linux/Mac server. Using Topaz is very helpful in debugging connectivity problems since it removes the variables associated with other client applications (such as GBS, GemTools, Jade, etc.). That is, if you are having trouble connecting with GemTools, we are likely to ask you to try to connect using Topaz.
Fortunately, it is relatively easy to run Topaz on Windows. Open a web browser on http://seaside.gemstone.com/downloads/x86.Windows_NT/ and download GemBuilderC3.1.0.1-x86.Windows_NT.zip (this assumes that you are connecting to a 3.1.0.1 server). Unzip this into a convenient location on your Windows machine (e.g., C:\gemstone\). Open a command shell (Start, All Programs, Accessories, Command Prompt), navigate to the ‘bin’ directory created by unzipping the download, and start ‘topaz’. At this point you can enter the usual Topaz commands and try to login to your server. Following is a copy of the screen when I used Topaz on Windows to login to my database:
C:\GemStone\GemBuilderC3.1.0.1-x86.Windows_NT\bin>topaz _____________________________________________________________________________ | GemStone/S64 Object-Oriented Data Management System | | Copyright (C) VMware, Inc. 1986-2012 | | All rights reserved. | +-----------------------------------------------------------------------------+ | PROGRAM: topaz, Linear GemStone Interface (Remote Session) | | VERSION: 3.1.0.1, Fri Aug 24 10:24:31 2012 | | BUILD: gss64_3_1_0_x_branch-28937 | | BUILT FOR: Pentium/Windows_NT | | MODE: 32 bit | | RUNNING ON: 1-CPU jfoster-xpvm: Intel CPU, Windows NT 5.1 build 2600 Service| | Pack 3 | | PROCESS ID: 828 DATE: 11/28/2012 11:19:36 Pacific Standard Time | |_____________________________________________________________________________| neither topazini.tpz nor $HOME\topazini.tpz were found topaz> set user DataCurator pass swordfish topaz> set gemstone jfoster0 topaz> set gemnet !tcp@10.118.32.15#netldi:10460#task!gemnetobject topaz> login [Info]: libssl-3.1.0.1-32.dll: loaded [11/28/2012 11:20:47.435 Pacific Standard Time] gci login: currSession 1 rpc gem processId 30480 OOB keep-alive interval 0 successful login topaz 1> run 100 factorial printString % 93326215443944152681699238856266700490715968264381621468592963895217599993229915 608941463976156518286253697920827223758251185210916864000000000000000000000000 topaz 1> logout topaz> exit
C:\GemStone\GemBuilderC3.1.0.1-x86.Windows_NT\bin>
The things I typed are in bold. The things you need to change are in italics. Specifically, you need to provide the name of your stone (perhaps it is ‘seaside’), the IP address (or hostname if you have an entry in your hosts file) and port number (or service name if you have an entry in your services file) for your NetLDI that will start your gem.
One of the most common problem people have installing and starting GemStone/S is getting shared memory configured properly. This post will discuss shared memory in general and GemStone’s use of shared memory in particular.
One of the features of modern operating systems is that each operating system process runs in its own address space and (with help from the hardware) the process is not permitted to read or write outside its assigned area. This protection makes it much less likely that a misbehaving application will harm other applications or even crash your whole machine (as was common on personal computers during the era prior to Windows XP and Mac OS X). This protection comes with the overhead that if two processes want to share data they typically need to communicate through the operating system (typically using TCP/IP, as if they were on separate machines). Processes that want to share a large amount of data are allowed to request a chunk of memory from the operating system that can be shared. Because this is not common, most operating systems are configured to provide only a limited amount of memory for this use and without adjustments to those settings GemStone cannot run.
Each machine where a GemStone process is running will have a Shared Page Cache (SPC) that holds copies of 16 KB pages from the file system extents that make up the repository. The SPC is a chunk of shared memory allocated by a SPC “monitor” and to which other processes attach. When the stone starts it forks a process to be the SPC monitor and then waits for the SPC to become available. If the SPC does not appear within a set time then the stone reports an error and dies. This error will be similar to the following:
The stone was unable to start a cache page server on host '<stone's host>'. Reason: The cache monitor connect failed. Monitor process (9999) did not start.
The SPC monitor log (a file next to the stone log whose name ends with ‘pcmon.log’) will have the following:
| GemStone could not retrieve the IPC identifier associated with the memory | | key 9999999. shmget() error = errno=22,EINVAL, Invalid argument (programmer | error). | | | GemStone could not attach to the shared page cache. | [SpcMon trace]: ... cache creation failed ... [SpcMon trace]: ... if the errno is (EINVAL) it is likely because the cache size is less than the operating system imposed minimum or greater than the operating system maximum.
As suggested in the error message, the cache size is likely greater than the operating system maximum. If this is the first error you get after installing GemStone then it is likely that you have not configured the operating system for adequate shared memory. To see the existing limits, you can ask for a listing of the inter-process communication statistics:
ipcs -l
This provides a list similar to the following (from my Ubuntu 10.04 desktop machine at work):
------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 4592442 max total shared memory (kbytes) = 4592440 min seg size (bytes) = 1
This machine is configured to make 4 GB of RAM available to be used as shared memory, and the maximum segment is also 4 GB. We can also query the current settings with the following command:
sysctl -a 2> /dev/null | grep kernel.shm
On my machine this reports the following:
kernel.shmmax = 4702660608 kernel.shmall = 1148110 kernel.shmmni = 4096
These numbers mean the same thing as shown above, but the first two lines use different values. While ipcs reports both the max segment size and the max overall shared memory as kbytes, sysctl reports the max segment size (kernel.shmmax) as bytes and the max overall shared memory (kernel.shmall) as 4 KB pages.
These settings are configured at boot time by reading /etc/sysctl.conf and applying defaults if no values are provided. To see the configured values, enter the following:
grep kernel.shm /etc/sysctl.conf
In my case I see the following lines:
kernel.shmmax = 4702660608 kernel.shmall = 1148110
If you edit /etc/sysctl.conf (as root using sudo) and reboot your machine then the new values should take effect. If you want to make changes without rebooting your machine, use something like the following:
sudo sysctl -w kernel.shmmax=4702660608
Once you have things properly configured then the SPC monitor should be able to allocation a SPC and the stone should start. To see the memory use, you can enter the following
ipcs
On my machine the output includes the following:
------ Shared Memory Segments -------- key shmid owner perms bytes nattch status ... 0x00000000 1409046 jfoster 600 393216 2 dest 0xe50206db 8683543 jfoster 660 529203200 8 locked ...
There are a number of segments of 384 KB that are marked to be destroyed (dest), and one segment of about 500 MB. My configuration file includes the following:
SHR_PAGE_CACHE_SIZE_KB = 500000;
The pcmon.log file shows that the number of pages requested:
Number of pages 31250.
Since each page is 16 KB, this would request 512000000 bytes for pages. It turns out that there are other data structures that need to be shared, including the statistics captured by statmonitor, and the memory requested will be slightly larger than the configured amount. This means that you should set shmmax and shmall to be larger than your SHR_PAGE_CACHE_SIZE_KB setting (or ask for less than the configured maximum).
The GemStone/S 64 Bit Install Guide has a good discussion of how to calculate shared memory needs, but at present it contains an error in that the shmmax setting is incorrectly shown as being the same as the shmall setting. Remember that shmmax is in bytes and shmall is in pages (typically 4 KB).
A recent discussion on the GLASS mailing list presented an interesting problem. Say you have a collection of domain model objects that each have a start date and an stop date, and you want to find all the objects that are within a date range. I was first exposed to this problem 25 years ago in a healthcare system tracking patient admit and discharge dates, but it applies to many common situations (from hotels to video or car rentals).
For this discussion we will consider the following example: from the collection of [A-H] we want to get the subset from begin to end, or [D-F].
Of course, one could iterate over the entire collection and test each object, but this will be inefficient for large collections. At the other extreme would be to have an end-of-day job that create a collection for that date and add items to the day’s collection. This would give very quick lookup, but would have a cost in storage.
A typical attempt to solve this problem would be to create an index on the start date and the stop date and then build a result set of items that start before the end point and stop after the begin point. The problem with this approach is that it likely creates two intermediate results, [A-F] and [D-H], that together are larger than the original collection (before returning the intersection of those intermediate results). In GemStone, this can generally be done in such a way that the intermediate results hold the object IDs (or OOPs), but the objects are not actually read from disk. This is typically a good solution, but others are possible.
Decades ago my good friend, Carl Zimmerman, came up with another approach that uses one index and avoids the large intermediate results. The following is adapted from his design to work with GemStone’s indexing system.
Each object has an ‘indexableDateRange’ instance variable (with an equality index defined) that encodes the duration (in days) and the start point (a Date after 1900) as a SmallInteger. The value is set as follows:
indexableDateRange := start isNil
ifTrue: [0]
ifFalse: [stop isNil
ifTrue: [start asDays]
ifFalse: [(stop subtractDate: start) + 1 * 100000 + start asDays]].
Note a few things about this expression. If the value is zero, then the activity has not started. If the value is less than 100000, then the activity is ongoing. If the value is greater than 100000, then the activity has finished. Note that this design handles dates from 1901 to 2173. Handling dates outside that range or other units, such as minutes, is left as an exercise for the reader (or an opportunity for consulting!).
Let’s start by creating some test data. The following creates 1000 objects that have a start date from 2000 through most of 2009 and durations of up to 10 days (or still in progress).
| set random origin |
set := UserGlobals at: #'James' put: IdentitySet new.
set createEqualityIndexOn: #'key' withLastElementClass: SmallInteger.
System commitTransaction.
random := Random new.
origin := Date newDay: 1 monthNumber: 1 year: 2000.
1000 timesRepeat: [
| start stop key |
start := origin addDays: (random integerBetween: 1 and: 3650).
stop := start addDays: (random integerBetween: 0 and: 10).
stop = start ifTrue: [stop := nil].
key := start isNil
ifTrue: [0]
ifFalse: [stop isNil
ifTrue: [start asDays]
ifFalse: [(stop subtractDate: start) + 1 * 100000 + start asDays]].
set add: key -> (Array with: start with: stop).
].
System commitTransaction.
The following demonstrates a “brute-force” search that tests every object:
| start stop set resultSet | set := UserGlobals at: #'James'. start := (Date newDay: 1 monthNumber: 1 year: 2005) asDays. stop := start + 6. resultSet := set select: [:each | each value first asDays <= stop and: [each value last isNil or: [start <= each value last asDays]]. ]. resultSet
The following code demonstrates the Zimmerman search algorithm:
| start stop set resultSet duration |
set := UserGlobals at: #'James'.
start := (Date newDay: 1 monthNumber: 1 year: 2005) asDays.
stop := start + 6.
resultSet := IdentitySet new.
resultSet addAll: (set select: {:each | each.key <= stop}). "Items that are in-progress"
duration := 100000. "start at one-day duration"
[true] whileTrue: [
| low high stream |
low := duration + start - (duration // 100000).
high := duration + stop.
resultSet addAll: (set select: {:each | (each.key >= low) & (each.key <= high)}).
duration := duration + 100000.
stream := set selectAsStream: {:each | each.key >= duration}.
stream atEnd ifTrue: [^resultSet].
duration := stream next key // 100000 * 100000.
].
resultSet
This approach starts with all the events that started before the stop point and are still in progress. Then it iterates over the completed items starting with those that have a one-day duration. A range of keys is computed based on the start, stop, and current duration, and the equality index is used to get the matching items for that duration. For domains in which there are a limited number of durations (such as video rentals), there are relatively few intermediate result sets and none have any unneeded objects. Given GemStone’s efficient handling of IdentitySets and intersection operations, I suspect that this is not worth the trouble, but it is an interesting approach. This approach does fault in one extra object for each duration after 1, but since the needed information is actually in the indexing data structures, this could be avoided.
Anyway, it is something to think about (and measure).
The default GLASS setup starts a ‘maintenance gem’ that performs two tasks: (1) expiring Seaside sessions and (2) performing repository-wide garbage collection (‘mark for collection’ or MFC). This maintenance gem is configured to use up to 200 MB for temporary object space and with other memory allocations, such as for persistent objects, the total memory usage can be twice that amount. It remains logged in continually expires sessions every minute and doing an MFC every hour.
The original no-cost license allowed for up to 1 GB shared page cache (SPC) and up to 4 GB for the total repository size. In a busy system where you could have at least 25% of the object space in memory (and had adequate memory for each of the Gems) doing an hourly MFC was important (to avoid having excess garbage) and not too expensive.
In some situations, however, the maintenance gem may be causing excess overhead. If you are running GLASS in the “cloud” (e.g., on SliceHost or some other virtual server), then the cost of RAM may provide a significant constraint on your SPC. Also, if your system is not used heavily, then there might not be enough garbage (primarily from expired sessions) to justify frequent MFC. Finally, since the no-cost license now allows for unlimited repository size, the consequence of running out of space is not so dire.
This is important because MFC is a comparatively heavy-weight operation and can take a lot of time. On some larger customer databases, it can take many days and the user experience often suffers. The size of the database is really not the most important factor; more important is the percent of the database that can fit in the SPC. If you are running in the cloud with 512 MB of RAM and are allowed only 1/8th of a CPU, then a 2 GB database in a 256 MB SPC could see a significant decrease in performance during an MFC.
To determine the necessary MFC frequency, take a look at the maintenance gem log. This will show the number of sessions expired each minute and the number of possible dead objects found during each MFC. You can go through the log and add up the possible dead during a 24-hour period. The next step is to go to the admingcgem log and look for an entry labeled “Starting doSweepWsUnion” with a timestamp shortly after the MFC completed. This should show a “PD size” that is approximately the number reported by the MFC. A few lines down will be a couple entries showing a count of objects removed from the possibleDead. Just before the next “Starting doSweepWsUnion” will be a final (and generally lower) “possible dead size”. You can take the total of these final sizes to get an estimate of the useful work done by the MFC.
In one case, we saw hourly MFCs take about 10 minutes each and over the course of a day the repository garbage collection process found 1.5 million dead objects. If each object is about 120 bytes, this is less than 200 MB. In this case it was worthwhile to switch to a daily MFC and do it during off-peak time when it would not affect the users.
To implement this change required an edit to $GEMSTONE/seaside/bin/runSeasideGems[30] to comment out the lines to start the maintenance gem. Instead, you can create a new script modeled on $GEMSTONE/seaside/bin/startMaintenance[30] that does not have an endless loop for expiring sessions and doing an MFC but does the work once then exits. You can then set up a cron job to call your new script.
I realize that I’m not providing all the details of the new setup here. The goal is more to provide an exploration of alternatives.
[Update: Download from ftp://ftp.gemstone.com/pub/GemStone64/3.0.0/]
VMware vFabric GemStone/S 3.0.0 is now live on vmware.com.
Highlights of what’s new in the GemStone/S 64 Bit 3.0.0 release:
- Significant performance improvements and scalability features, including a redesigned Smalltalk virtual machine with just-in-time (JIT) compilation to native code, and parallelized, load-balancing garbage collection and repository scan operations.
- Internal redesign for compatibility and portability, including new features supporting the Seaside web framework, and improved support for the ANSI Smalltalk standard and for portability to other Smalltalk dialects.
- Various other features, such as Foreign Function Interface (FFI), simplifying the interface to third-party run-time libraries.
GA date: 07/14/2011
Download portal: http://downloads.vmware.com/d/info/datacenter_downloads/vmware_gemstone_s/3.0
Release Notes & documentation for GemStone/S: http://community.gemstone.com/display/GSS64/GemStoneS+64+Documentation
No-cost License Key: http://seaside.gemstone.com/etc/
Programming Gems (on GemStone) 