You are currently browsing the category archive for the ‘Uncategorized’ category.
Great news! The GemStone/S team is becoming an independent company after 3 years as part of VMware. The core engineering team, including me, along with Norm Green and Dan Ware, have formed GemTalk Systems.
For more information, visit our company web site at http://gemtalksystems.com.
At GemTalk Systems, we’re the people who built the product. We’re excited to become a company with a dedicated focus on Smalltalk, GemStone/S and allied initiatives. You’ll see an increase in innovation on the product, and customers will see a seamless transition in support.
GemTalk will be at the STIC conference (Phoenix in June). Plan to visit us there!
Our email addresses are changing, too. They are all in the form of email@example.com. But the old email addresses will continue to work for a few months.
If you have questions about this exciting transition, feel free to contact me.
A GemStone/S 64 Bit system needs to have various maintenance tasks performed regularly. One of these is managing log file, backups, and transaction logs. I recently put together a basic Linux VM that has GemStone/S 64 Bit installed, but little more (the last link here). One of the non-Smalltalk things that I sometimes struggle with is figuring out which file can be deleted. The following demonstrates some bash scripting that seems to address a few of my questions. It starts by deleting all but the most recent two log files for pcmon, pagemanager, and symbolgem. It then does some garbage collection activity and a full backup. It then deletes all but the most recent two backups. Finally, if there are two backups in the last two days, it deletes all transaction logs more than two days old. This does not do as much error checking as should be done in a serious production environment (such as try restoring the backup, apply transaction logs, etc., before deleting things), but it demonstrates some of the things that can be done in bash to do date-specific cleanup.
#!/bin/bash # cleanup log files cd /opt/gemstone/log (ls -t *pcmon.log | head -n 2; ls *pcmon.log) | \ sort | uniq -u | xargs rm (ls -t *pagemanager.log | head -n 2; ls *pagemanager.log) | \ sort | uniq -u | xargs rm (ls -t *symbolgem.log | head -n 2; ls *symbolgem.log) | \ sort | uniq -u | xargs rm # do backup topaz -l -T 50000 << EOF output push backup.out errorCount send SystemRepository markForCollection send SystemRepository reclaimAll send SystemRepository startNewLog run SystemRepository fullBackupCompressedTo: '/opt/gemstone/backups/backup-' , (DateTime now asSeconds // 60) printString. % logout errorCount exit EOF # cleanup backups cd /opt/gemstone/backups (ls -t backup* | head -n 2; ls backup*) | sort | uniq -u | xargs rm if [ "2" -le "`find . -mtime -2 -name 'backup*' | wc -l`" ]; then cd /opt/gemstone/data find . -mtime +2 -name 'tranlog*' | xargs rm fi
While the strict definition of GLASS (GemStone, Linux, Apache, Seaside, and Smalltalk) specifies a particular technology for each layer of the stack, other technologies can be used for the OS (e.g., Mac OS X), the web server (e.g., lighttpd or nginx), and the web framework (e.g., AidaWeb). I’ve been running GemStone/S 64 Bit on Mac OS X for some time and have had a local (laptop) configuration much like the tradition Linux setup with FastCGI routing requests to three gems that receive and process Seaside requests. This worked well at the beginning when Mac OS X included FastCGI (as part of its built-in support for Ruby on Rails). This has changed in the later releases; starting with 10.7 (Lion) and continuing with 10.8 (Mountain Lion), FastCGI is no longer included in the operating system. This has broken my setup.
GLASS uses FastCGI to route requests to long-running server processes (typically a Topaz process) that remains logged in to the GemStone database. The Topaz processes can run on a different host from the web server (Apache or whatever), using the ‘ExternalServer’ (discussed here and here). While FastCGI is sometimes thought to be out-of-date, it is more accurate to say that “There is not much development on FastCGI because it is a very stable protocol / application.” Apache now provides a fast cgi module (here), that is commonly described as a replacement to mod_fastcgi, but it doesn’t support the external server configuration used by GLASS.
Getting GLASS to work on Mac OS X 10.8 requires jumping through some hoops, but now that I’ve done it I’ll describe the steps I took (please feel free to suggest alternatives in the comments).
I used the App Store application to find and install Xcode (4.6), then installed the command line tools (from Xcode Preferences->Downloads or from Apple). Next, I installed the latest MacPorts. Then, from a Terminal I entered the following command:
sudo port install mod_fastcgi
This did a full Apache build, but also created the needed modules. Instead of running the MacPorts build of Apache, I copied the new module to the expected directory:
sudo cp /opt/local/apache2/modules/mod_fastcgi.so \
Then I was able to run my usual setup with FastCGI on Mac OS X 10.8!
While there are a variety of ways of interacting with GemStone/S 64 Bit, the most basic way is to use Topaz, the command-line GemStone C Interface (GCI) client that has been part of GemStone since the beginning. While GemStone/S 64 Bit as a server is not available for Microsoft Windows, Topaz is available as a Windows client application that can be used to connect to a Unix/Linux/Mac server. Using Topaz is very helpful in debugging connectivity problems since it removes the variables associated with other client applications (such as GBS, GemTools, Jade, etc.). That is, if you are having trouble connecting with GemTools, we are likely to ask you to try to connect using Topaz.
Fortunately, it is relatively easy to run Topaz on Windows. Open a web browser on http://seaside.gemstone.com/downloads/x86.Windows_NT/ and download GemBuilderC220.127.116.11-x86.Windows_NT.zip (this assumes that you are connecting to a 18.104.22.168 server). Unzip this into a convenient location on your Windows machine (e.g., C:\gemstone\). Open a command shell (Start, All Programs, Accessories, Command Prompt), navigate to the ‘bin’ directory created by unzipping the download, and start ‘topaz’. At this point you can enter the usual Topaz commands and try to login to your server. Following is a copy of the screen when I used Topaz on Windows to login to my database:
C:\GemStone\GemBuilderC22.214.171.124-x86.Windows_NT\bin>topaz _____________________________________________________________________________ | GemStone/S64 Object-Oriented Data Management System | | Copyright (C) VMware, Inc. 1986-2012 | | All rights reserved. | +-----------------------------------------------------------------------------+ | PROGRAM: topaz, Linear GemStone Interface (Remote Session) | | VERSION: 126.96.36.199, Fri Aug 24 10:24:31 2012 | | BUILD: gss64_3_1_0_x_branch-28937 | | BUILT FOR: Pentium/Windows_NT | | MODE: 32 bit | | RUNNING ON: 1-CPU jfoster-xpvm: Intel CPU, Windows NT 5.1 build 2600 Service| | Pack 3 | | PROCESS ID: 828 DATE: 11/28/2012 11:19:36 Pacific Standard Time | |_____________________________________________________________________________| neither topazini.tpz nor $HOME\topazini.tpz were found topaz> set user DataCurator pass swordfish topaz> set gemstone jfoster0 topaz> set gemnet !firstname.lastname@example.org#netldi:10460#task!gemnetobject topaz> login [Info]: libssl-188.8.131.52-32.dll: loaded [11/28/2012 11:20:47.435 Pacific Standard Time] gci login: currSession 1 rpc gem processId 30480 OOB keep-alive interval 0 successful login topaz 1> run 100 factorial printString % 93326215443944152681699238856266700490715968264381621468592963895217599993229915 608941463976156518286253697920827223758251185210916864000000000000000000000000 topaz 1> logout topaz> exit
The things I typed are in bold. The things you need to change are in italics. Specifically, you need to provide the name of your stone (perhaps it is ‘seaside’), the IP address (or hostname if you have an entry in your hosts file) and port number (or service name if you have an entry in your services file) for your NetLDI that will start your gem.
… are available here. My presentation was Smalltalk in the Cloud and included a demo of pushing an Aida application (using Pharo/Cog) to a public cloud.
Some time ago I described the steps I used to add Perl as a simple runtime and framework to Cloud Foundry. Since then many changes have taken place in Cloud Foundry and those steps no longer work.
I now have something that works as of 30 August 2012. To see the changes go to my github repository and view the changes to vcap, vcap-staging, stager, and cloud_controller. Note that the recent refactoring of Cloud Foundry means that instead of isolating changes to one repository, we now make changes to four repositories.
Of course, there may be more efficient ways of doing this, but this is what I’ve found to work for me today!
James Robertson and I discussed Cloud Foundry on his podcast.
As part of a project to simplify running GemStone/S 64 Bit on the Macintosh, I need to be able to modify the OS kernel settings to allow for shared memory. The changes need to be made as root and the typical way to do it is using sudo from a command shell. This requires using the command line interface as well as making various calculations, all of which is somewhat familiar to an experienced GemStone DBA but is not very nice as an initial experience with GemStone, particularly for Mac users who have come to expect a better user experience.
The currently-approved way to do privileged operations on the Mac is to create a “helper tool” (a small C application) that does the minimal task. A client application (running as a regular user) then can install the tool and (with appropriate authorizations) “bless” it so that launchd can start it as root when it is needed. Apple provides an example of how to install a helper tool (SMJobBless), but the sample helper does not actually do anything in the way of providing a service to the sample application. It seems that there are a number of requests on the web for a more complete sample, and so rather than moving straight to my application I started by enhancing the sample app. You can find the result on github.
Communication is done using Unix domain sockets. That is, rather than listening on a port, the server listens on a file. This allows a safer naming scheme that avoids overlap better than port numbers. It also limits requests to clients on the same machine. Socket programming is reasonable straightforward, but somewhat tedious. Because it is stream-based, one can’t be sure that an entire message has been received so there needs to be code that figures out how many bytes to expect and then waits till they come. Of course, you shouldn’t wait forever, so there are timing issues as well.
If you are building a Mac application that needs a helper tool, then take a look and let us know what is missing…
I have consolidated and updated my changes to Cloud Foundry to accomodate GemStone/S 64 Bit and have the changes on github. You can use that code rather than the recent series of posts that are now a bit outdated. I will be making additional updates as I add multi-machine capabilities to reflect some of the things learned in preparing for last month’s STIC presentation. Stay tuned!
One of the most common problem people have installing and starting GemStone/S is getting shared memory configured properly. This post will discuss shared memory in general and GemStone’s use of shared memory in particular.
One of the features of modern operating systems is that each operating system process runs in its own address space and (with help from the hardware) the process is not permitted to read or write outside its assigned area. This protection makes it much less likely that a misbehaving application will harm other applications or even crash your whole machine (as was common on personal computers during the era prior to Windows XP and Mac OS X). This protection comes with the overhead that if two processes want to share data they typically need to communicate through the operating system (typically using TCP/IP, as if they were on separate machines). Processes that want to share a large amount of data are allowed to request a chunk of memory from the operating system that can be shared. Because this is not common, most operating systems are configured to provide only a limited amount of memory for this use and without adjustments to those settings GemStone cannot run.
Each machine where a GemStone process is running will have a Shared Page Cache (SPC) that holds copies of 16 KB pages from the file system extents that make up the repository. The SPC is a chunk of shared memory allocated by a SPC “monitor” and to which other processes attach. When the stone starts it forks a process to be the SPC monitor and then waits for the SPC to become available. If the SPC does not appear within a set time then the stone reports an error and dies. This error will be similar to the following:
The stone was unable to start a cache page server on host '<stone's host>'. Reason: The cache monitor connect failed. Monitor process (9999) did not start.
The SPC monitor log (a file next to the stone log whose name ends with ‘pcmon.log’) will have the following:
| GemStone could not retrieve the IPC identifier associated with the memory | | key 9999999. shmget() error = errno=22,EINVAL, Invalid argument (programmer | error). | | | GemStone could not attach to the shared page cache. | [SpcMon trace]: ... cache creation failed ... [SpcMon trace]: ... if the errno is (EINVAL) it is likely because the cache size is less than the operating system imposed minimum or greater than the operating system maximum.
As suggested in the error message, the cache size is likely greater than the operating system maximum. If this is the first error you get after installing GemStone then it is likely that you have not configured the operating system for adequate shared memory. To see the existing limits, you can ask for a listing of the inter-process communication statistics:
This provides a list similar to the following (from my Ubuntu 10.04 desktop machine at work):
------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 4592442 max total shared memory (kbytes) = 4592440 min seg size (bytes) = 1
This machine is configured to make 4 GB of RAM available to be used as shared memory, and the maximum segment is also 4 GB. We can also query the current settings with the following command:
sysctl -a 2> /dev/null | grep kernel.shm
On my machine this reports the following:
kernel.shmmax = 4702660608 kernel.shmall = 1148110 kernel.shmmni = 4096
These numbers mean the same thing as shown above, but the first two lines use different values. While ipcs reports both the max segment size and the max overall shared memory as kbytes, sysctl reports the max segment size (kernel.shmmax) as bytes and the max overall shared memory (kernel.shmall) as 4 KB pages.
These settings are configured at boot time by reading /etc/sysctl.conf and applying defaults if no values are provided. To see the configured values, enter the following:
grep kernel.shm /etc/sysctl.conf
In my case I see the following lines:
kernel.shmmax = 4702660608 kernel.shmall = 1148110
If you edit /etc/sysctl.conf (as root using sudo) and reboot your machine then the new values should take effect. If you want to make changes without rebooting your machine, use something like the following:
sudo sysctl -w kernel.shmmax=4702660608
Once you have things properly configured then the SPC monitor should be able to allocation a SPC and the stone should start. To see the memory use, you can enter the following
On my machine the output includes the following:
------ Shared Memory Segments -------- key shmid owner perms bytes nattch status ... 0x00000000 1409046 jfoster 600 393216 2 dest 0xe50206db 8683543 jfoster 660 529203200 8 locked ...
There are a number of segments of 384 KB that are marked to be destroyed (dest), and one segment of about 500 MB. My configuration file includes the following:
SHR_PAGE_CACHE_SIZE_KB = 500000;
The pcmon.log file shows that the number of pages requested:
Number of pages 31250.
Since each page is 16 KB, this would request 512000000 bytes for pages. It turns out that there are other data structures that need to be shared, including the statistics captured by statmonitor, and the memory requested will be slightly larger than the configured amount. This means that you should set shmmax and shmall to be larger than your SHR_PAGE_CACHE_SIZE_KB setting (or ask for less than the configured maximum).
The GemStone/S 64 Bit Install Guide has a good discussion of how to calculate shared memory needs, but at present it contains an error in that the shmmax setting is incorrectly shown as being the same as the shmall setting. Remember that shmmax is in bytes and shmall is in pages (typically 4 KB).