One of the most common problem people have installing and starting GemStone/S is getting shared memory configured properly. This post will discuss shared memory in general and GemStone’s use of shared memory in particular.

One of the features of modern operating systems is that each operating system process runs in its own address space and (with help from the hardware) the process is not permitted to read or write outside its assigned area. This protection makes it much less likely that a misbehaving application will harm other applications or even crash your whole machine (as was common on personal computers during the era prior to Windows XP and Mac OS X). This protection comes with the overhead that if two processes want to share data they typically need to communicate through the operating system (typically using TCP/IP, as if they were on separate machines). Processes that want to share a large amount of data are allowed to request a chunk of memory from the operating system that can be shared. Because this is not common, most operating systems are configured to provide only a limited amount of memory for this use and without adjustments to those settings GemStone cannot run.

Each machine where a GemStone process is running will have a Shared Page Cache (SPC) that holds copies of 16 KB pages from the file system extents that make up the repository. The SPC is a chunk of shared memory allocated by a SPC “monitor” and to which other processes attach. When the stone starts it forks a process to be the SPC monitor and then waits for the SPC to become available. If the SPC does not appear within a set time then the stone reports an error and dies. This error will be similar to the following:

The stone was unable to start a cache page server on host '<stone's host>'. 
Reason: The cache monitor connect failed. 
Monitor process (9999) did not start.

The SPC monitor log (a file next to the stone log whose name ends with ‘pcmon.log’) will have the following:

| GemStone could not retrieve the IPC identifier associated with the memory |
| key 9999999. shmget() error = errno=22,EINVAL, Invalid argument (programmer
| error). |
| |
 GemStone could not attach to the shared page cache.
|
 [SpcMon trace]: ... cache creation failed ...
 [SpcMon trace]: ... if the errno is (EINVAL) it is likely because the
 cache size is less than the operating system imposed
 minimum or greater than the operating system maximum.

As suggested in the error message, the cache size is likely greater than the operating system maximum. If this is the first error you get after installing GemStone then it is likely that you have not configured the operating system for adequate shared memory. To see the existing limits, you can ask for a listing of the inter-process communication statistics:

ipcs -l

This provides a list similar to the following (from my Ubuntu 10.04 desktop machine at work):

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 4592442
max total shared memory (kbytes) = 4592440
min seg size (bytes) = 1

This machine is configured to make 4 GB of RAM available to be used as shared memory, and the maximum segment is also 4 GB. We can also query the current settings with the following command:

sysctl -a 2> /dev/null | grep kernel.shm

On my machine this reports the following:

kernel.shmmax = 4702660608
kernel.shmall = 1148110
kernel.shmmni = 4096

These numbers mean the same thing as shown above, but the first two lines use different values. While ipcs reports both the max segment size and the max overall shared memory as kbytes, sysctl reports the max segment size (kernel.shmmax) as bytes and the max overall shared memory (kernel.shmall) as 4 KB pages.

These settings are configured at boot time by reading /etc/sysctl.conf and applying defaults if no values are provided. To see the configured values, enter the following:

grep kernel.shm /etc/sysctl.conf

In my case I see the following lines:

kernel.shmmax = 4702660608
kernel.shmall = 1148110

If you edit /etc/sysctl.conf (as root using sudo) and reboot your machine then the new values should take effect. If you want to make changes without rebooting your machine, use something like the following:

sudo sysctl -w kernel.shmmax=4702660608

Once you have things properly configured then the SPC monitor should be able to allocation a SPC and the stone should start. To see the memory use, you can enter the following

ipcs

On my machine the output includes the following:

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status 
...
0x00000000 1409046 jfoster 600 393216 2 dest 
0xe50206db 8683543 jfoster 660 529203200 8 locked
...

There are a number of segments of 384 KB that are marked to be destroyed (dest), and one segment of about 500 MB. My configuration file includes the following:

SHR_PAGE_CACHE_SIZE_KB = 500000;

The pcmon.log file shows that the number of pages requested:

Number of pages 31250.

Since each page is 16 KB, this would request 512000000 bytes for pages. It turns out that there are other data structures that need to be shared, including the statistics captured by statmonitor, and the memory requested will be slightly larger than the configured amount. This means that you should set shmmax and shmall to be larger than your SHR_PAGE_CACHE_SIZE_KB setting (or ask for less than the configured maximum).

The GemStone/S 64 Bit Install Guide has a good discussion of how to calculate shared memory needs, but at present it contains an error in that the shmmax setting is incorrectly shown as being the same as the shmall setting. Remember that shmmax is in bytes and shmall is in pages (typically 4 KB).