Linux Understanding SHMMAX and SHMALL Settings
Linux and understanding SHMMAX and SHMALL kernel paramaters are the
key to your oracle databases performance. These two parameters are
pretty important paramaters and they affect how
Oracle creates and manages SGA’s.
Your operating system has to provide an application access to a certain
type of memory with certain types of controls. It’s the Unix kernel
that allows applications to access
shared memory (IPC), and how much memory, in multiple big chunks or small chunks etc…
Sometimes, portions of memory get used leaving memory segments out of order. So these chunks of memory, from
now
on called segments, can get used inefficiently leaving unused memory
segments. This can be caused by many reasons which I won’t get into
now. So when Orale wants to access some shared memory on startup, it
knows how much memory it needs for it’s SGA and has to figure out the
best and most efficient way to access the resource. If Oracle needs 8GB
or shared memory but can only find 4 seperate 2GB segments that are out
of order, it will then have no choice but to use this memory model.
This non-contiguous memory model isn’t the best. Oracle will use shared
memory in 3 different ways or memory models. The first memory model as
just described, the non-contiguous model where the memory segments are
scattered over the place. Another memory model being multiple memory
segments that are in fact contiguous, so this is better, less overhead
than the non-contiguous model. But the ultimate is one big segment that
is the size of the SGA. So in this case an 8 GB segment. And that is
where the tuning comes into play. How to tune the kernel and it’s
paramaters to meet your Oracle applications shared memory needs. As I
mentioned in the opening paragraph, its the SHMMAX that is the key
kernel paramater in Linux to tune the SGA shared memory requirement.
Shared memory and SHMMAX and SHMALL
Just to clarify a couple of definitions, I will go over some
terminology. The term shared memory is a term used to describe a type
of memory management in the Unix kernel. It is a memory region that can
shared between different processes. Oracle uses this shared memory
when using and implementing the SGA, which has to be visible to all
database sessions. Using a shared memory model, this avoids creating
duplicate copies of memory. So programs can eaily share data. These
regions of memory, because they are shared need to have controls on who
can access them and who can’t. A locking mechanism is required. This is
where
semaphores
come into play. Basically, flags that are either on or off. If the
memory is being used, the process turns on the flag and other processes
have to wait until the semaphore is freed or the flag is turned off. So
shared memory needs semaphores, they go hand in hand. I should
probably mention that an application can access shared memory
programatically through a set of POSIX C routines, common on most Unix
like operating systems including Linux. The IPC or shared memory API’s
have common IPC calls like shmget() and shmat() which allow the
programmer to get a shared memory segment, or attach to a segment.
There are many more. Now the kernel paramaters SHMMAX and SHMALL need
to be defined now. SHMMAX is really just the maximum size of a single
shared memory segment. It’s size is represented in bytes”. And SHMALL
is the sum of all shared memory segments on the whole system. But it is
measured in number of pages. Will leave it at that for now.
At times on very busy transactional databases, you can run out of
semaphores. This seriously limits the performance of a database. You can
look at your semaphores as follows:
# ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 65537 root 600 1
0x00000000 229378 oracle 660 300
0x00000000 265667 oracle 660 300
0x00000000 247100 oracle 660 360
Enough of a semaphore tangent. Linux being a type of Unix, there are
300 different ways of finding the value of a system paramaters. The
traditional way is calling ipcs to list the shared memory limits:
# ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096 <--- this is SHMMNI
max seg size (kbytes) = 67108864 <--- this is SHMMAX
max total shared memory (kbytes) = 17179869184<- this is SHMALL
min seg size (bytes) = 1
Or cat’ing the /proc/* data structures
# cat /proc/sys/kernel/shmall
4294967296
# cat /proc/sys/kernel/shmmax
68719476736
Now, how do you configure SHMALL for an Oracle database
Right off the top, since SHMALL is the sum of all the shared memory
segments on your system, you had better make it smaller than your total
system memory. If I remember correctly, back in the 90′s, when memory
was more expensive than the US space program, on HP-UX systems we used
to define shared memory SHMALL equivalent to much larger than physical
memory. Swap disks were busy then. I don’t know if it’s possible to do
this now, but it’s not recommended. I’ll have to test that some day for
fun. SHMALL should also be less than the sum of all the SGA’s configured
on the system. If you allow more shared memory than the physical
memory on the system, then your asking for a world of hurt. Obviously
your system has to be sized appropriately for the work to be done. If
you create a new database instance and start Oracle which will then want
to create a new SGA, shmget() -> shmat() etc… it will fail with the
much feared “out of memory” etc… This just means that SHMALL is doing
it’s job properly and preventing Oracle from taking what it isn’t
allowed to take.
A typical error message caused by there being insufficient memory when an Oracle instance is starting would be somehing like:
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
This means SHMALL is doing it’s job. As shared memory is a device, ie. /dev/shm, the error message above makes sense.
The logic behind finding the SHMALL value for your system
Configuring the kernel parameter SHMALL is mostly just basic math.
It’s a decision that involves finding the amount of memory you want to
give to Oracle, or how much you can give away. This memory I am talking
about can be talked about in terms of physical memory. Physical memory
shouldn’t be confused with virtual memory which has swap space
included. From the physical memory we take away the memory that the
operating system needs. Linux will use a lot of memory for disk caching
and other assorted data structures. So lets be moderately generous and
say we will give the Unix kernel 1GB, this is a good rule.
Working example
System memory 12GB
Linux Kernel 1GB
Memory for Oracle 11GB (System memory – Linux Kernel)
So we can assign SHMALL the value of 11GB. But… SHMALL is measured in
memory pages not bytes. We need to divide the 11GB by the systems page
size value. It’s almost always 4K which is the recommended size, but
to be safe, you run the following command:
# getconf PAGE_SIZE
4096
We have to convert the 11GB byte value into a page value because as I
mentioned above, SHMALL is measured in page size. I don’t make the
rules, sorry. I use bc, the unix calculator and I don’t need decimal
places.
# bc
bc 1.06.95
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
11*1024*1024*1024 <--- 11 x 1K x 1K x 1K gives me 11GB value in bytes
11811160064
11811160064/4096 <--- 11GB value in bytes divided by 4k (page size)
2883584 <--- page size value to assign to SHMALL
This is a tad conveluted, but
easy math. Lets make this SHMALL kernel paramater permanent by adding
it to the /etc/sysctl.conf file.
# vi /etc/sysctl.conf
kernel.shmall = 2883584
The logic behind finding the SHMMAX value for your system
If you read the release and install and configuration documentation for Oracle 11g, Oracle recommends you set
shmmax
to half of your physical memory or 4 GB less 1 byte. Choose whichever
is lower. The interesting 4 GB less 1byte memory setting is based on
the virtual memory addressing limitations of the 32 bit (x86)
processors. All 64 bit processors can address much larger memory
addresse space and is not limited like it’s 32 bit brethren. So much
larger shared memory segments can be permitted and created and used.
When Oracle first starts up, after having read it’s configuration
files it will know how much memory it needs for it’s assorted SGA’s.
The best fit for an Oracle SGA shared memory segment is when the SGA
fits into one shared memory segment. As I mentioned earlier, Oracle can
use non-contiguous shared memory segments, but its not Oracle’s chosen
memory model. Therefore by defenition, if the SGA is larger than the
value of SHMMAX, then Oracle has to use one of the two other remaining
shared memory models. Contiguous or non-contiguous memory segments.
I try to set the SHMMAX to the size of the largest SGA defined on
your database server. If your system has 6 Oracle instances configured,
2 require 2 GB SGA’s, 2 require 1.5GB SGA’s and 2 require 1 GB SGA’s,
you would set the SHMMAX to 2GB.
Edit your sysctl.conf and add the SHMMAX value to 2 GB in bytes
# vi /etc/sysctl.conf
kernel.shmmax = 2147483648
Or, as this is Unix and there are many ways of doing things, use sysctl -w argument=xyz which writes to the sysctl.conf file.
# sysctl -w kernel.shmmax=2147483648
Now reload the parameters by asking sysctl politely to to reread it’s configuration
# sysctl –p
Now that you have configured shmmax and shmall, and all the other
steps required to configure you system for Oracle, I will jump ahead
very fast here and install Oracle and create a test db instance.
# su - oracle
$ cd /tmp/2wherever/install/files/are
$ ./runInstaller
….
# service oracle start
Create a test databse
$ dbca
etc…..
# ipcs -a
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x6c6c6536 0 root 600 4096 0
0x2270894c 2359297 oracle 660 1512046592 30
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 0 root 600 1
0x00000000 65537 root 600 1
0x00000000 294914 apache 600 1
0x00000000 327683 apache 600 1
0x396b9144 983044 oracle 660 154
------ Message Queues --------
key msqid owner perms used-bytes messages
The last few steps have been exceedingly oversimplied, but the point
here is to show the shared memory use. You will also see that user
oracle has grabbed 154 semaphores. This isn’t a huge number, and in
this case it’s only a test db instance so no real traffic.
The kernel parameters can be tuned dynamically with the sysctl command. Meaning you can change a kernel parameter on the fly. But this won’t survive a reboot. To make this type of change permanently the best and possibly easiest way is to either edit the /etc/sysctl.conf file and then save it with the sysctl -p command below.