Linux Memory Usage
If memory is a bottleneck on a system, this means that there is not enough memory for all of the active threads to keep their working sets resident. When the amount of free memory drops below lotsfree, the page daemon will begin trying to free up pages that are not being used. If the amount of free memory drops too low, the swapper will begin swapping LWPs and their associated processes.
When the page daemon and the swapper need to execute, this will decrease the amount of execution time the application receives. If the page daemon is placing pages that have been modified on the pagefree list, these pages will be written out to disk. This will cause a heavier load on the I/O system, up to the maxpgio limit.
The only utility that provides statistics on the hardware cache is vmstat -c. Cache misses are a potential bottleneck if they occur frequently enough. However, even when using vmstat -c, they are extremely difficult to detect.
Linux Memory Bottlenecks
There are many possible solutions to a memory bottleneck, depending on the cause. Your main goal is to make more efficient use of existing memory. If a system is consistently low on memory, there are a few potential remedies that can be applied.
If several large processes are in execution at the same time, these processes might be contending with each other for free memory. Running these at separate times may lower the demand. For example, running several applications sequentially might put less of a load on the system than running all of them concurrently.
The memcntl (or related library calls such as madvise) can be used to more effectively manage memory being used by an application. These routines can be used to notify the kernel that the application will be referencing a set of addresses sequentially or randomly. They can also be used to release memory that is no longer needed by the application back to the system.
The freepages.high, freepages.low, freepages.min, and page-cluster can be used to control when the kernel begins paging and swapping and at what rate. They can be used to modify the impact the page daemon and swapper have on system performance.
Applications that use shared libraries require less physical memory than applications that have been statically linked if sharing takes place.
Functions that call one another within an application should be placed as close together as possible. If these functions are located on the same page, then only one physical page of memory is needed to access both functions.
The setrlimit can be used to force applications to keep their stack and data segments within a specific size limit.
Adding more physical memory, or at least more and faster swap drives, is always a solution as well.