How to solve the problem when free is nearly 0 on Linux

Problem

One day, the upload feature of our server was failed because there were not enough memory on server.
So I logged into server and excute the free command:

$ free -g
             total       used       free     shared    buffers     cached
Mem:            15         15          0          0          2          0
-/+ buffers/cache:         12          2
Swap:           17          0         17

However, the result of the top command showed that the free memory was 0 without any process which costed so much memory:

$ ps aux | awk '{mem += $6} END {print mem/1024/1024}'
0.595089

After a long time research on Google, I found that there was another kind of cache, Slab, whichfree command count as used memory.

Slab

Slab Allocation is a kind of memory management mechanism which imported since Linux 2.2 which is used to cache data objects of the kernel. It can be understood as a kernel-specific object pool, which can improve system performance and reduce memory fragmentation. (After Linux 2.6.23, SLAB became the default allocator.)

Infomation about slab memory:

$ cat /proc/meminfo

And we can also use the slabtop command or /proc/slabinfo to watch the detail of Slab Memory.

Solve the Problem

Clear Reclaimable Cache of Slab

We can clear the reclaimable cache manualy by change the flag in /proc/sys/vm/drop_caches:

echo 2 > /proc/sys/vm/drop_caches

Change the System VM Config

Change the system vm config will have effect the timing of memory cleanup. However there is a risk of negative impact on system performance.

vm.vfs_cache_pressure: The larger the value is the faster that inode cache and dentry cache’s recycle will be.
vm.min_free_kbytes: By default, it is calculated based on the current memory at boot time. The larger the value, the sooner the system will start memory recovery.
vm.swappiness: The larger the swappiness is, the higher the swap rate of the system will be.