Delete clean cache to free up memory on your slow Linux server, VPS

Many Linux systems, servers and VPS’s run on low memory and over time you will see a degradation of speed and responsiveness. By default, Linux got excellent Memory Management and it knows when to clean up cache to free up enough Memory to execute the next command. However, saying that, more new features being added to Linux everyday and when you are playing games, running a Web Server, a Database (i.e. MySQL, PostgreSQL, MariaDB etc.), Network Storage (NAS / SAN ), you will see there’s a drop on speed and responsiveness. By deleting and cleaning pagecache, dentries and inodes related cache data from Memory, you can get free up some of your Memory (RAM) which then makes rest of system work bit faster. This article will show you 3 different options to delete and clean cache to free up memory on your slow Linux server and small VPS’s.

Using drop_caches to clean cache to free up memory

Starting Linux Kernel v2.6.16 ono we have a new mechanism to have the kernel drop the page cache and/or inode and dentry caches on command, which can help free up a lot of memory. However, before we do that, we need to discuss about clean and dirty caches.

Clean and dirty caches

When you run something on a Linux system or server, Kernel will try to cache the response for a period of time so that the next time the same request is made, instead of running a complex lookup in disk/process, it can just fetch that info directly from Memory/RAM and send back a response. This is one of the main reasons Linux systems are so much faster and responsive. Alternatively, Linux systems will store data/info in Memory first before writing it to disk. So it goes both ways. Ideally, the data in Disk/database should be the same in Memory. But when you’re playing games, or it’s a busy Linux server, there will be some delay before these two (disk-data and memory-data) can sync up.

Cleaning cache is easy. But in Linux we have what we call clean and dirty cache.Let’s have a quick look at the definition of these two types of caches and later I will discuss why they are important when you clean cache.

Dirty Cache

Dirty Cache refers to data which has not yet been committed to the database (or disk), and is currently held in computer memory. In short, the new/old data is available in Memory and it is different to what you have in database/disk.

Clean Cache

Clean cache refers to data which has been committed to database (or disk) and is currently held in computer memory. This is what we desire where everything is in sync.

 

Kernel options for dropping cache

Following explanation is taken from Linux Kernel docu:

https://www.kernel.org/doc/Documentation/sysctl/vm.txt

drop_caches

Writing to this will cause the kernel to drop clean caches, as well as reclaimable slab objects like dentries and inodes.  Once dropped, their memory becomes free.

To free pagecache:
    echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
    echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
    echo 3 > /proc/sys/vm/drop_caches

This is a non-destructive operation and will not free any dirty objects.

To increase the number of objects freed by this operation, the user may run `sync' prior to writing to /proc/sys/vm/drop_caches.  This will minimize the number of dirty objects on the system and create more candidates to be dropped.

This file is not a means to control the growth of the various kernel caches (inodes, dentries, pagecache, etc...)  These objects are automatically reclaimed by the kernel when memory is needed elsewhere on the system.

Use of this file can cause performance problems.  Since it discards cached objects, it may cost a significant amount of I/O and CPU to recreate the dropped objects, especially if they were under heavy use.  Because of this, use outside of a testing or debugging environment is not recommended.

Based on that explanation, we can say that drop_caches doesn’t touch dirty caches and only drops clean caches. That means after you’ve used the standard “echo 3 > /proc/sys/vm/drop_caches” command, it will only drop clean caches and all the dirty caches are left behind. This isn’t too bad but we want to free up maximum Memory / RAM for our server.

A nice explanation from stackexchange regarding the importance of syncing cache:

“sync” only makes dirty cache to clean cache. cache is still preserved. drop_caches doesn’t touch dirty caches and only drops clean caches. So to make all memory free, it is necessary to do sync first before drop_caches in case flushing daemons hasn’t written the changes to disk.

So in this article, I will use sync and drop_caches together to ensure we are syncing dirty caches (hence making them clean cache) and then dropping the whole lot:

Three drop_caches options to clean caches

Option 1: Free pagecache

This is suitable for webserver as they tend to store lots of pagecaches.

To free pagecache:

sync; echo 1 > /proc/sys/vm/drop_caches

Option 2: Free dentries and inodes info

To free dentries and inodes:

sync; echo 2 > /proc/sys/vm/drop_caches

Option 3: Free the whole lot – pagecache, dentries and inodes

This the maximum memory cleanup you can do on any Linux system without killing a process.

To free pagecache, dentries and inodes:

sync; echo 3 > /proc/sys/vm/drop_caches

When do you run drop_caches? Is it safe?

drop_caches is not a means to control the growth of the various kernel caches (inodes, dentries, pagecache, etc…)  These objects are automatically reclaimed by the kernel when memory is needed elsewhere on the system.

If you use drop_caches command on a busy server, it can cause performance problems. Since drop_caches discards cached objects, it may cost a significant amount of I/O and CPU to recreate the dropped objects, especially if they were under heavy use. Running this command is always a bad idea when your server is under heavy usage.

Depending on your location (geo-location) your server traffic will increase decrease on different time. For example: US users login and browse between 8am – 9pm, that mean you don’t want to run this command on your server if your main userbase is from US. At the sametime, when it’s 12am in US, it’s 12pm in China (I am guessing here). So if your userbase is from China, you might want to reconsider when you want to run this command on your Linux server or VPS to free up memory. Personally, I prefer to do it just before my server gets busier (I get US, India and Chinese users) that means there is little down-period, but I can still squeeze it somewhere.

WARNING / CAUTION

Because just after your run drop_caches, your server will get busy re-populating memory with inodes and dentries, original Kernel documentation recommends not to run this command outside of a testing or debugging environment. But what if you are a home user or your server is getting too busy and almost filling up it’s memory. You need to be able trade the benefits with the risk. Read below for my explanation.

Home users vs Servers

There are two types of Linux users:

1. Home Users – i.e. Desktop
2. Servers – i.e. WebServers

For Home users, duh! do whatever you want … you can just do much damage. You clean cache, and the system will be slowed for about 3 seconds and then re-populate memory with necessary files. You can safely run echo 3 to cleanup maximum memory:

sync; echo 3 > /proc/sys/vm/drop_caches

For Servers, (i.e. WebServer, A VPS with Low memory), it is much safer to use option 1, cleaning pagecaches only.

sync; echo 1 > /proc/sys/vm/drop_caches

Why? cause pagecaches fills up massive part of your servers memory and in case of a Apache webserver, 50% of memory is used for pagecaches.

Example scenarios

You need to run free -m and drop_caches to see the differences:

My Desktop

Run the following command to see how much memory is being committed:

free -m

Then run

sync; echo 3 > /proc/sys/vm/drop_caches

As you can see, my desktop was doing next to nothing, so there’s not much to cleanup.

My server

But because my server is always busy and it’s got a small memory pool (for a server), running drop_caches makes a massive difference:

super@myserver [~]# 
super@myserver [~]# free -m
             total       used       free     shared    buffers     cached
Mem:         12792      12151       1640          0       1177      11263
-/+ buffers/cache:       1711      12080
Swap:        11999       1131      11868
super@myserver [~]# 
super@myserver [~]# sync; echo 1 > /proc/sys/vm/drop_caches
super@myserver [~]# 
super@myserver [~]# free -m
             total       used       free     shared    buffers     cached
Mem:         12792       1831      11960          0          0       1132
-/+ buffers/cache:        697      12094
Swap:        11999       1131      11868
super@myserver [~]# 

How awesome is that? In an instant I recovered a massive pool of memory and my terminal to the server became lot more responsive.

Use cron to drop caches regularly

Its a good idea to schedule following in crontab to automatically flushing cache on regular interval.

Option 1: drop_caches using cron

Edit crontab file and add the following line in red at the end of it.

# crontab -e
0 4 * * * sync; echo 1 > /proc/sys/vm/drop_caches

The command sync; echo 1 > /proc/sys/vm/drop_caches will execute every minute of 4am every day will execute at 4:00am every day.

Option 2: drop_caches using sysctl

Alternative you can use also sysctl to change the drop_caches value:

# crontab -e
0 4 * * * sync; /sbin/sysctl vm.drop_caches=1

Option 3: drop_caches using a bash script

Create a shell script say clearcache.sh under root partition and enter following contents

#!/bin/sh
sync; echo 1 > /proc/sys/vm/drop_caches

Now, set the permission of script /root/clearcache.sh to 755

chmod 755 /root/clearcache.sh

Now edit crontab file

crontab -e

Enter following line to set cronjob for clearing cache everyday at 4am

0 4 * * * /root/clearcache.sh

Restart crond service

Restart crond service for each options (you only need to choose any of the options above)

/etc/init.d/crond restart

Performance gain and supporting graphs

There seems to be lot of posts and questions around whether this method should be used or not. Here’s my scenario/explanation:

I’ve struggled months with Linux Kernel panic issue: hung_task_timeout_secs and blocked for more than 120 seconds and tweaking vm.dirty_ratio and vm.dirty_backgroud_ratio allowed my VPS to function normally for few months. Then traffic increased again (and sudden bursts of traffic from twitter and reddit) and I couldn’t afford to increase RAM anymore. This kernel hack (be that may outside of recommended usage) allowed me to use cron and Monit to

1. Clear pagecache,
2. Free up enough memory to restart HTTPD automagically.

without having to do a cold restart. I am not claiming this a perfect solution but if used with ‘sync‘ and  if database optimization is enabled, it is yet to fail me.

Let me put it this way, it is also against recommendation to send a crafted fragmented packet to a server causing an outage, but we’ve all tested our server’s against such attacks.

I don’t think a big organization needs such a hack as for them it’s just easier to upgrade RAM, but for the little guys and VPS owners, this could mean the difference between 90% and 99.9% uptime. (I’ve got 99.7% uptime but that’s my fault, I keep testing in Prod server).

Here’s the RAW logs from today:

someuser@someserver [/logs]# grep 'Jan 14' memcleaner.log
Jan 14  00:10:52 127.0.0.1 PreScript Memory Usage = 85 %
Jan 14  00:10:52 127.0.0.1 PageCache Cleaned
Jan 14  03:02:55 127.0.0.1 PreScript Memory Usage = 97 %
Jan 14  03:02:55 127.0.0.1 PageCache Cleaned
Jan 14  05:42:54 127.0.0.1 PreScript Memory Usage = 86 %
Jan 14  05:42:54 127.0.0.1 PageCache Cleaned
Jan 14  09:02:50 127.0.0.1 PreScript Memory Usage = 87 %
Jan 14  09:02:50 127.0.0.1 PageCache Cleaned
...snipped...

and Monit logs

someuser@someserver [/logs]# egrep 'Jan 14.*trying to restart' monit.log
[Jan 14 03:03:17] info     : 'apache' trying to restart
...snipped...

and lastly here’s sar logs

someuser@someserver [/logs]# sar -r
00:00:01    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
00:10:02       610068   3214528     84.05    108604   2351084   1696672     28.89
03:00:01       747668   3076928     80.45    162776   2116960   1709460     29.11
03:10:03      1484612   2339984     61.18     60640   1592920   1723404     29.35
03:20:01      1326836   2497760     65.31     73848   1710880   1857400     31.63
05:30:01       785668   3038928     79.46    165852   2161624   1581312     26.93
05:40:01       770060   3054536     79.87    168364   2167852   2104060     35.83
05:50:01      1414824   2409772     63.01     70100   1618396   1830092     31.16
...snipped...

As you can see, PageCache was cleaned 4 times overnight, service was restarted only once when Memory reached 97%. I had a massive spike of traffic (200%) at 3am. Normally, most servers would get stuck at and become unresponsive when memory is 97% or +, refer to the Kernel panic link above. This whole time, sar was just cruising along thinking everything just fine (cause it’s set to every 10minutes to save resources) … We use 8GB+ RAM minimum at work for just reverse proxies, but a VPS is no match for that as it’s doing everything from DNS, DB, Webserver, syslog, sshd, cPanel, backup etc.

Lastly, here’s my graphs:

Graphs – Apache Accesses Graphs

Graphs – Disk Stats iops

Graphs – Disk stats Latency

Graphs – Disk stats Throughput

Graphs – Disk stats Utilization

Graphs – IOstat

Graphs – Inodes table usage

Graphs – Interrupts

Uptime

Last but not the least, my uptime during this whole time.

As you can see, there were two spikes (the first one caused by me using a script to test load, but it only affects Apache) but 3am one from Reddit and Twitter which caused Disk, IO, Inodes and Apache spikes. They were taken care without human intervention and I slept like a baby.

Saying all that, I am not claiming this is ideal, more like interim solution. If you can just put more resources in, doh, you don’t need anything like this.

Conclusion

I don’t recommend using a cron to run this command as bad things might happen. Let’s just say your server had some unusual user activity in an odd period from a different geo-location than you see (i.e. your post went viral in reddit) and cron is trying to run this command at the same time. The result: instant server crash and possibly a corrupted database. Use it with caution and only when you are logged in. But then again, a properly configured 1GB VPS can take 20-40 concurrent users and run this command at any time.

Home users, go crazy, your online games or low powered Computer can only work better with this command. Server and VPS users, use with caution, only use it when you are unable to increase Memory or you have a good timeframe with low user activities. Whether to use echo 1, echo 2 or echo 3, is upto you and your judgment.

NOTE: Found a better way of maintaining these using crontab and some additional scripts. I will post them later when I am sure they are suitable for every Linux blends.

Thanks for reading. Share and spread.