This article describes how to install and use sar (sysstat) a system performance tools for Linux. It comes with plenty of sar examples and usage. Sar is part of the sysstat package. According to the package description it includes the following system performance tools:

• sar: collects and reports system activity information;
• iostat: reports CPU utilization and disk I/O statistics;
• mpstat: reports global and per-processor statistics;
• pidstat: reports statistics for Linux tasks (processes);
• sadf: displays data collected by sar in various formats.

Using sar you can monitor performance of various Linux subsystems (CPU, Memory, I/O..) in real time. You can also collect all performance data on an on-going basis, store them, and do historical analysis to identify bottlenecks. I do not take credits for this post, I have just collected pieces of info from different websites that I found useful and put them together. I thank Ramesh Natarajan from TheGeekStuff.com for publishing his original post back in 2011. I couldn’t have done any better. Credits added as due.

What SAR can do?

In this article I will demonstrate how to install and configure sysstat package (which contains sar utility) and explains how to monitor the following Linux performance statistics using sar.

• Collective CPU usage
• Individual CPU statistics
• Memory used and available
• Swap space used and available
• Overall I/O activities of the system
• Individual device I/O activities
• Context switch statistics
• Run queue and load average data
• Network statistics
• Report sar data from a specific time

The statistics reported by sar deal with I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities and TTY statistics, among others. Both UP and SMP machines are fully supported.

sysstat – SAR examples and usage

This first part describes SAR examples and usage along with installation.

Step 1. Install sysstat Package

sysstat is usually not installed in Linux system by default. You need to install it. You also need to make sure you have the correct repositories added. As I personally use Kali Linux, I’ve added Kali Linux official repositories. Follow your distro documentation to add yours. But it is pretty straight forward.

To install sysstat package:

sudo apt-get install sysstat
(or)
yum install sysstat
(or)
rpm -ivh sysstat-10.0.0-1.i586.rpm

Step 2. Enable stat collection

In new systems once you’ve installed sysstat, it enables statistics collections by default. In case it is not you can enable it by editing 1 word in the following file:

vi /etc/default/sysstat

Change ENABLED=”false” to ENABLED=”true”

#
# Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat
# and /etc/cron.daily/sysstat files
#

# Should sadc collect system activity informations? Valid values
# are "true" and "false". Please do not put other values, they
# will be overwritten by debconf!
ENABLED="true"

Save the file. Restart sysstat.

service sysstat restart
(or)
/etc/init.d/sysstat restart

Once it’s restarted, it will start collecting data.

Step 3. Changing collection intervals

Default sysstat configuration collects data every 10 minutes. You can change it easily. If you got a fast system or you need more intensive monitoring of your system (i.e. a web server with enough RAM and Processor), then you might want to reduce the collection interval from 10 minutes to 2 minutes. Alternatively, you can change it to 15 minutes if you got a slow system. Either way, it’s easy to do and here are the steps:

vi /etc/cron.d/sysstat

# The first element of the path is a directory where the debian-sa1
# script is located
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin

# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2

Change

5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
To

*/2 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
Save the file and restart sysstat.

service sysstat restart
(or)
/etc/init.d/sysstat restart

Step 4. Check sysstat version

Once installed, verify the sar version using the following command:

root@kali:~# sar -V
sysstat version 10.0.5
(C) Sebastien Godard (sysstat <at> orange.fr)

Step 5: Practical sar examples and usage

There are two ways to use sar.

1. sar followed by an option (without specifying a saXX data file). This will look for the current day’s saXX data file and report the performance data that was recorded until that point for the current day.
2. sar followed by an option, and additionally specifying a saXX data file using -f option. This will report the performance data for that particular day. i.e XX is the day of the month.

In all the examples below, we are going to explain how to view certain performance data for the current day. To look for a specific day, add “-f /var/log/sysstat/saXX” at the end of the sar command.

All the sar command will have the following as the 1st line in its output.

root@kali:~# sar -u
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

21:42:01        all      8.20      0.00      1.21      0.31      0.00     90.28
Average:        all      7.76      0.02      1.97      0.43      0.00     89.82

21:42:26          LINUX RESTART

21:44:01        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      6.70      0.00      0.63      0.28      0.00     92.38

root@kali:~#

• Linux 3.14-kali1-amd64 – Linux kernel version of the system.
• (kali) – The hostname where the sar data was collected.
• 17/06/14 – The date when the sar data was collected.
• _x86_64_ – The system architecture
• (4 CPU) – Number of CPUs available on this system. On multi core systems, this indicates the total number of cores.

As a last minute check, let’s check some historical Sysstat files.

root@kali:~# ls /var/log/sysstat/
sa03  sa04  sa05  sa17  sa28

root@kali:~# sar -u -f /var/log/sysstat/sa03
Linux 3.14-kali1-amd64 (kali)     03/06/14     _x86_64_    (4 CPU)

18:13:32          LINUX RESTART

18:14:01        CPU     %user     %nice   %system   %iowait    %steal     %idle
18:16:01        all      4.20      0.00      0.82      0.53      0.00     94.45

Average:        all      1.43      0.00      0.36      0.30      0.00     97.90
root@kali:~#

In this way, you can go back since the date you’ve enabled sysstat / sar to check historical data.

1. CPU Usage of ALL CPUs (sar -u)

This gives the cumulative real-time CPU usage of all CPUs. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on the last field “%idle” to see the cpu load.

root@kali:~# sar -u 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

21:57:24        CPU     %user     %nice   %system   %iowait    %steal     %idle
21:57:25        all      1.01      0.00      0.51      0.51      0.00     97.98
21:57:26        all      1.03      0.00      0.51      0.00      0.00     98.46
21:57:27        all      1.26      0.00      0.25      0.50      0.00     97.98
Average:        all      1.10      0.00      0.42      0.34      0.00     98.14
root@kali:~# 

Following are few sar examples:

• sar -u Displays CPU usage for the current day that was collected until that point.
• sar -u 1 3 Displays real time CPU usage every 1 second for 3 times.
• sar -u ALL Same as “sar -u” but displays additional fields.
• sar -u ALL 1 3 Same as “sar -u 1 3″ but displays additional fields.
• sar -u -f /var/log/sysstat/sa10 Displays CPU usage for the 10day of the month from the sa10 file.

2. CPU Usage of Individual CPU or Core (sar -P)

If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.

“-P ALL” indicates that it should displays statistics for ALL the individual Cores.

In the following sar examples under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.

root@kali:~# sar -P ALL 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:05:24        CPU     %user     %nice   %system   %iowait    %steal     %idle
22:05:25        all      1.25      0.00      0.50      0.25      0.00     97.99
22:05:25          0      0.00      0.00      0.00      1.00      0.00     99.00
22:05:25          1      3.00      0.00      1.00      0.00      0.00     96.00
22:05:25          2      1.02      0.00      0.00      0.00      0.00     98.98
22:05:25          3      0.00      0.00      0.99      0.00      0.00     99.01

“-P 1″ indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).

root@kali:~# sar -P 2 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:07:40        CPU     %user     %nice   %system   %iowait    %steal     %idle
22:07:41          2      1.00      0.00      0.00      0.00      0.00     99.00
Average:          2      1.00      0.00      0.00      0.00      0.00     99.00
root@kali:~#

Following are few sar examples:

• sar -P ALL Displays CPU usage broken down by all cores for the current day.
• sar -P ALL 1 3 Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).
• sar -P 1 Displays CPU usage for core number 1 for the current day.
• sar -P 1 1 3 Displays real time CPU usage for core number 1, every 1 second for 3 times.
• sar -P ALL -f /var/log/sysstat/sa10 Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.

3. Memory Free and Used (sar -r)

This reports the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

root@kali:~# sar -r 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:08:54    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact
22:08:55      4644240   3550716     43.33    153180   2204384   2612376     27.44   1756372   1438360
22:08:56      4644240   3550716     43.33    153180   2204384   2612376     27.44   1756468   1438360
22:08:57      4643744   3551212     43.33    153188   2204384   2612376     27.44   1757084   1438360
Average:      4644075   3550881     43.33    153183   2204384   2612376     27.44   1756641   1438360
root@kali:~#

 

Following are few sar examples:

• sar -r
• sar -r 1 3
• sar -r -f /var/log/sysstat/sa10

4. Swap Space Used (sar -S)

This reports the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

root@kali:~# sar -S 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:10:03    kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
22:10:04      1325052         0      0.00         0      0.00
22:10:05      1325052         0      0.00         0      0.00
22:10:06      1325052         0      0.00         0      0.00
Average:      1325052         0      0.00         0      0.00
root@kali:~#

Following are few sar examples:

• sar -S
• sar -S 1 3
• sar -S -f /var/log/sysstat/sa10

Notes:

• Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.
• Use “sar -H” to identify the hugepages (in KB) that are used and available.
• Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
• Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

5. Overall I/O Activities (sar -b)

This reports I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

• tps – Transactions per second (this includes both read and write)
• rtps – Read transactions per second
• wtps – Write transactions per second
• bread/s – Bytes read per second
• bwrtn/s – Bytes written per second

root@kali:~# sar -b 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:11:28          tps      rtps      wtps   bread/s   bwrtn/s
22:11:29         0.00      0.00      0.00      0.00      0.00
22:11:30         0.00      0.00      0.00      0.00      0.00
22:11:31         0.00      0.00      0.00      0.00      0.00
Average:         0.00      0.00      0.00      0.00      0.00
root@kali:~#

Following are few sar examples:

• sar -b
• sar -b 1 3
• sar -b -f /var/log/sysstat/sa10

Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

6. Individual Block Device I/O Activities (sar -d)

To identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition), use “sar -d”

root@kali:~# sar -d 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:12:08          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
22:12:09       dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
22:12:09      dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:       dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
root@kali:~#

In the above example “DEV” indicates the specific block device.

For example: “dev53-1″ means a block device with 53 as major number, and 1 as minor number.

The device name (DEV column) can display the actual device name (for example: sda, sda1, sdb1 etc.,), if you use the -p option (pretty print) as shown below.

root@kali:~# sar -p -d 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:13:33          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
22:13:34          sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
22:13:34          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
root@kali:~#

Following are few sar examples:

• sar -d
• sar -d 1 3
• sar -d -f /var/log/sysstat/sa10
• sar -p -d

7. Display context switch per second (sar -w)

This reports the total number of processes created per second, and total number of context switches per second. “1 3″ reports for every 1 seconds a total of 3 times.

root@kali:~# sar -w 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:14:45       proc/s   cswch/s
22:14:46        14.00   2817.00
22:14:47        14.00   3355.00
22:14:48        14.00   2450.00
Average:        14.00   2874.00
root@kali:~#

Following are few sar examples:

• sar -w
• sar -w 1 3
• sar -w -f /var/log/sysstat/sa10

8. Reports run queue and load average (sar -q)

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3″ reports for every 1 seconds a total of 3 times.

root@kali:~# sar -q 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:15:18      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
22:15:19            0       392      0.31      0.26      0.29         0
22:15:20            1       392      0.31      0.26      0.29         0
22:15:21            0       392      0.31      0.26      0.29         0
Average:            0       392      0.31      0.26      0.29         0
root@kali:~#

Note: The “blocked” column displays the number of tasks that are currently blocked and waiting for I/O operation to complete.

Following are few sar examples:

• sar -q
• sar -q 1 3
• sar -q -f /var/log/sysstat/sa10

9. Report network statistics (sar -n)

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.

sar -n KEYWORD

KEYWORD can be one of the following:

• DEV – Displays network devices vital statistics for eth0, eth1, etc.,
• EDEV – Display network device failure statistics
• NFS – Displays NFS client activities
• NFSD – Displays NFS server activities
• SOCK – Displays sockets in use for IPv4
• IP – Displays IPv4 network traffic
• EIP – Displays IPv4 network errors
• ICMP – Displays ICMPv4 network traffic
• EICMP – Displays ICMPv4 network errors
• TCP – Displays TCPv4 network traffic
• ETCP – Displays TCPv4 network errors
• UDP – Displays UDPv4 network traffic
• SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
• ALL – This displays all of the above information. The output will be very long.

root@kali:~# sar -n DEV 1 1 
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:16:15        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
22:16:16         eth0      4.00      4.00      0.92      0.26      0.00      0.00      0.00
22:16:16           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:         eth0      4.00      4.00      0.92      0.26      0.00      0.00      0.00
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
root@kali:~#

10. Report Sar Data Using Start Time (sar -s)

When you view historic sar data from the /var/log/sysstat/saXX file using “sar -f” option, it displays all the sar data for that specific day starting from 12:00 a.m for that day.

Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 10:00:00″, it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.

You can combine -s option with other sar option.

For example, to report the load average on 26th of this month starting from 11 p.m in the morning, combine the -q and -s option as shown below.

root@kali:~# sar -q -f /var/log/sysstat/sa17 -s 23:00:01  | head
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

23:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
23:02:01            2       381      0.19      0.36      0.37         0
23:04:01            0       381      0.19      0.28      0.33         0
23:06:01            0       382      0.07      0.20      0.30         0
23:08:01            1       382      0.17      0.18      0.28         0
23:10:01            1       373      0.06      0.14      0.25         0
23:12:01            0       372      0.10      0.13      0.23         0
23:14:01            0       372      0.36      0.19      0.24         0
root@kali:~#

There is no option to limit the end-time. You just have to get creative and use head command as shown below.

For example, starting from 2300hrs (11PM) if you want to see 7 entries, you have to pipe the above output to “head”

root@kali:~# sar -q -f /var/log/sysstat/sa17 -s 23:00:01  | head
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

23:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
23:02:01            2       381      0.19      0.36      0.37         0
23:04:01            0       381      0.19      0.28      0.33         0
23:06:01            0       382      0.07      0.20      0.30         0
23:08:01            1       382      0.17      0.18      0.28         0
23:10:01            1       373      0.06      0.14      0.25         0
23:12:01            0       372      0.10      0.13      0.23         0
23:14:01            0       372      0.36      0.19      0.24         0
root@kali:~#

Step 6. If you want to save the statistics for further analysis to a file use:

root@kali:~# sar -A > $(date +`hostname`-%d-%m-%y-%H%M.log)

Conclusion

sysstat or sar is very powerful and useful tool to have. It allows you to monitor your Linux or Solaris systems activity closely and using sar you should be able to identify bottlenecks. Now this is not the end of sar usage and examples. You can use kSar which is the graphical end of sar to view graphs in a proper Graphical interface. kSar allows you to dig down into historical data or even get data from a different workstation to view in a remote machine.

Thanks for reading and don’t forget to share. Stay tuned for kSar guide.

References

• sysstat (sar): http://sebastien.godard.pagesperso-orange.fr/]
• ksar: http://sourceforge.net/projects/ksar/
• TheGeekstuff: TheGeekStuff
• Debian: http://www.debian.org/
• Ubuntu: http://www.ubuntu.com/
• CentOS: http://www.centos.org/
• Gentoo: http://www.gentoo.org/