Checking Linux system performance with sar (2024)

Sar is a system utility that gives us many ways to examine performance on a Linux system. It provides details on all aspects of system performance including system load, CPU usage, memory use, paging, swapping, disk usage, device load, network activity, etc.

The name “sar” stands for “system activity report,” and it can display current performance, provide reports that are based on log files stored in your system’s /var/log/sa (or /var/log/sysstat) folder, or be set up to automatically produce daily reports. It’s part of sysstat – a collection of system performance monitoring tools.

To check if sar is available on your system, run a command like this:

$ which sar/usr/bin/sar

If not, you will need to install it with a command like “yum install sysstat” or “apt install sysstat”. You will then be able to run commands like these which collect performance details from your system:

$ sar -d 5 2Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)12:38:42 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util12:38:47 PM sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:47 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:47 PM sdb 0.40 0.00 6.40 0.00 16.00 0.01 8.50 0.3412:38:47 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:47 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:47 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util12:38:52 PM sda 2.40 0.00 388.80 0.00 162.00 0.01 2.00 0.3412:38:52 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:52 PM sdb 1.40 0.00 21.60 0.00 15.43 0.01 2.00 0.2812:38:52 PM sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0012:38:52 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Average: DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %utilAverage: sda 1.20 0.00 194.40 0.00 162.00 0.00 2.00 0.17Average: sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Average: sdb 0.90 0.00 14.00 0.00 15.56 0.01 3.44 0.31Average: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00Average: zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Note that this report shows device usage with activity displayed for /dev/sda and /dev/sdb. It includes two 5-second reports plus the averaging. If you delve into the sar man page, you’ll find explanations of each data column. In the above example, the fields included in the above example include:

DEV the devicetps Total number of transfers per secondrkB/s Number of kilobytes read from the device per secondwkB/s Number of kilobytes written to the device per seconddkB/s Number of kilobytes discarded for the device per secondareq-sz Average size (in kilobytes) of the I/O requestsaqu-sz Average queue length of the requestsawait Average time (in milliseconds) for I/O requests%util Percentage of elapsed time during which I/O requests were issued to the device

The next command pulls data on memory usage with three 5-second reports plus the averages.

$ sar -r 5 3Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)12:09:54 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty12:09:59 PM 3089584 5231848 471144 7.75 35944 2257060 2126524 17.50 645168 2021508 4812:10:04 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021552 4812:10:09 PM 3089584 5231848 471140 7.75 35944 2257060 2126524 17.50 645168 2021560 144Average: 3089584 5231848 471141 7.75 35944 2257060 2126524 17.50 645168 2021540 80

Using a wide terminal window will clearly make the details a lot easier to view.

Looking at current performance stats

The sar commands shown above are all using real time data. In other words, sar is collecting the data as directed and displaying it. No log files are involved. Older stats will not be available.

The default display is CPU. So, if you don’t specify an option, you’ll see something like this:

$ sar 10 2Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/16/2021 _x86_64_ (2 CPU)10:17:50 AM CPU %user %nice %system %iowait %steal %idle10:18:00 AM all 0.05 0.00 0.15 0.05 0.00 99.7510:18:10 AM all 0.05 0.00 0.20 0.00 0.00 99.75Average: all 0.05 0.00 0.18 0.03 0.00 99.75

To save the output from a command like this in a file, you could run the command like this:

$ sar -o sarfile 6 2

Keep in mind that the data file created (sarfile in this example) will not be a text file that you can display with a cat or more command. However, you can display it using a sar command that includes the file name as an argument:

$ sar -f sarfileLinux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)10:15:52 AM CPU %user %nice %system %iowait %steal %idle10:15:58 AM all 0.17 0.00 0.17 0.25 0.00 99.4210:16:04 AM all 0.08 0.00 0.33 0.00 0.00 99.58Average: all 0.12 0.00 0.25 0.12 0.00 99.50

Looking at previous performance stats

You can easily set sar up to run automatically on your system, store its collected stats in daily files, produce daily reports and remove the oldest of the files after a given number of days (default is 28). This allows you to review performance on recent dates and compare what you’re seeing with current stats. This can often help you answers questions like “What’s different today?” — often the first question you’re likely to ask yourself when problems arise.

Once you’ve collected data using sar, you can look at it with sar commands like this command that extracts data from the sa14 data file:

$ sar -f /var/log/sa/sa14 -bLinux 5.11.15-300.fc34.x86_64 (dragonfly) 06/14/2021 _x86_64_ (2 CPU)02:54:22 PM LINUX RESTART (2 CPU)03:00:19 PM tps rtps wtps dtps bread/s bwrtn/s bdscd/s03:10:19 PM 1.02 0.02 1.00 0.00 0.01 69.05 0.0003:20:19 PM 1.05 0.02 1.03 0.00 0.01 75.03 0.0003:30:19 PM 0.99 0.02 0.97 0.00 0.28 74.35 0.00

To do this, you need to first enable sar to run automatically and save performance stats in data files. The process for doing this is slightly different depending on which Linux distribution you are using.

Fedora

On Fedora, you would use commands like these to enable sar to collect daily stats:

# systemctl start sysstat.service# systemctl enable sysstat.service# systemctl status sysstat.service

After this, sar will store its data files in the /var/log/sa directory. A couple days after setting it up, you can expect to see something like this:

$ ls -ltotal 1096-rw-r--r--. 1 root root 103292 Jun 14 23:50 sa14 
The numbers in the file names (e.g., 14) represent the day of the month. The three "sa" files (sa14, sa15 and sa16) are the data files for each of the three days and the sar14 and sar15 files are the performance reports for the first two days. On Fedora, you can determine how many days' worth of data will be saved by using this command:
$ grep HISTORY /etc/sysconfig/sysstatHISTORY=28
Ubuntu
To enable sar on Ubuntu and related systems, edit the /etc/default/sysstat file and change ENABLED="false" to ENABLED="true".
ENABLED="true"
The log files will be stored in /var/log/sysstat.
Viewing daily reports
To view any of the daily sar reports (e.g., sar15), you can simply use commands like more or cat. These files are simple text files.
Once a day has ended (shortly after midnight), a full-day report will be available with a name like "sar15". While files like "sa14" are used to collect the raw data, files like "sar14" are text files that contain the daily reports. They contain quite a lot of performance details.
$ wc -l sar154701 sar15$ more sar15Linux 5.11.15-300.fc34.x86_64 (dragonfly) 2021-06-15 _x86_64_ (2 CPU)00:00:19 CPU %usr %nice %sys %iowait %steal %irq %soft %guest %gnice %idle00:10:19 all 0.07 0.00 0.12 0.05 0.00 0.01 0.01 0.00 0.00 99.7400:10:19 0 0.03 0.01 0.05 0.02 0.00 0.00 0.01 0.00 0.00 99.8800:10:19 1 0.10 0.00 0.20 0.08 0.00 0.01 0.01 0.00 0.00 99.6000:20:19 all 0.07 0.00 0.10 0.04 0.00 0.00 0.01 0.00 0.00 99.7700:20:19 0 0.01 0.00 0.02 0.02 0.00 0.00 0.00 0.00 0.00 99.9400:20:19 1 0.12 0.00 0.18 0.07 0.00 0.01 0.01 0.00 0.00 99.6000:30:19 all 0.06 0.00 0.10 0.06 0.00 0.01 0.01 0.00 0.00 99.76…
To extract data from one of the daily data files (not the report files), you would use a command like this using the -f option to specify the data file to be used:
$ sar -f sa15 | head -22Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)12:00:19 AM CPU %user %nice %system %iowait %steal %idle12:10:19 AM all 0.07 0.00 0.14 0.05 0.00 99.7412:20:19 AM all 0.07 0.00 0.11 0.04 0.00 99.7712:30:19 AM all 0.06 0.00 0.12 0.06 0.00 99.7612:40:19 AM all 0.06 0.00 0.12 0.05 0.00 99.7712:50:19 AM all 0.07 0.00 0.11 0.05 0.00 99.7701:00:19 AM all 0.07 0.02 0.12 0.05 0.00 99.7401:10:19 AM all 0.07 0.00 0.11 0.05 0.00 99.7701:20:19 AM all 0.30 0.00 0.12 0.04 0.00 99.5301:30:19 AM all 0.07 0.00 0.11 0.04 0.00 99.7701:40:19 AM all 0.07 0.00 0.11 0.03 0.00 99.7801:50:19 AM all 0.07 0.00 0.12 0.03 0.00 99.7802:00:19 AM all 0.06 0.00 0.12 0.03 0.00 99.7802:10:19 AM all 0.06 0.00 0.13 0.04 0.00 99.7702:20:19 AM all 0.06 0.00 0.12 0.04 0.00 99.7802:30:19 AM all 0.06 0.00 0.12 0.02 0.00 99.7902:40:19 AM all 0.07 0.00 0.12 0.03 0.00 99.7902:50:19 AM all 0.07 0.94 0.20 0.05 0.00 98.7603:00:19 AM all 0.18 0.00 0.12 0.03 0.00 99.6703:10:19 AM all 0.06 0.00 0.12 0.04 0.00 99.78
You can specify a start (-s) and end (-e) times in addition to the file name as in this next example.
$ sar -s 11:00 -e 12:00 -f sa15Linux 5.11.15-300.fc34.x86_64 (dragonfly) 06/15/2021 _x86_64_ (2 CPU)11:00:19 AM CPU %user %nice %system %iowait %steal %idle11:10:19 AM all 0.06 0.00 0.17 0.05 0.00 99.7311:20:19 AM all 0.05 0.00 0.17 0.05 0.00 99.7311:30:19 AM all 0.05 0.00 0.17 0.05 0.00 99.7311:40:19 AM all 0.18 0.00 0.17 0.05 0.00 99.6011:50:19 AM all 0.06 0.02 0.18 0.05 0.00 99.69Average: all 0.08 0.00 0.17 0.05 0.00 99.70
Options, options and options
You can use sar to display stats related to:

• Paging (sar -B)
• I/O and transfer rates (sar -b)
• Block devices (sar -d)
• Currently mounted file systems(sar -F)
• Huge page utilization(sar -H)
• Interrupts (sar -I)
• Power management (sar -m)
• Network statistics (sar -n)
• Run queue and load average (sar -q)
• Memory usage (sar -r)
• Swap utilization (sar -S)
• CPU utilization (sar -u)
• Status of inode, file and other kernel tables (sar -v)
• Swapping (sar -W)
• Context switches (sar -w)
• TTY device activity (sar -y)

In fact, the easiest way to get a listing of all of these options and what they represent is to ask sar for help. Notice that for some options, like networking, you have a choice of which variety of activity (e.g., TCP or sockets) you want to see.

$ sar –helpUsage: sar [ options ] [  [  ] ]Main options and reports (report name between square brackets):-BPaging statistics [A_PAGE]-bI/O and transfer rate statistics [A_IO]-dBlock devices statistics [A_DISK]-F [ MOUNT ]Filesystems statistics [A_FS]-HHugepages utilization statistics [A_HUGE]-I {  | SUM | ALL }Interrupts statistics [A_IRQ]-m {  [,...] | ALL }Power management statistics [A_PWR_...]Keywords are:CPUCPU instantaneous clock frequencyFANFans speedFREQCPU average clock frequencyINVoltage inputsTEMPDevices temperatureUSBUSB devices plugged into the system-n {  [,...] | ALL }Network statistics [A_NET_...]Keywords are:DEVNetwork interfacesEDEVNetwork interfaces (errors)NFSNFS clientNFSDNFS serverSOCKSockets(v4)IPIP traffic(v4)EIPIP traffic(v4) (errors)ICMPICMP traffic(v4)EICMPICMP traffic(v4) (errors)TCPTCP traffic(v4)ETCPTCP traffic(v4) (errors)UDPUDP traffic(v4)SOCK6Sockets(v6)IP6IP traffic(v6)EIP6IP traffic(v6) (errors)ICMP6ICMP traffic(v6)EICMP6ICMP traffic(v6) (errors)UDP6UDP traffic(v6)FCFibre channel HBAsSOFTSoftware-based network processing-q [  [,...] | PSI | ALL ]System load and pressure-stall statisticsKeywords are:LOADQueue length and load average statistics [A_QUEUE]CPUPressure-stall CPU statistics [A_PSI_CPU]IOPressure-stall I/O statistics [A_PSI_IO]MEMPressure-stall memory statistics [A_PSI_MEM]-r [ ALL ]Memory utilization statistics [A_MEMORY]-SSwap space utilization statistics [A_MEMORY]-u [ ALL ]CPU utilization statistics [A_CPU]-vKernel tables statistics [A_KTABLES]-WSwapping statistics [A_SWAP]-wTask creation and system switching statistics [A_PCSW]-yTTY devices statistics [A_SERIAL]

That’s a LOT of choices! You can, however, use the -A option to collect the data provided by all of the options included included in the string “bBdFHSvwWy”.

Wrap-Up

While sar is easy to install and use, it provides a lot of valuable details on how your system is performing and might help you drill down to the cause of a problem when one arises.

The only “problem” with sar is that it generates a LOT of data and you’ll have to become accustomed to using it and familiar with what routine performance on your system looks like before unusual performance or reasons for concern will jump out at you.