Samples of performance management procedures

The following describes typical approaches to performance management and describes a general procedure for troubleshooting performance problems. References are made to various parameters. For the descriptions and values of these parameters, see UNRESOLVED XREF-0.

Checking for excess swapping

We'll first look at paging activity, because the paging in and paging out is costly in terms of both disk and CPU overhead. Run the sar -pgrwud report. With this information, you can determine whether more memory is needed.

If the vflt/s value shown by sar -p is greater than 50, then look at sar -g. High values for vscan/s and pfree/s imply that the page-stealing daemon is working overtime to find free pages because of a memory shortage. A high attach rate and a high vflt rate indicate a memory shortage, also.

Other indicators of a memory shortage are the freemem value of sar -r and the swpot/s value of sar -w. A value of swpot/s that is consistently greater than ``0'' is also an indicator of a memory shortage.

If memory shortages occur frequently, increase memory in one of two ways: Uninstalling optional kernel utilities that are not needed by your applications frees the memory used by the utilities so it can be used by user applications. If this is not possible, you probably need to add extra memory.

Checking for disk slowdowns

If the value of %wio (from the sar -u report) is greater than 10 percent, or if the %busy for a disk drive (obtained by sar -d ) is greater than 50 percent, then the system has a disk slowdown. Some ways to alleviate a disk slowdown are:

• Organize the filesystem to minimize disk activity. If you have multiple disks, distribute the filesystems for a more balanced load.

• Consider adding more memory if the condition persists. Additional memory reduces swapping/paging traffic and allows pages to remain in memory (reducing the number of user-level reads and writes that need to go out to disk).

• Consider adding an additional disk and balancing the most active filesystems across disks.

• Increase the amount of buffer space (BUFHWM) if sar -b consistently indicates low hit rates
  (%rcache < 90, %wcache < 65) and high bread/s and bwrite/s. The %rcache and %wcache values are low temporarily when you start up an application or bring in new data. However, if these values are always low, your system may be a heavier than average user of filesystem data.

• Consider changing the logical block size of filesystems or even changing the filesystem type. (See ``Selecting a logical block size for a vxfs filesystem'' and UNRESOLVED XREF-0.)

Checking for modem interrupts

Run sar -y and review the report describing activity on terminal devices. If the number of modem interrupts per second, mdmin/s, is much greater than 0, your system may have faulty communications hardware.

Checking for table overflows

Run the sar -vt report to check for potential table overflows. This report lets you know if overflows have occurred in the process or inode tables. Overflows can occur if a memory shortage occurs or if the maximum table size is reached. If memory is the problem, the table limits have not been reached. If the table limits are reached, the system tunable should be increased. To avoid overflows in these tables, increase the Process Limit Parameter NPROC and the Filesystem Parameters S5NINODE, SFSNINODE, VXFSNINODE depending on the filesystems you are using. Use the System Tuner or the idtune command. See ``Performing basic system monitoring and tuning'' for information about the System Tuner, or idtune(ADM).

Analyzing disk resources

You can obtain a system activity report for I/O resources by executing

   sar -du

Analyzing memory resource usage

You can obtain a system activity report for memory resource usage by executing

   sar -pgrwud

Shifting the workload to off-hours

Identify jobs that can be shifted to off-hours.

• Examine the files in /var/spool/cron/crontabs to see if jobs are queued up for peak periods that might better be run at times when the system is idle.

• Use the ps(C) command to determine what processes are heavily loading the system.

• Use the process accounting command accton to determine what processes are heavily loading the system.

• Encourage users to run large, noninteractive commands during off-hours. You may also want to run such commands with a low priority by using the nice or batch shell commands.