A Cookbook Of VAX/VMS Performance Slowdown - First Things To Look For COMPONENT: PERFORMANCE OP/SYS: VMS, Version 4.n LAST TECHNICAL REVIEW: 17-OCT-1988 SOURCE: Customer Support Center/Colorado Springs USA NOTE: This article is approximately 1250 lines long. It is recommended that you get a hardcopy before trying to read it. OVERVIEW: A VAX/VMS system may occasionally experience a performance slowdown. This article describes some of the first things to look for when running VMS Version 4.n. It is not a detailed performance cookbook since it only handles some of the more common problems. Any numbers presented here are only "rules of thumb" based on a VAX 11/780 processor and should be adjusted accordingly. REFERENCES: For a more detailed look at VMS performance in general, refer to the "Guide To VAX/VMS Performance Management", (April 1986). Also note that several products are available for looking at overall system performance (SPM and VPA) and at individual process performance (PCA). CAUTION: If a symptom or solution is not fully understood, do not apply it because it could compound existing problems. For this reason, WHAT is being done and WHY it is being done are important to understand before making any changes. Always plan recovery procedures in case a change does not produce the desired effect. COMMON PERFORMANCE TROUBLE-SHOOTING PROCEDURES ---------------------------------------------- PREPARATION: Before you begin, check to see that AUTOGEN has been run. AUTOGEN is designed to set initial SYSGEN parameters for your particular system. It is documented in the "VAX/VMS System Manager's Reference Manual", April 1986, Chapter 11. We recommend that you run the AUTOGEN command procedure before going into any performance tuning. This will put any site-specific SYSGEN items into the file SYS$SYSTEM:MODPARAMS.DAT. Although AUTOGEN is not a "performance tool", per say, it can solve many performance problems caused by manually changing SYSGEN parameters that are interrelated to other unchanged parameters. AUTOGEN is also recommended whenever you change your hardware configuration. For example, changes in the number of devices or adding more memory. 1) RAISE YOUR PRIORITY SO YOU CAN LOOK FOR THE CAUSE OF THE PROBLEM. At the DCL prompt, issue the command $SET PROCESS/PRIORITY=15 to raise your process priority. This should help improve the system's response to your commands unless the system is hung or a Real-Time process is taking all the CPU time. This command requires ALTPRI privilege and may take several minutes to execute depending on the exact nature of the slowdown. If you have Real-Time processes on your system, you may need to elevate your priority above them to get timely response. If this is the case, the Real-Time processes may be in a compute loop blocking other lower priority processes from executing. 2) CHECK FOR CONSOLE ERROR MESSAGES RELATING TO PAGE FILE SPACE. Typical errors to look for are: SYSTEM-W-PAGEFRAG, Page file badly fragmented, system continuing SYSTEM-W-PAGECRIT, Page file space critical, system trying to continue These errors indicate that your page file is too small to support the current activity on the system. You may have to make it larger or create a secondary page file. These errors will not always show up when you have a page file space problem. However, when they do show up, you know you have a page file that is too small for the current load. See Step 7 for further information on what you could do in this case. 3) CHECK FOR CONSOLE ERROR MESSAGES RELATING TO POOL PROBLEMS. SYSTEM-W-POOLEXPF, Pool expansion failure This error indicates that VMS needed to allocate a block of memory to extend either a lookaside list or non-paged dynamic memory and there was not enough free memory left to do this. See Step 9 for further information. 4) CHECK FOR PROCESSES IN SYSTEM RESOURCE WAIT STATES. Enter the command $ SHOW SYSTEM at the DCL prompt and look down the "STATE" column to see if any processes are in a Resource Wait state. Typical Resource Wait states to look for are: RWAST Waiting for an AST to complete. This can occur for many reasons but most occurrences are due to an AUTHORIZE quota not being large enough or the process is waiting for an I/O that is not completing for some reason. This is usually only a system problem if the process has exclusive access to a file or lock that other processes are trying to get access to. The other processes are blocked, waiting for this process to release the file or lock. This will usually hang only the processes on the system needing the locked resource. For example, a process having exclusive access to the VMSMAIL file could cause users logging in to "hang" when VMS gets around to checking how many mail messages they have outstanding but will not effect users already logged in or not invoking MAIL. NOTE: Processes can be in RWAST and have NO impact on the other processes in the system. Unless you know the process has a resource that is blocking other processes from executing, it is usually best to look for other performance related problems before considering a process found in RWAST. If the process is hung on an I/O to a device, you can occasionally free the device by turning it off and on again or hooking up a terminal to that port and entering XONs, carriage returns, etc. If this fails, you will have to reboot the system to eliminate the RWAST process. Take a CRASH of the hanging system to get a dump file for later examination using the DCL command $ ANALYZE/CRASH. SHOW DEVICE/FILE can hang a terminal when another process has an open file and is in RWAST. The command will try to send an AST to the RWAST process to see what files it has open. Since the process is already stuck in RWAST, the second AST will not get serviced, thus causing the SHOW DEVICE/FILE command to hang. A more detailed analysis of how to determine what the process in RWAST is waiting for can be found in another database article dealing with determining why a process might be in a RWAST state. RWMBX Mailbox full - this process cannot continue writing to a mailbox since the mailbox is full. Many times, using SHOW PROCESS/CONTINUOUS/ID= will show a process in RWMBX as continuing to work, the CPU time continues to increase, but it is writing to a mailbox faster than the receiving process can read the messages. This resource wait state is most often caused by an application error. Things to check would include the following: 1. Is the reader process reading the messages often enough or completely enough? 2. Is the writer program at a higher priority than the reader program? 3. When the mailbox is created, is the buffer size specified (SYS$CREMBX parameter BUFQUO) or is the default buffer size taken (SYSGEN parameter DEFMBXBUFQUO)? Increasing the size of this parameter may help alleviate the problem by allowing more messages to be placed into the mailbox, however, it may only delay the problem. 4. Does the writer program receive an error indication if the mail box is full so it could check for things like "is the reader process still out there?" If so, you may want to use the system service SYS$SETRWM (Set Resource Wait Mode) to get an error status back for certain conditions that would have normally placed it into RWAST. 5. Has the process writing to the mailbox exceeded it's BYTLM or BIOLM Authorization parameters? Another database article that gives a more detailed look at the RWMBX state deals with processes in RWMBX (Resource Wait Mailbox Full). RWMPB Waiting for the Modified Page Writer to write the Modified RWMPE Page list to the page file. These states usually indicate the page file is full. See Step 7 for further information on what to do in this case. This state occurs whenever the Modified Page List has grown above the SYSGEN parameter MPW_WAITLIMIT. It can also be caused by processes page faulting heavily and continually filling up the Modified Page List. If this is the case, increasing the process' Working Set, or setting MPW_WAITLIMIT higher than MPW_HILIMIT may help reduce the problem. Another database article deals with processes in RWMPB, RWMPE state, the PAGEFRAG, and PAGECRIT error messages. It provides a more detailed description of these states. RWNPG Nonpaged Dynamic Memory full and Paged Dynamic Memory RWPAG full can both indicate a problem with your pool areas not being large enough. See Step 9 for more information. There are two other database articles which give a more detailed look at these states. They deal with processes in RWNPG state (Resource wait for non-paged pool) and processes in RWPAG state (Resource wait for paged pool). RWSCS Distributed Lock Manager wait, waiting to coordinate lock activity on a cluster. For more information on locks refer to the lock discussion in Step 17. Processes making heavy use of locks or the file system will often be seen in this state. It does not necessarily mean a performance problem since this state is usually short lived and the process will continue working once it gets the needed lock. If this state is persistent, you may want to organize the cluster's locks such that the node most heavily using the lock(s) will be the master for the locks. Again, see Step 17 for more information. Another cause of this state persisting which can also cause cluster slowdowns and sometimes cluster hangs, is insufficient pool resources to allocate the message blocks for communicating to the other nodes in the cluster. Many times the least powerful node on the cluster, most often with the smallest amount of physical memory, can slow up the cluster when it begins running out of pool resources. See Step 9 for information on how to check your pool resources. RWSWP Waiting for space in the swap file to become available. See Step 8 for further information on what to do in this case. A database article which describes this process state deals with processes in RWSWP state (resource wait for swap file space). Less common wait states that can also indicate performance problems are: COLPG Collided Page Wait, several processes cause page faults on the same shared page at the same time. The first process that faults this page will be placed into PFW. The second and succeeding processes enter the COLPG state. If several processes are in COLPG, a shareable image, page file, or pageable section file may be on a disk that is oversaturated with I/O, thus taking a "long" time to resolve the initial page fault. See Step 13 and PFW below. FPG Free page wait, waiting for free memory to be placed on the free list. If processes stay in this state, you should investigate why the Free List has become depleted: 1. Has a page file become full (processes in RWMPx state), and the Modified List takes up the rest of free memory (SHOW MEMORY)? 2. Is a single process taking up most of memory (SHOW SYSTEM looking at the Ph.Mem column)? Is the swapper able to trim/swap processes to free up memory? 3. Does SHOW MEMORY, SHOW ERROR or ANAL/ERROR show any memory problems? MUTEX Indicates this process has requested exclusive access to a Mutual Exclusion Semaphore (MUTEX) that can not be granted because another process already has that MUTEX. If processes are hung in MUTEX, another process (possibly in RWAST) may have been granted access to the data area coordinated by the MUTEX and never released it. To see what data area the MUTEX is waiting for, use the following commands: $ ANAL/SYSTEM SDA> SHOW SUMMARY ! find process in MUTEX SDA> SET PROC/IND=! use "indx" number in 2nd column SDA> SHOW PROCESS ! look at "Event Flag Wait Mask" SDA> EXIT ! leave SDA $ SEARCH SYS$SYSTEM:SYS.MAP "" This should give you the data structure the process is waiting for, but will not tell you who has it or what the problem is. If processes remain in MUTEX, and several SHOW SYSTEM commands do not show them getting CPU time, you may have to reboot the system. Take a CRASH dump for later analysis using ANALYZE/CRASH. If there is a process in RWxxx, try to free that process up before taking the system down (it could be the process with the MUTEX). PFW Page Fault Wait means the process must wait for a page to be read in from disk. If you see many processes in this state, you may want to look into disk contention problems since this usually occurs when a disk containing a page file or installed image files is overloaded with I/O requests or is encountering I/O errors. See Step 13 for more information. It may be that the disk containing the page has temporarily gone offline as well, in which case it will show up in Mount Verification. 5) CHECK TO SEE IF A PROCESS IS USING A MAJORITY OF THE CPU TIME. At the DCL prompt, issue the command MONITOR PROCESS/TOPCPU and look for a process taking all or most of the available CPU time. A typical slowdown problem occurs when one or two compute intensive processes (often BATCH jobs) take CPU time away from interactive users. If the top CPU processes have a very low priority, they are not "hogging" the system, they are merely taking available CPU time for themselves because no higher priority process was ready to run. To check the process priority for those users taking most of processor, use the SHOW SYSTEM command and check the priority column. "Normal" interactive priorities will typically fluctuate between 4 and 9. If CPU time seems evenly distributed between the top CPU users, it is likely no single process is hogging the system unless all the top CPU users are compute intensive and together are hogging the system. It may be the CPU simply has too many computable processes to service. To check for this occurrence, see if the CPU is at, or very near, 100% busy using the MONITOR SYSTEM display. If this is the case, check to see how many processes are in COM (computable) state, displayed to the right of the CPU BUSY graph. If you consistently have more than 6 processes computable and they all get a fairly even distribution of the CPU, the current work load may be too much to expect a better response to without a shift in work load or a more powerful CPU. The more processes in your COMputable queue, the more likely you need a more powerful VAX processor or another VAX processor to offload some of the processing to. In this case, refer to the "Guide to VAX/VMS Performance Management", April 1986, Sections 3.5, 4.4, and 5.4. BATCH jobs running CPU intensive programs can easily hog the CPU if there are several of them or if they run at priorities equal to or greater than the interactive users. To limit the impact of batch jobs, initialize the batch queues with a lower JOB_LIMIT and/or BASE_PRIORITY. The JOB_LIMIT queue qualifier will specify the number of concurrent BATCH jobs that can execute. Keeping the JOB_LIMIT low during prime time, approximately 2 or 3, could improve interactive performance. The JOB_LIMIT could then be increased to 5 or 6 for the evening hours using the SET QUEUE command. Because VMS uses automatic priority boosting, setting batch jobs to a BASE_PRIORITY=3 may still interfere with your interactive users and a BASE_PRIORITY=2 may be appropriate if this is the case. If the top process shown by MONITOR PROCESS/TOPCPU is often the SWAPPER, the system may be experiencing a physical memory shortage and trying to reclaim memory. See Steps 6 and 10 for further information on this type of problem. If one of the top processes is OPCOM, check to see if you have security options activated. If you do, you may want to reconsider activating this option. There is a database article dealing with the DCL command SET AUDIT/ALARM/ENABLE=ALL causing multiple console messages that may assist you in checking for this. If you find you are compute bound and have a mixture of interactive I/O bound and compute bound processes, you may be able to improve "response time" for the interactive users by reducing the SYSGEN parameter QUANTUM. QUANTUM basically sets up the time slice for processes on the system. By making QUANTUM shorter, you reduce the amount of processor time a single compute bound process can take before VMS starts looking for another job to schedule. The smaller you make QUANTUM, the more VMS overhead is introduced looking for processes to schedule. The larger you make QUANTUM, the longer a compute bound process can tie up the processor before another process is scheduled. The default value for QUANTUM is 20, meaning 200 milli- second time slices or a new time slice every 1/5th of a second. If you have 6 compute bound processes of equal priority, the 6th process may not get any CPU time for over a second. QUANTUM is a dynamic SYSGEN parameter and can be experimented with on the live system without rebooting. If you see no noticeable effect when changing QUANTUM, it would probably be best to return it to it's original value. If you do see a beneficial effect in changing QUANTUM, you will probably want to make the change to the bootable SYSGEN parameters (the CURRENT SYSGEN database) and note the change in the file MODPARAMS.DAT. Remember, any noticeable changes you see when changing QUANTUM may be subjective since system activity changes. Therefore, it may or may not be benefiting your system in the long run. More powerful processors, such as the 8500 series and up, would be more likely be benefit from lowering QUANTUM. 6) CHECK FOR SWAPPING. If VMS starts swapping processes out to disk, those processes swapped out will have to get swapped back into memory in order to run. VMS Version 4.n does not like to swap processes out because there is high system overhead in doing so. The following commands may be used to check system swapping: $ SHOW SYSTEM To see if any processes show up with the comment. $ MONITOR SYSTEM To see if processes are in LEFO, HIBO, or $ MONITOR STATE COMO states. If your system normally does some swapping, it is not unusual to see processes in LEFO (Local Event Flag wait Outswapped - usually waiting for a terminal I/O) and HIBO (Hibernating Outswapped). But processes in COMO (Computable Outswapped), are computable and need to get read back into memory in order to execute. Bringing this process back into memory may cause another process to get swapped out to make room for it. $ MONITOR IO To check the INSWAP RATE. If it is near zero or averages less than one per second, you may not have a swapping problem even if several processes are in LEFO and HIBO. If your system is swapping and this swapping is degrading your system's performance, check to see if swapping is really necessary. If SHOW MEMORY consistently shows that you have plenty of free memory yet swapping is occurring, check the SHOW MEMORY display to see if you are using all your BALANCE SET SLOTS. BALANCE SET SLOTS are Process Header slots in memory. If all the Process Header slots are used, the SWAPPER will swap someone out to reuse the slot for another process. This is called BALANCE SET SWAPPING and is generally recommended to increase the SYSGEN parameter BALSETCNT to two less than MAXPROCESSCNT to help reduce this type of swapping. Further discussion of swapping can be found in the "Guide to VAX/VMS Performance Management", April 1986, Sections 2.2.2, 4.2.3, and 5.2.11-5.2.14. If swapping is occurring and lack of memory prevents any other way of operating, it may be beneficial to swap processes out with larger Working Sets. Normally, the swapper will trim ALL processes on the system to their WSQUOTA and often down to the SYSGEN parameter SWPOUTPGCNT before swapping them out to disk. If the swapped out processes become computable very soon thereafter, they may cause another process to get swapped out in order to get this process back into memory. Once they get back into memory, they may have to page fault heavily to get any work done. But, they may get their working sets trimmed again because swapping is going on in the system. This could cause them to continue paging heavily and eventually get swapped out again. Two SYSGEN parameters that may assist in controlling this type of behavior are LONGWAIT and SWPOUTPGCNT. By raising LONGWAIT from it's default of 30 seconds to 180 seconds (3 minutes) or 300 seconds (5 minutes), you control those processes better that get swapped out. LONGWAIT tells the swapper to look for processes that have had no activity (been in LEF or HIB for LONGWAIT seconds) as the first swap candidates. Raising LONGWAIT improves the probability of users getting swapped out who have walked away from their terminals. SWPOUTPGCNT tells the swapper how many pages to trim everyone's Working Set to before swapping a process out of memory. Raising SWPOUTPGCNT from 60 pages to either the WSDEFAULT or the average WSQUOTA could reduce the amount of swapper trimming that occurs and also allow processes to have larger Working Sets when they are swapped back into memory. Unfortunately, there are no guarantees as to the effectiveness of these strategies. They must be tried on a trial-and-error basis to find an "optimum" amount of swapping. THE "BEST" SOLUTION TO REDUCE SWAPPING IS TO ADD MEMORY. Other things you could do to help reduce swapping would be to INSTALL shareable images so only one copy will be in memory at any one time, allowing many people to share the same copy. You may also set the SYSGEN parameter ACP_XQP_RES to 1 to make sure you are sharing file system code. These alternatives are discussed in the "Guide to VAX/VMS Performance Management", April 1986, Sections 1.6, 2.2.3, and 4.2.4.3. Also note that programs that keep track of other processes using the $GETJPI system service can induce more swapping on an already swapping system. This is because the $GETJPI call to a swapped process can cause that process to become computable. This will cause it to be swapped into memory in order to respond to the $GETJPI request. If possible, on a swapping system, check to see if the process is outswapped before using the $GETJPI system service on it. Programs that perform automatic log off tend to exhibit this behavior of inducing more swapping on an already swapping system. 7) CHECK FOR PAGE FILE SPACE GETTING FULL. The current amount of page file space being used can be found using the SHOW MEMORY/FILES command. A rule of thumb is to have 50% of the page file free on a busy system. If the page file starts to fill up, the system will slow down. When it is full, VMS can hang. There are 3 methods of adding more page file space. If the system disk has sufficient room, you can use the command procedure @SYS$UPDATE:SWAPFILES and specify a larger size for the page file. The system will have to be rebooted to use the larger page file. This procedure is documented the in the "VAX/VMS System Manager's Reference Manual", April 1986, Section 11.11. If this command procedure returns an error message indicating the 'File Header is too full', you will have to do an IMAGE BACKUP and RESTORE to compress the files and free up enough contiguous space on the system disk. Occasionally, the command procedure will get you very close to what you want for page file space and that may be sufficient without having to compress the disk. NOTE: Do NOT delete the system PAGE FILE, SWAP FILE or DUMP FILE!! The fact VMS is using them, is not known to the file system. If you delete one of these files on a running system, new files may be written to the same locations on disk. As a result, the data file will become corrupted when VMS writes to one of these 'system' files. If VMS trys to read something from one of these files, for example, if it brings in some pages from what used to be the primary page file, the results are unpredictable. The second alternative is to set up a secondary page file. This is usually done on a disk other than the system disk to distribute the I/O load across multiple disk drives. This can be done on the system disk to allow the system to continue without having to reboot. If it is temporarily placed on the system disk, we recommend that the temporary file later be deleted and the primary page file increased via SWAPFILES to the needed size. To create and install a secondary page file use the SYSGEN utility as follows: $ RUN SYS$SYSTEM:SYSGEN SYSGEN> CREATE disk2:[000000]PAGEFILE2.SYS/SIZE=50000 ! This command creates a new page file, PAGEFILE2, with a ! size of 50000 blocks. The actual size used will vary from ! system to system, but try to size it so at least 50% of the ! total page file space will be free. SYSGEN> INSTALL disk2:[000000]PAGEFILE2.SYS/PAGEFILE ! This command installs the page file so it will be known ! to VMS and can start being used. This should be ! set up in the beginning of your SYS$MANAGER:SYSTARTUP.COM ! file so it will be automatically installed on each reboot ! after the disk has been mounted. An example of this is ! given in the "VAX/VMS System Manager's Reference ! Manual", April 1986, Section 2.3.10. More information on the ! SYSGEN CREATE and INSTALL commands can be found in the "VAX/VMS ! System Generation Utility Reference Manual", September 1984. SYSGEN> EXIT NOTE: If you have a secondary page file and need to increase it's size, you will have to reboot the system at least once. Because the VMS File System has the secondary file(s) opened, you will be unable to extend it's size while it is installed. Since you can not deinstall a secondary page or swap file, you will have to: a) Modify the SYSTARTUP.COM file to do a SYSGEN CREATE command to the new file size just before the secondary paging file is installed. b) Reboot the system which will first extend the size of the secondary paging file and then install it. c) Remove from SYSTARTUP.COM the SYSGEN CREATE command added to increase the size of the secondary paging file. The number of page files you can create is controlled by the SYSGEN parameter PAGFILCNT. A process will generally use the page file with the most free space at the time the process is created. The process will continue paging to/from that page file until the process is terminated. Therefore, creating a secondary page file on a running system may not reduce the usage of the primary page file until users can log out and log back in. It may be beneficial to keep PAGFILCNT 1 or 2 higher than the total number of page files you have on the system. This would allow you to install new secondary page files on a running system in case you eventually run out of page file space and do not have the luxury of rebooting the system with a larger page file immediately. NOTE: Certain things will ONLY page to the primary page file and not to the secondary page file. These things include pageable portions of VMS (PAGEDYN), RMS global buffers, and global page-file sections (scratch global sections not mapped to a file). If the primary page file is nearly full and the secondary (being much larger) is hardly used, your system may require a larger primary page file to handle this load. The third alternative, useful if you are limited on disk space, is to use the primary page file as the system dump file as well. This would enable you to remove the SYS$SYSTEM:SYSDUMP.DMP file to free up more space on your system disk. This is described in the "VAX/VMS System Manager's Reference Manual", April 1986, Section 3.4.3. One way to do this on a system very tight on system disk space is as follows: a) RENAME SYS$SYSTEM:SYSDUMP.DMP SYS$SYSTEM:SYSDUMP.OLD b) Reboot the system so SYSDUMP will no longer be mapped and DELETE SYS$SYSTEM:SYSDUMP.OLD to free up disk space. c) Make the current page file larger: $ RUN SYS$SYSTEM:SYSGEN SYSGEN> CREATE SYS$SYSTEM:PAGEFILE.SYS/SIZE=nnnnn SYSGEN> EXIT ! Expand the size of your primary page file with the ! SYSGEN CREATE command to at least your memory size plus ! 1004 blocks. If you encounter an error indicating that the ! File Header is too full you may need to compress the ! system disk with an Image BACKUP and RESTORE first. d) If you want to have crash dumps go to the page file, you can increase the SYSGEN parameter SAVEDUMP: $ RUN SYS$SYSTEM:SYSGEN SYSGEN> USE CURRENT SYSGEN> SET SAVEDUMP 1 SYSGEN> WRITE CURRENT SYSGEN> EXIT e) If you want to copy your crash dumps to another disk on each reboot, you can set up your SYS$MANAGER:SYSTARTUP.COM file to copy the dump from the page file before the page file becomes heavily used. You could also use the commands given in the "VAX/VMS System Manager's Reference Manual", April 1986, Section 2.3.6: $ MOUNT DQA1: USERDISK !disk must be mounted before copy $ ANAL/CRASH SYS$SYSTEM:PAGEFILE.SYS !go get last crash COPY DQA1:[CRASH]SYSDUMP.DMP;1 !copy to dump disk EXIT f) Reboot the system to use the new primary page/dump file. Another reason for processes entering RWMPB or RWMPE states is when a process fills the Modified Page List very quickly and the Modified Page List is not getting written out fast enough to keep the process out of RWMPx for very long. If the page files have plenty of room, this may be the case. If so, you may be able to minimize the RWMPx state by raising MPW_WAITLIMIT as described in the first "Temporary fix" listed below. Raising MPW_WAITLIMIT above MPW_HILIMIT gives you a "buffer area" that could give the SWAPPER's Modified Page Writer time to write out the Modified List while it continues to grow above MPW_HILIMIT. Using the DCL command MONITOR PROCESS/TOPFAULT may point out one particular process as doing most of the page faulting on the system. It may be that raising this process's WSEXTENT could reduce it's page fault rate significantly. Temporary Fix: If your page file has filled up and you have: (1) lots of free memory available (2) a requirement that the system remain up a while longer (3) can't currently set up a secondary page file because of lack of disk space or PAGEFILCNT is too small to allow another page file to be installed (4) believe that you only need a "little" more page file space to keep everyone going to completion THEN you could try increasing the allowable size of the Modified Page List before processes are placed in RWMPx state. On a running system, you could execute the following: $ SHOW MEMORY !See how much Free Memory you have left $ RUN SYS$SYSTEM:SYSGEN SYSGEN> SHOW MPW_WAITLIMIT !Get current waitlimit. SYSGEN> SET MPW_WAITLIMIT 1500 !Set it to 2 or 3 times current value, !but less than free memory available. SYSGEN> WRITE ACTIVE !Make it stick on live system. SYSGEN> EXIT This may allow your processes to complete normally, however, may only delay the problem of not enough page file space. It is meant only as a temporary fix until the page file can be made large enough to handle the demand. Another temporary fix is to use the SYSDUMP.DMP file as a temporary page file. This is recommended only for short term conditions and will overwrite any contents in the dump file: $ RUN SYS$SYSTEM:SYSGEN SYSGEN> INSTALL SYS$SYSTEM:SYSDUMP.SYS/PAGEFILE SYSGEN> EXIT Some users will still have to log out and log back in to start using the new secondary page file. NOTE: A page file can be used for both paging and swapping. This occurs when the swap file becomes full and tends to downgrade the overall performance of the system. To check if a page file is also used for swapping, use the SHOW MEMORY/FILE/FULL command. To eliminate the need for using the page file for swap slots make the swap file larger, per Step 8 below. 8) CHECK FOR SWAP FILE SPACE GETTING FULL. The current amount of swap file space is being used can be determined using the SHOW MEMORY/FILES command. A good rule of thumb is to have 25% of the swap file free on a busy system. If the swap file starts to fill up, the system will slow down with processes going into a RWSWP state. There are three approaches to adding more swap file space. The first approach is to use the command procedure @SYS$UPDATE:SWAPFILES as mentioned in Step 7 above. The second alternative is to set up a secondary swap file, also described in Step 7 above, but installing it with "/SWAPFILE" instead of "/PAGEFILE". The number of secondary swap files you can create is controlled by the SYSGEN parameter SWPFILCNT. Unlike the page file, a process can switch swaps file it is using as its Working Set size changes. In fact, if the swap file is full and there is enough contiguous space in the page file, the page file can be used for swap slots. This is generally not recommended as it can degrade system performance. Setting SWPFILCNT to 1 or 2 higher than the total number of swap files on your system will allow you to add more swap files later if you find it necessary. A third alternative is to reduce your users WSQUOTA Authorize parameters so less space is taken in the swapfile. Even if no swapping takes place on your system, space for each process will be allocated in the swapfile just in case VMS does have to swap. These "swap slots" are allocated up to the user's WSQUOTA Authorization parameter. Therefore, reducing WSQUOTA for all users should result in less space being used in the swapfile. If you have plenty of free memory, processes will still be allowed to grow to their WSEXTENT value even if their WSQUOTA is made lower. A quick workaround is to use the SYSDUMP.DMP file as a temporary swap file. This is recommended only for short term conditions and will overwrite any contents in the dump file: $ RUN SYS$SYSTEM:SYSGEN SYSGEN> INSTALL SYS$SYSTEM:SYSDUMP.SYS/SWAPFILE SYSGEN> EXIT 9) CHECK FOR POOL EXPANSION PROBLEMS. The DCL command SHOW MEMORY/POOL/FULL can be used to list the initial and current allocations for the non-paged pool areas. If you find that one or more of the lookaside lists has been extended, an adjustment is indicated for its initial allocation. For example, if the LRP list from a SHOW MEMORY/POOL/FULL command appeared like: Large Packet (LRP) Lookaside List Packets Bytes Pages Current Total Size -------> 158 103648 203 Initial Size (LRPCOUNT) -------> 75 49200 97 Maximum Size (LRPCOUNTV) 300 196800 385 Free Space 38 24928 Space in Use 120 78720 Packet Size/Upper Bound (LRPSIZE + 80) 656 Lower Bound on Allocation 480 The "Initial Size" was not enough and had to grow from 75 packets to 158 packets, the "Current Total Size". The related SYSGEN parameter LRPCOUNT should be changed in SYS$SYSTEM:MODPARAMS.DAT and the command procedure @SYS$UPDATE:AUTOGEN.COM executed. AUTOGEN is documented in the "System Manager's Reference Manual", Version 4.4, Chapter 11. The SRP, IRP, LRP and NPAGEDYN areas can be checked against their "Initial Size" values. PAGEDYN is checked against it's free space left and it's largest contiguous block. "Rules of thumb" for the Nonpaged Dynamic Memory (NPAGEDYN) and Paged Dynamic Memory (PAGEDYN) areas are that they should be at least 20-25% free. Setting PAGEDYN so that 40% would be free should not impact memory utilization and will make more space available should a busy system suddenly demand more Paged Dynamic Memory. If PAGEDYN's Largest Contiguous block is less than 20000 bytes, you may start to see INSUFFICIENT DYNAMIC MEMORY errors when users try to log into the system. If a specific pool area is getting depleted and you do not feel it should be, you may be able to check what types of data structures are taking up the space. You can enter: $ ANAL/SYSTEM SDA> SHOW POOL/SUMMARY !make sure you can see all the pool areas SDA> SET OUTPUT pool.lis !output data to a file SDA> SHOW POOL/SUMMARY !get information written to the file SDA> EXIT To help interpret the different pool data structures look at a database article dealing with what SDA's SHOW POOL/SUMMARY symbols mean. --------------------------------------------------------------------------- If you have reviewed the above steps and still have not found the source of your performance slowdown, it would be best to study the "Guide to VAX/VMS Performance Management", April 1986, before considering any system changes. It is also recommended that the system be monitored over an extended period of time to make sure the correct performance problem is unmasked and any fixes are not to correct side effects created by the root problem. The list that follows is not exhaustive and is provided to give you some direction. A good summary of things to look for with the MONITOR utility is described in Section 3.8 of the "Guide to Performance Management",(April 1986). --------------------------------------------------------------------------- 10) CHECK FOR A MEMORY SHORTAGE/EXCESSIVE SWAPPER TRIMMING. The VMS Version 4.n SWAPPER has the duty of managing physical memory on a system-wide basis. It performs this through swapping (see Step 6) and Working Set Trimming which is described in the "VAX/VMS Guide to Performance Management", April 1986, Section 2.2.2. When the Free Memory list falls below the SYSGEN parameter FREELIM, the swapper is awakened to obtain at least as many free pages as specified by FREEGOAL. If the DCL command SHOW MEMORY indicates there are no free BALANCE SET SLOTS, proceed to Step 6 since the SWAPPER may be trimming processes for BALANCE SET SWAPPING. However, if there are free BALANCE SET SLOTS available, check the size of the FREE LIST and the MODIFIED LIST. Many times when the SWAPPER is trimming Working Sets you will see the following: (1) The Modified Page List will grow quickly and go beyond MPW_HILIMIT. This causes processes to page fault heavily to "recover" those pages trimmed out of their Working Sets. (2) MONITOR PROCESS/TOPCPU showing the SWAPPER process (3) SHOW SYSTEM showing processes getting their Working Sets (the Ph.Mem column) cut back to or below WSQUOTA (sometimes to SWPOUTPGCNT) (4) MONITOR PAGE showing the FREE LIST oscillating between FREELIM and FREEGOAL (ie, memory shortage). For general swapping considerations, see Step 6. You may also check for any users taking unreasonably large amounts of memory. This can be determined from the SHOW SYSTEM display. Look at the Working Set sizes shown on the rightmost column, Ph.Mem. This should be done several times to get a realistic view of how memory is being used. If you find such processes, they may have an unreasonably large WSQUOTA parameter that is squeezing other processes Working Sets down or even causing them to get swapped out. If all users are getting fairly equal treatment, no one sticks out as taking a large amount of memory, check to see if your page cache is unnecessarily large. To do this use the SHOW MEMORY display, to calculate the total usable memory by subtracting the total number of pages in all of memory minus the pages allocated to VMS (at the bottom of the display). Next, to calculate the number of pages in the page cache add the two SYSGEN parameters FREEGOAL and MPW_THRESH. If the total page cache is more than 15% of the total usable memory, the page cache may be too large. (FREEGOAL + MPW_THRESH) ------------------------------------------------- < 0.15 (TOTAL_pages - pages_permanetly_allocated_to_VMS) Running the AUTOGEN command procedure will usually fix a problem such as this and is described in the "VAX/VMS System Manager's Reference Manual", April 1986, Chapter 11. If AUTOGEN has not been run on the system, it is highly recommended that it be executed since it can often "fix" other performance related parameters as well. The SHOW SYSTEM display's right-most column, Ph.Mem, is the current number of pages for each process's Working Set. If after running SHOW SYSTEM several times it appears that everyone is getting trimmed fairly evenly and the size of the page cache is reasonable, there may be more demand for memory than the system's current capacity. If processes are getting trimmed to the SYSGEN parameter SWPOUTPGCNT, increasing SWPOUTPGCNT could help reduce the amount of "second-level trimming" the swapper will perform but may induce some swapping. It may be sufficient to increase SWPOUTPGCNT to the average WSDEFAULT, or to a value between WSDEFAULT and WSQUOTA. Another alternative is to reduce WSQUOTA such that the SWAPPER will have to perform "first-level trimming" to the users WSQUOTA and not have to go to "second-level trimming" down to SWPOUTPGCNT. For a discussion on Swapper Trimming, see the "Guide to VAX/VMS Performance Management", April 1986, Section 2.2.2. THE BEST ALTERNATIVE FOR THIS KIND OF PROBLEM IS MORE PHYSICAL MEMORY. To assist in calculating a first guess as to the amount of memory you may need, add the following values: (1) From SHOW MEMORY get the amount of space taken up by VMS on the last line of the display. (2) Add the SYSGEN parameters FREEGOAL and MPW_HILIMIT. If FREEGOAL is less than 1000 you should use the value of 1000 in place of FREEGOAL. (3) Add approximately 1600 pages for VMS System process Working Sets such as OPCOM, JOB_CONTROL, etc. (4) Add the result of multiplying the number of processes to run on the system by their WSQUOTA value (e.g. how much memory would be taken up with the maximum expected processes all running at their WSQUOTA Authorize parameter). This result would be a first guess if all processes were in memory at their WSQUOTA value. NOTE: This type of guess represents an "ideal" situation and may be larger than actually needed. pages_permanetly_allocated_to_VMS + MAXIMUM OF (FREEGOAL, 1000) + MPW_HILIMIT + 1600 (for "typical" VMS process pages) + number_of_processes_on_system * average_WSQUOTA ----------------------------------------------------- Total_Amount_Of_Physical_Memory_Needed_First_Guess 11) CHECK FOR AUTOMATIC WORKING SET DECREMENTING. Another item that can degrade performance is Automatic Working Set Decrementing. Check the SYSGEN parameters PFRATL and WSDEC. If they are BOTH non-zero, you have Automatic Working Set Decrementing turned on. Automatic Working Set Decrementing will take pages away from processes that are Page Faulting less than PFRATL and often times will induce more Page Faulting. If several SHOW SYSTEM displays indicate Working Sets are oscillating ("Ph.Mem" keeps raising and then falling again for many processes), but there is always plenty of free memory on the free list (SHOW MEMORY), then your system could be hurting from Automatic Working Set Decrementing. Some layered products automatically set these parameters in the file SYS$SYSTEM:MODPARAMS.DAT. It is generally recommended that Automatic Working Set Decrementing be turned OFF unless the system is understood to benefit from it. If you have Automatic Working Set Decrementing turned ON because it was beneficial at one point, it may be useful to reevaluate it's benefit by testing the system for a period of time with it turned OFF. For most systems, having PFRATL turned on decreases system performance rather than increases it. To turn Automatic Working Set Decrementing OFF, the SYSGEN parameter PFRATL should be set to 0. This is a dynamic parameter and can be changed on a live system. If this seems to help the performance problem it, should also be changed for the next reboot (SYSGEN'S CURRENT database) and for the next run of AUTOGEN (MODPARAMS.DAT). Automatic Working Set Decrementing is documented in the "Guide to VAX/VMS Performance Management", April 1986, Sections 2.2.1, 4.2.1.7, 5.2.7 and 5.2.8. 12) CHECK FOR HEAVY PAGE FAULTING. The MONITOR PAGE command gives the system's overall Page Fault Rate. Check the SYSTEM FAULT RATE (third row from the bottom). It should average under 1 per second. This is documented in the "Guide to VAX/VMS Performance Management", April 1986. Section 3.6.2.2. If this value is high, VMS itself could be page faulting heavily to service the users. To reduce the SYSTEM FAULT RATE it is recommended that the SYSGEN parameter SYSMWCNT be raised in 10% increments until the average SYSTEM FAULT RATE is under 1. This parameter is not dynamic and requires the system be rebooted. If your System Fault Rate is very high, you can approximate an initial value for SYSMWCNT using the following formula and SYSGEN parameters: GBLPAGES SPTREQ PAGEDYN Initial_SYSMWCNT = ----------- + ------ + --------- 128 5 1024 On a VAX 11/780, the overall PAGE FAULT RATE ideally should average under 100 faults per second, with less than 10% of the page faults being "hard page faults" (page faults to disk). To calculate the "hard page fault rate", subtract the overall PAGE FAULT RATE from the sum of all the "soft(memory) page faults": FREE LIST FAULT RATE, MODIFIED LIST FAULT RATE, DEMAND ZERO FAULT RATE, and WRT IN PROGRESS FAULT RATE. For an explanation of these and for more information on page faulting, refer to the "Guide to VAX/VMS Performance Management", April 1986, Sections 3.6.1.1, 4.2.1-4.2.1.8. If page faults are high, check to see if only particular users are page faulting heavily using the command MONITOR PROCESS/TOPFAULT. It may be that increasing their Working Set sizes with AUTHORIZE can help reduce the overall page fault rate if these users can keep more of their needed pages in their Working Set. This may NOT be possible for programs with very large arrays that process the data from one end of the array to the other end (Finite Element Analysis, CAD, etc). This is because such programs can easily have arrays that exceed available physical memory. Such programs will usually continue to page fault heavily even when given larger Working Set sizes. But, in such cases, design of the array structures themselves could help alleviate some page faulting by indexing into the arrays such that most memory accesses will be to locations on the same pages or an adjacent page before moving farther into the arrays. One method of calculating the "ideal" Working Set size for for a particular image would be to run the program alone on a system with plenty of free memory and on an account with WSEXTENT set very high (up to your WSMAX). Run the program on one terminal and monitor it on another terminal using SHOW PROCESS/CONTIN/ID=. In the top left column watch the WORKING SET size and the VIRTUAL PAGE size as the program executes. The VIRTUAL PAGE size is the total number of virtual pages the program has. This value indicates how much memory the entire program would take if all of it had to fit into memory. The WORKING SET size tells how many pages are currently available to the program in memory without having to page fault. On many types of programs you will see the WORKING SET grow very quickly at first and then start to level off as it gets most of it's Working Set in memory. If the program does level off it's WORKING SET, meaning it is not Page Faulting enough for VMS to warrant giving it additional memory, this could be a good WSQUOTA value. If the WORKING SET does not level off but climbs to WSEXTENT and stays there, the program could need additional Working Set increases to reduce page faults. The experiment could be performed again with a larger WSEXTENT and WSMAX, but the program may require more memory than is feasible with your current physical memory size. If it is the case, you will have to live with high page faults when running this program unless you can add more physical memory to keep more of the process virtual address space in memory. 13) CHECK FOR HEAVY DISK I/O. MONITOR IO, MONITOR DISK and MONITOR DISK/ITEM=QUEUE are all useful when looking at a possible Disk I/O problem. MONITOR PROCESS/TOPDIO is useful when looking for the top user of "Direct I/O", which is made up of mostly Disk I/O but can also include DMA devices, Logical and Physical QIOs to other devices. Disk I/O performance is described in the "Guide to VAX/VMS Performance Management", April 1986, Sections 3.7, 4.3 and 5.3. If MONITOR DISK shows a disk with more than 25 I/O operations per second (on an RA81 disk drive, less for slower disk drives), that disk is considered heavily loaded and distributing files being accessed on that disk across several disk drives could be looked at as a way to improve overall performance. If MONITOR DISK shows a high I/O rate, MONITOR DISK/ITEM=QUEUE may show a high queue length (the average number of I/O requests outstanding at any one time during the MONITOR sample interval). A large queue length indicates that the device is not able to keep up with the demand that is being placed on it and I/O requests must wait before being serviced. MONITOR IO also gives an indication of how many new files you are opening every second (FILE OPEN RATE) and how fragmented your files are (WINDOW TURN RATE). The WINDOW TURN RATE is an indication of how many times VMS had to map to another portion of a file header in order to locate the file extents on disk. A high WINDOW TURN RATE can be accompanied by high Kernal Mode Time if VMS must "split" many I/O requests between the different extents of the file on disk. If you feel disk fragmentation could be a problem, you should consider compressing your files on disk with an IMAGE BACKUP and RESTORE operation. To see how individual files are fragmented you could use the DCL command DUMP/FILE/HEADER/BLOCK=(COUNT:0) and look at the "Map area" to see how many "Retrieval pointers" are set up. For each extent of the file on disk, there will be a new block count and Logical Block Number (LBN) in the "Map area". The more "Retrieval pointers" you have, the more fragmented your files. Refer to the "Guide to VAX/VMS Performance Management", April 1986, Section 3.7.4.2, for more information on fragmentation. Several general suggestions for VMS performance, including the disabling of high-water marking, RMS file extend parameters, and reducing I/O to the system disk, are given in the "Guide to VAX/VMS Performance Management", April 1986, Section 1.6. These suggestions are useful for disk related performance problems. There may be particular processes only that are consuming most of the Direct I/O requests on the system. To identify such processes use the MONITOR PROCESS/TOPDIO command. It may be that using RMS buffering could reduce the overall I/O rate for these processes by keeping more information in memory. RMS Global Buffering could be used for files that are accessed heavily by multiple processes in the system. If RMS Global Buffering is considered, you may also need to increase the following SYSGEN parameters: (1) Global Page File Quota (GBLPAGFIL) since the SET FILE/GLOBAL_BUFFER command creates a scratch global section that can take space from the primary page file (2) Global Pages (GBLPAGES) since global pages are needed to point to the section (3) Global Sections (GBLSECTIONS) since one section is needed for each file global buffering will be applied against. The MONITOR IO display also gives you an indication of how busy your system is doing Mailbox writes and Logical Name Translations. If the number of Logical Name Translations is consistently above 100 on an 11/780, you may be degrading performance by simply translating Logical Names. This can also be accompanied by high kernel mode time. One cause of this could be extensive use of long Logical "path" names, using the "search list" feature of DCL described in the "VAX/VMS DCL Concepts Manual", April 1986, Section 4.8. Another factor affecting Disk I/O is the effectiveness of the file system's caches. These caches are described briefly in the "Guide to Performance Management", April 1986, Sections 5.3.3, 4.3.2.2 and 3.7.4.2. If the caches are ineffective you will notice delays whenever you go to disk to get more information. If MONITOR FILE_SYSTEM_CACHE shows cache hits lower than 70% (with "reasonable" attempt rates) then the related ACP_ cache SYSGEN parameter should be increased in MODPARAMS.DAT and AUTOGEN run. These parameters will take space out of both Nonpaged Dynamic Memory and Paged Dynamic Memory. Therefore, we highly recommend running AUTOGEN so the appropriate pool area will also be increased. One cache area, the BITMAP CACHE HIT %, can typically run under 30% effective because of its very low "Attempt Rate". Therefore, you do NOT need to be concerned about the BITMAP CACHE HIT %, or about any cache area on your system that averages well under 1 attempt per second. File system caches are occasionally not allocated because there is insufficient PAGEDYN. To see how many caches are allocated, issue the command "SHOW DEVICE/FULL " and look in the lower right side of the display. The line entitled "Maximum buffers in FCP Cache" will give the number of buffers currently allocated. If this value is 14, there was not enough PAGEDYN to allocate the entire file system cache so the minimum values were taken. When this occurs, the display from SHOW MEMORY/POOL/FULL will frequently show lots of free bytes in PAGEDYN, but the problem is not enough free bytes for ALL the caches the system would need to allocate. If VMS can not get ALL the space for ALL the caches specified, it will take the minimum values for all of them. SPECIAL NOTE: With VMS Versions 4.5 and earlier, AUTOGEN did not correctly calculate increases in ACP_DINDXCACHE as effecting PAGEDYN. It is recommended that the following line be added to MODPARAMS.DAT to allocate sufficient space in PAGEDYN to handle the Directory Index Cache if you are increasing ACP_INDXCACHE: ADD_PAGEDYN = ( 512 * ACP_DINDXCACHE * 11 / 10 ) 14) CHECK FOR HEAVY BUFFERED I/O. MONITOR IO and MONITOR PROCESS/TOPBIO are useful when looking for "buffered" I/O activity (usually terminal I/O). More than 100 terminal I/O operations per second, or an Interrupt Stack time more than 20% (MONITOR MODE) is usually cause for concern. DZ11s, being character interrupt devices, are known to cause this kind of behavior. One way to easily reduce Interrupt Stack time, thus freeing up more time for user programs, is to reduce the baud rate on your DZ11 lines. Reducing the baud rate from 9600 to 4800 could reduce your Interrupt Stack time almost in half. Another method is to look at alternatives for your terminal I/O lines. For large formatted I/O operations (more than 60 characters per second), a DMF32, DMZ32, or LAT11 terminal server can handle up to 3 times as many I/O operations as a DZ11 for the same usage of CPU overhead. They can perform 5-10 times as many operations for Physical I/O as a DZ11 with the same CPU usage overhead. Increasing the SYSGEN parameter MAXBUF, the maximum bytes for a buffered I/O operation, can occasionally reduce Buffered I/O load by allowing larger I/O requests. Buffered I/O is described in the "Guide to VAX/VMS Performance Management", April 1986, Sections 3.5.2.2.1, 4.3.4, and 5.3.4. 15) CHECK FOR PROCESSOR MODES. See if your processor is spending excessive time in any one mode. Check in the "Guide to VAX/VMS Performance Management", April 1986, Sections 3.5.2.2, 3.5.2.2.1,, 4.3.4.1, 4.3.4.2, and 4.4.3. The MONITOR MODE command is useful for starting problem isolation if the CPU is spending excessive times in any of the modes. The CPU modes are briefly described below: INTERRUPT STACK The time the processor spends responding to interrupts. If it averages more than 15%, it could indicate possible high device interrupts, often from character interrupt devices such as DZ11s. If it is consistently very high, over 70%, it could indicate a faulty device continually interrupting the processor. Other things to look for with high interrupt stack time include SCS cluster traffic and DLOCK, Distributed Lock Management (see Step 17 for more information on Distributed Lock Management). KERNEL MODE The time the processor spends in system functions such as system services, handling page faults, doing file and DECnet processing, handling local lock management, and setting up for physical I/O to devices. Often times, especially on slower processors such as the 11/750, high kernel mode time can be attributed to file fragmentation as the QIO requests will have to be split into multiple requests. A kernel mode time over 25% is cause for investigation as to a possible problem area. The INTERRUPT STACK time and the KERNEL MODE time together should average under 40% of the total CPU time. Running BACKUP during prime time can greatly degrade overall system performance since BACKUP could take up to 50% of the CPU, mostly in kernel mode. EXECUTIVE MODE The time the processor spends in RMS activity, often directly related to file design. SUPERVISOR MODE The time the processor spends in DCL functions. This should not be significant unless you are heavily using DCL command procedures. If it is high (over 15%) you might consider rewriting some of your command procedures as programs instead. USER MODE Time spent running user programs. COMPATIBILILITY MODE Time spent emulating the PDP-11 RSX system. IDLE TIME Time spent in the NULL process, "unused" CPU time. 16) CHECK DECNET ACTIVITY. More than 100 packets/second (which can be determined from MONITOR DECNET; the sum of arriving, departing, and transit) is cause for concern as 30% of the CPU is then being used for communications. Network file copies using FAL over high speed links have a maximum transfer rate of 60Kbytes/sec and can use about 50% of the CPU time on a 11/780. For slow links (56 Kbit/sec and lower), use large buffers and error-free links whenever possible. For fast links (1 Mbit/sec and faster), the CPU speed rather than the link speed, is usually the bottleneck. Check for DECnet errors since bad lines resulting in many errors and retry attempts could quickly tie up the processor. 17) CLUSTER TUNING. One node in a cluster can effect the performance of the entire cluster, so each node must be INDIVIDUALLY tuned before trying to tune a cluster. For information on cluster load balancing, refer to the "Guide to Performance Management", April 1986, Sections 3.5.2.4. Clusters can be very sensitive to pool expansion problems so pool areas in any cluster should be monitored periodically as described in Step 9. Multiple systems sharing disks, increase the possibility of I/O bottlenecks to heavily used disks (see Step 13 for general rules of thumb for Disk I/O). Using an aggregate maximum I/O rate of around 20 I/Os per second to a disk on the cluster can help determine how many nodes can effectively use this disk. If the user/system files can be distributed across several disks, the I/O load to any single disk could be reduced. Also, bound volume sets can help distribute the I/O load across multiple disks. Bound Volume sets are discussed in the "VAX/VMS System Manager's Reference Manual", April 1986, Section 7.1.2, and in the "Guide to VAX/VMS Disk and Magnetic Tape Operations", April 1986, Section 3.4.2. Multiple nodes sharing the same system disk can reduce cluster performance if the system disk becomes saturated with I/O requests. Secondary page and swap files on other disks can help reduce the I/O load to a common system disk and distribute the load across several other disks in the cluster. Other suggestions for off-loading particular files on the system disk can be found in the "Guide to VAX/VMS Performance Management", April 1986, Section 1.6. VMS Version 4.6 allows the primary page and swap files to be placed on a disk other than the system disk. Lock management can play an important role in clustered systems, especially if you have a wide range of processor power from one node to another (ie, a cluster with a 11/750 an a 8800). Because both Lock Management and Lock Directory Lookup are distributed across all nodes in a cluster, the efficiency of the cluster could be tied to the slowest node. MONITOR DLOCK gives some idea of how many lock requests are coming/going from/to other nodes in the cluster. If the ENQs and DEQs total over 400 per second, an 11/780 could be spending over 60% of it's time just responding to lock requests. If MONITOR DLOCK shows over 100 in "Dir Functn Rate" the processors could be spending much of their time looking on other nodes to find out who has what locks. A more serious (but less likely) problem would occur if there are any "Deadlock Message Rate" values that are non-zero. This usually means that VMS detected a possible deadlock condition and had to take CPU resources on possibly ALL nodes in the cluster to get the problem resolved. Such a condition would most likely be caused from poor program design in the use of locks. It is desirable to let the more powerful nodes, usually those nodes that will have more users accessing more files, to "master" the most locks in the system. One way to help insure this is to boot the more powerful nodes first and have them mount the disk drives before the less powerful nodes. There are several locks associated with each mounted device, so if specific nodes will be heavy users of specific disks, then those disks should be mounted first on the node expected to get the most use out of then. File locks, used for sharing files across the cluster, should also be controlled by the more powerful processors. If it is possible, start applications or open the most used files on the more powerful nodes before the less powerful nodes start trying to access those files. Another function of the Distributed Lock Management system is Lock Directory activity. This is controlled by the SYSGEN parameter LOCKDIRWT and is best described in the "Guide to VAX/VMS Performance Management", April 1986, Section 3.5.2.4, p. 3-25. The Lock Directory activity establishes where to look for different locks in the cluster and can be monitored with MONITOR DLOCK looking at the "Dir Functn Rate" field. This can seriously impact Local Area VAXclusters (LAVC) as the Ethernet speeds are significantly slower than the "big" cluster's CI speed. By default, VMS Version 4.5A will establish LOCKDIRWT to 10 for the boot node and 1 for all the satellite nodes. This is fine if the satellite nodes do not have local disks. But if the satellite nodes do have local disks and most of the file processing will be off these local disks, then LOCKDIRWT should be redistributed so the satellite node will not have to go to the boot node to get directory information on the many locks it is trying to use. NOTE: When a node in a cluster goes down, the cluster will go through a "cluster state transition" and redistribute the locks throughout the cluster. Therefore, a less powerful VAX can become the Lock Manager for most of the cluster by virtue of having stayed up the longest. Currently there is no easy way to redistribute the locks again without going through another "cluster state transition" (ie, shutdown and reboot the less powerful node so it will have to give up it's locks) or shutting down all application programs that might have locks out so they release those locks and then restart the application programs on the more powerful processors first.