From: MERC::"uunet!VM1.NODAK.EDU!owner-anu-news" 21-APR-1993 17:33:06.33 To: Multiple recipients of list ANU-NEWS CC: Subj: Splitting ANU files over several disks In article <1993Apr18.220744.1@alien.gici.com>, laut@alien.gici.com (Bill Laut) writes: > A nice option would be to allow us to explicitly specify the device/directory > of a given newsgroup and/or hierarchy, so that we can split the disk > consumption across multiple spindles without resorting to bound volumes This is already provided for you by VMS - use search list in logical name(s). Let me elaborate on this and broaden up a theme somewhat. I will also touch on the issue of disk fragmentation since this is a closely related theme - so please bear with me. I speak of ANU-NEWS V6.1 (beta versions 5,6,7) but I believe that most of it applies also to V6.0. The purpose of logical names NEWS_ROOT and NEWS_DEVICE is somewhat blurred as described in NEWS_STARTUP.COM_TEMPLATE : $ ! NEWS_DEVICE points to the same directory as NEWS_ROOT, but is setup $ ! as a concealed device. This implies that NEWS_ROOT and NEWS_DEVICE differ only in the /TRANSLATION=CONCEALED attribute of a logical name and that ANU-NEWS expects them to point to the same top directory - which is fortunately *not true*. There are two completely unrelated set of files/directories which *happen* to reside in the same directory *if* you follow the suggestion in NEWS_STARTUP.COM_TEMPLATE (which may *not* be a good idea if you have more than one disk drive - more on that later). The thruth is: 1. that NEWS_ROOT points to a directory which must contain the following files: NEWS.ITEMS, NEWS.GROUPS, HISTORY.V60, NNTP_ACCESS.NEWS 2. that NEWS_DEVICE (a concealed logical name) must point to a directory containing top level hierarchy directories of news items (such as COMP.DIR, REC.DIR, ...). I suggest to the ANU-NEWS maintainers to change the comments in NEWS_STARTUP.COM_TEMPLATE to distinguish the purpose of the two logicals. The moving of index-seq files to another disk has already been mentioned in this group a couple of times, once even done by a patch, which clearly indicates the confusion / lack of information in this area in the documentation. The NEWS_DEVICE file system hiararchy (containing text of news items) lives a very dynamic life and tends to fragment disk in no time - - which is not that bad for item files themselves (since they are usually short and do not live long), but it sure gives hell to any *other* files living on that disk. The only thing that is worth considering for this disk (to reduce fragmentation) is specifying CLUSTER SIZE of 3 or even 4 during disk INIT. I believe it is a waste of time to run disk defragmenter in this disk. The index-sequential files in NEWS_ROOT (specially NEWS.ITEMS) also live a dynamic life but in a different way. - If you put these index-seq files (by defining NEWS_ROOT) on another much less fragmented disk (make sure the initial copy makes it in one or very few pieces - use DUMP/HEAD/BLOCK=END:0 to check). and - if you optimize FDL for NEWS.ITEMS and specify a large chunk for file allocation extensions (here is my FDL setup as an example: AREA 0 EXTENSION 13216; AREA 1 EXTENSION 248; AREA 2 EXTENSION 3900 ) and - if you perform CONVERT/RECLAIM on these index-sequential files (the /RECLAIM qualifier is important!) regularly (say once a week) which reclaims space in the file occupied by deleted records and makes it available for new records, than the disk fragmentation becomes a non-issue, since: - you end up with one badly fragmented disk with news items (NEWS_DEVICE) and you do not worry about it - you end up with another disk holding NEWS_ROOT which gets almost no fragmentation from ANU-NEWS. Having file NEWS.ITEMS not badly fragmented brings a *significant* performance boost ! I have been running our ANU-NEWS with this configuration of disks for more than a month now and I am very pleased with the result. I also made sure the *.BATCH files on NEWS_MANAGER (which should *not* be on the same disk as item files) get allocation in large chunks changing #define ALQ_NONDEF to 1 and changing #define OPEN_MODE to: #define OPEN_MODE "w","alq=256","deq=128","fop=cbt" in NNTP_SERVER.C . Now back to the original question: > A nice option would be to allow us to explicitly specify the device/directory > of a given newsgroup and/or hierarchy, so that we can split the disk > consumption across multiple spindles without resorting to bound volumes If after moving the index-sequential files to another disk you still feel a need to split the news items hierchy to more than one disk, you may copy part of a news hierarchy to another disk (at least the directories) and put a search list as a value of a logical name NEWS_DEVICE. Here is an example: DISK1:[NEWS] contains the full hierarchy, e.g.: ALT.DIR, COMP.DIR, CONTROL.DIR, GNU.DIR, JUNK.DIR, MISC.DIR, NEWS.DIR, REC.DIR, SCI.DIR, SOC.DIR, TALK.DIR, VMSNET.DIR (with all the subdirectories) DISK2:[NEWS] contains only a part of the hierarchy, e.g.: ALT.DIR, GNU.DIR (with all the subdirectories) Now you set up the NEWS_DEVICE logical name as follows: $ DEFINE/SYSTEM/TRANS=(conc)/EXEC NEWS_DEVICE disk2:[NEWS.],disk1:[NEWS.] (NOTE: specify the partial hierarchy *first*) This will tell RMS to look in the partial hierarchy *first* and if the required directory/file is not there, than it will look into the next directory in the search list, eventually finding the required file. The newly created item files will be created in the first directory in the list which contains the specified (sub)directory. You may remove ALT.DIR and GNU.DIR (in our example) later from DISK1 by hand or you may leave to the skim mechanism to eventually remove all old items from the original directories. All this is done by RMS alone and ANU-NEWS knows and cares nothing about the trick. Taking the idea a bit further, it also gives an answer to the question posted to this group a couple of weeks ago on how would it be possible to store a particular (generally uninteresting) newsgroup under a particular user's directory / disk quota. The concept would even make it possible to archive e.g. long news items with a long expiration date on another disk and still leave them accessible in a usual manner. Sorry for a rather long text but I hope this will help some sites to forget the nightmare of CONVERTs (without /RECLAIM) producing even more fragmented NEWS.INDEX than the original file, unsuccesful disk defragmenters, poor performance (specially during SKIM), running out of file headers in INDEXF.SYS without ANU-NEWS telling you about the real cause of trouble and so on. For me it works like charm ! -- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !! Mark Martinec, System Manager tel +386 1 159-199 !! !! J. Stefan Institute, Jamova 39 fax +386 1 161-029 !! !! 61000 Ljubljana, Slovenia e-mail: Mark.Martinec@ijs.si !! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!