From: CSBVAX::MRGATE!RELAY-INFO-VAX@CRVAX.SRI.COM@SMTP 28-SEP-1988 23:49 To: ARISIA::EVERHART Subj: Re: VMS vs. UNIX file system Received: From KL.SRI.COM by CRVAX.SRI.COM with TCP; Wed, 28 SEP 88 17:30:13 PDT Received: from CitHex.Caltech.Edu by KL.SRI.COM with TCP; Wed, 28 Sep 88 17:18:33 PDT Date: Wed, 28 Sep 88 17:15:21 PDT From: carl@CitHex.Caltech.Edu (Carl J Lydick) Message-Id: <880928131853.164e@CitHex.Caltech.Edu> Subject: Re: VMS vs. UNIX file system To: phri!marob!samperi@nyu.edu, necntc!encore!bzs@ames.arc.nasa.gov, unmvax!mike@ucbvax.Berkeley.EDU, gorodish!guy@sun.com, IMHW400%INDYVAX.BITNET@CitHex.Caltech.Edu, PKLAMMER%CUDENVER.BITNET@CitHex.Caltech.Edu, info-vax@CitHex.Caltech.Edu First, I'd like to point out that this entire discussion has been mistitled. The FILE SYSTEM of VMS is Files-11 ODS Level 2, (hereafter abbreviated ODS-2) and doesn't have anything to do with the internal format of files, stream vs. record I/O, or any of the other things that have recently been discussed under the heading "VMS vs. UNIX file system". In my opinion ODS-2 is in most ways far superior to the miserable excuse for a filesystem used by UNIX systems (and yes, I know, UNIX started out more as an implementation of a file-system than an operating system). UNIX (and here I'm basing my claims on personal experience with ULTRIX, SYSTEM V, NORMIX [a variation on AT&T's TSUNIX developed by Norman Wilson at Caltech to take advantage of VAX architecture; its main advantage over AT&T's product was that it allowed paging], and XENIX; there may be implementations of the UNIX file system which don't have the problems I describe, but I've never seen one) file systems tend to be a bit on the flakey side. When the system crashes, a full FSCK with interactive input from a UNIX guru is called for; the information on the disk (the combination of the state of the filesystem as recorded on the disk and the algorithms for cleaning it up as embodied in fsck) is frequently inadequate for reconstruction of the filesystem (perhaps someone should develop an "expert system" to talk to fsck? 8-}). ODS-2 has at times suffered similar problems (in particular, there was the problem a couple of versions of VMS ago involving extension of INDEXF.SYS by a VAXCluster member), but problems of this sort have been acknowledged to be bugs, and have been fixed fairly soon after they've been discovered, by and large; on UNIX, such problems have been around for long enough without any visible efforts to fix them that they've pretty much got to be considered features by now. Under ODS-2, every block on a disk (including the bad-block track on last-track devices) is either allocated to a file or is free to be allocated to a file. The UNIX file system has an entire class of blocks that aren't allocated to a file and that cannot be allocated to a file: they're called inodes. Since they're not in a file, they're set up as a (doubly?) linked list. Unallocated blocks other than inodes are described by another linked list, the freelist. One of the things fsck does is to search the disk for blocks that aren't in the freelist or the inode list, and tries to figure out which one they should belong to. ODS-2 has all the inode-equivalents (file headers) allocated to INDEXF.SYS. In case the file header for INDEXF.SYS itself becomes corrupted, there's a backup header for it in a readily locatable position on the disk. Information on whether a block (actually, a cluster) is allocated is stored in BITMAP.SYS. Information on whether a file header is in use is stored both in the file header and in a bitmap in INDEXF.SYS. This means that under ODS-2, you can, in principle, find the next free block or file header with a single read followed by a single VAX instruction (though on non-VAX-11 architecture VAXen, the single instruction is software-emulated). Under UNIX, you have to scan the inode list until you find a free inode; allocating the first free block involves removing it from the freelist (which could, I suppose, be done using a REMQUE instruction, if the version of UNIX takes advantage of the VAX queue instructions). Because ODS-2 uses a storage bitmap, searching for the first set of n contiguous free blocks can be done quite efficiently, again with only one read from disk required; under UNIX, you have to scan the freelist, which can be a fairly time-consuming process. Contrary to popular belief, ODS-2 does specify a mechanism for "soft links"; however, the principal interface to ODS-2, RMS, doesn't recognize that mechanism, and with the advent of volume sets, probably never will. The one advantage that I might concede that the UNIX file system has over ODS-2 is that it keeps a count of the number of links to a file, and when the count drops to zero, the file is deleted. On the other hand, RMS's ability to distinguish between manipulating a file and manipulating directory entries can be quite useful sometimes. Now, to address the question of VMS vs. UNIX I/O facilities. The original question was: > Can people who have had experience working with both VMS files (at the > FDL level) and UNIX files (at the inode level, say) comment on the > advantages and disadvantages of the file systems used by these operating > systems? My experience is mostly with the UNIX file system, so I was a > little surprised when I discovered recently that VMS text files, object > code files, and executable files all have different record structures. > What does the added complexity of having to deal with RMS, FDL, CONVERT, > etc., buy? To answer your question, let me start by inverting it: What does dealing with a file as a byte stream buy? Well, it makes it a lot easier to deal with the file in an application where you're really treating it as a byte stream; i.e., where the only context necessary for processing the data is local to the data being processed; e.g., text processing, where most of the time you treat '\n' simply as white space (and even then, sometimes '\n' means something different than ' '); or manipulation of the file as a whole (e.g. copying the file); or data processing where either 1) You're using binary data where the longest data element is a byte; or 2) You're using formatted data where all data items are the same size; and where the interpretation of each data item is the same (e.g., repeated observations of the same variable). In such cases, treating the file as a sequence of records gets in the way. In any other situation, treating the file as a sequence of records can have anything from trivial to critical advantages. E.g.: o Working with, say, experimental data in which each observation consists of measurements of several variables. In this case, record-oriented treatment of the file guarantees that each time you read from the file, you get data starting at the beginning of a new observation, rather than accidentally starting from the middle of a previous observation that hadn't all been read on the previous read. If the number of variables varies from observation to observation, then variable-length records are in order. You can use the record-length to determine how many variables were actually for a given record; if the number and type of variables is constant, you can use fixed-length records, which also provides a check that the record is valid (i.e., you aren't trying to read or write data for the wrong number of variables. This is true whether you're using formatted or binary data. With binary data, you also have the advantage of not having to parse the list of variables to be read: simply declare them as a structure (or, if you're going to figure out what variables are to be used based on data in the record, as a union of structures), and copy a record from the file to memory starting at the beginning of the structure. Right there, you've eliminated most of the overhead of doing the I/O. o Working with binary data of any sort where alignment is important. If you want to edit such a file, and there's anything in there that's sensitive to alignment, you want to make sure that you write exactly as much data as you read. Fixed-length files force you to do so, without code in your editor to check on this. This means that in principle, you can use the same editor to deal with text files and with files containing alignment-sensitive data. o Relative organization files permit you to do random-access I/O to files with variable-length records, without you application having too worry about padding cells to a uniform length. o Key-indexed files provide a uniform way of treating ISAM datasets. I once wrote an application where something that looked to the user like ISAM access to a database was the only reasonable way of treating the data, but for a CP/M system. I quickly figured out that writing an I/O subroutine library emulating RMS's treatment of key-indexed files made the task much simpler than putting the code to handle indexed access in the application itself, and that including both the index information and the data in the same file made it a lot easier to keep things consistent. Having RMS insist on dealing with a key-indexed file as a key-indexed file makes it difficult to accidentally corrupt say, your key information, and is worth the cost of having a different file organization to deal with. On the whole, what you gain by having the various file organizations, record attributes, and other RMS attributes associated with a file or a record is a reliable way to permit different programs to access the same file without your having to explicitl include code in all the programs to make sure that the data in the file is being treated appropriately. UNIX accomplishes something similar via "magic numbers". The key difference, and one that is fundamental to the philosophies of VMS and UNIX, is that when you try to access a file in the wrong fashion under VMS, you can check the file header to find out how the file is set up, while under UNIX you have to go looking for a program that uses the magic number you found in the file if you want to see how the file is organized. Some UNIX utilities use a magic string instead, which helps some, but the standardization that comes from the way RMS does it is, in my opinion, extremely worthwhile. Also, please bear in mind that the cost of having all these attributes associated with files is generally small, and when the cost is disproportionate, it is generally because of implementation stupidities in something at a higher level than RMS (e.g., one of the stupidest implementations of anything having to do with I/O I've ever seen is VAXCRTL. There's no simple way to create a file with the same attributes as those of an existing file, there's no way to force block I/O,..., ad nauseum.). There's nothing in RMS that prevents you from accessing any file (key-indexed files included) as a byte stream; you simply open it for Block I/O; it's the VAXCRTL that really gets in your way. Now, my reactions to selected parts of other responses to your question: > What's far more important, in my experience, is to have an orderly set > of access methods and to use them only where they are truly justified > (ie. simply because it's faster is not a good enough excuse if 99% of > the actual applications will perform faster than human response time > with either method, naive or sophisticated.) I take it that the person who made this comment works mainly with single-user operating systems. Just because the application sends data to my terminal as fast as I can read it already doesn't mean that system performance won't be helped by reducing the overhead associated with the application. As an admittedly outrageous example, suppose you have one interactive user running an application that can send data to his terminal exactly as fast as he can read it, and one batch job running at a lower priority. As long as the user is running the application, the batch job gets no CPU time at all. Reduce the overhead of the application, and the user will have to use his hold screen key (unless the application is doing pagination), but now the batch job actually gets to run. > I remember, for example, when the VMS HELP files went from a very > simple, textual format to their current library format and it made > working with them in new and creative ways nearly impossible (I had > written a full-screen access to the VMS help files in TECO, no > kidding, which was nearly impossible to salvage, I never bothered.) > I'm not sure the changeover was really much of an improvement, sped up > something which was fast enough already and added a lot of complexity > where it was unappreciated, adding a new help topic became more > complicated etc. Well, with the library format, HELP needs to read only the library header until it gets to the entry for the module containing the help text you want, then on its next read it gets the first part of the actual help info, instead of having to: 1) Read every intervening block of the file; and, much worse, 2) Scan every intervening block of the file to see if the desired help entry starts there. Or did the old format require a separate file for each top-level topic? If your users use HELP a lot (people whose first experience with an operating system is VMS tend to do so more than people whose first OS was UNIX; The "simple, textual" format of UNIX MAN pages makes printing a hard copy of the manual page you want to refer to the only reasonable way to use man), the library format for help is a substantial improvement over having to scan a large file for the desired information. > Not a flame, just trying to emphasize my point about it's good to > have access methods, but it tends to lead people astray into using > them just to avoid scanning a file when the latter would perform > fine and would greatly simplify later maintenance (typically, the > file can be manipulated with a text editor) etc. Avoiding scanning a file would make a difference of scanning a total of about 3 1/2 megabytes of text in the case of getting help on WRITE. That's a pretty substantial improvement, in my humble (but correct) opinion. > vms has file attributes directly associated with the file. qio does > read virtual blocks - but you can't easily convince rms to read a file > in some mode other than the mode the file was created with. Sure you can. Just or FAB$C_BIO into the FAB$B_FAC field of the FAB you use to open the file, and use SYS$READ instead of SYS$GET. Then RMS reads virtual blocks, even for key-indexed files. Don't confuse the stupidities of VAXCRTL with limitations of RMS. > Gee, it would be nice if I could look in VMS HELP for, say, all the > references to "logical names" by typein something as pure and innocent as: > $ SEARCH/NOEXACT SYS$HELP:HELPLIB.HLB "DEFINE" > Gosh, what a powerful idea! Imagine being able to invert the help file, > to find out "about" a word, instead of only "help defined for" a word! > > Golly, I'm so glad VMS help has been implemented in their super-optimum > el-primo library access method, so that SEARCH won't work on it (nor > EDIT, nor...) That would be a good point if: 1) Such automatically generated indices were generally worth the time it takes to read them. After you've generated such a beast, you've got to go through it by hand, look at the context of each occurrence of the string, and typically throw out about half the references. In doing so for any reasonable list of strings users are likely to be interested in, you'll probably end up reading the entire help library three or four times, but in random order. You'd be better off using the command: $ LIBRARY/EXTRACT=*/HELP SYS$HELP:HELPLIB Then using a text editor to read the file, pick out the useful keywords as they show up, tack on the name of the help item where you found the keyword, then sort and prettify the result. 2) DEC weren't already in the process of doing what I described in the above paragraph. I haven't looked at the V5 HELP yet, but under VMS 4.?, DEC added the top-level item HINTS to HELPLIB. This directs you to other help items based on fairly broad categories of what you're interested in. I expect this item and the items it points to directly to grow as time goes by. It's a lot more useful than knowing that the word "then" occurs 11 times in the man section for sh, considering that 8 of the "references" are irrelevent.