From:	US2RMC::"EVERHART@Arisia.GCE.Com" 24-NOV-1996 22:06:58.21
To:	GCE@Arisia.GCE.Com
CC:	
Subj:	vms-futures.txt

Above The Drivers: Suggestions for some VMS futures

From: Glenn C. Everhart
11/16/1996

Problem Statement:
   As file systems become larger and as VMS is trying to be positioned as
the best enterprise storage handling OS, a number of limitations exist which
can be fixed.

   I will not address security issues in this segment but will append some
older letters of mine which touch on those issues also.


Problem 1: VMS ODSn file structures are limited to 1044480 clusters on a
disk. If these were disk blocks, they are about 0.5 GB. Obviously we
can already easily have single volumes far larger than this, and their
size is growing all the time. The limit is currently addressed by
kludges (larger cluster factors, volume sets of virtual disks or using
virtual disks to partition the disks otherwise) which are ugly and which
make VMS increasingly look like a toy. This must be fixed, and can only
be fixed at Digital. 

Problem 2: User disk farms are becoming larger, and are growing by adding
many volumes. For the most part, each volume has its very own file
structure. Giant stripesets are not a good answer due the the cluster
number problem, and because they cannot grow with need. Volume sets
are much the same, though they lack the cluster count limit. They
nevertheless must be backed up together, and directory structures
wind up in the same areas. Something is needed so that multiple volumes,
each having a usable independent directory structure (so it can be
backed up, or added or removed from a larger pool trivially), could
be linked together and managed and accessed as a single directory
structure.

Problem 3: The ODS2 (and probably ODS5) directory system does not scale
well enough for use in enterprise systems. Directories must be contiguous
and require potentially linear searches to find items, and adding or
deleting items from the current structure can be painfully slow, and becomes
far worse once a directory exceeds a modest size. (The name length and
directory depth issues are being addressed in ODS5, but the basic directory
structure is not.) Spiralog may sometime scale better, but the V1 product
excels at write speed at the expense of read speed and may be junked for
something else. Many sites for a variety of reasons will not abandon ODS2/5
and something needs to be done for them.

Problem 4: A number of storage hierarchy operations are addressed very
poorly by current storage options due to limitations in caching. Where
some storage hierarchy members differ in speed from others by "a lot"
there is a way to prevent hierarchical lookups from degrading in speed
due to directory structures being on slow storage areas. The speedup
abilities are relevant for local vs. network files, for solid state vs.
rotating disks, or for rotating disks vs. jukeboxes or tapes or tape
jukeboxes. (I'll add that making netacp work for tcp/ip as well as DECnet
should be done also.)  We should be able to make storage hierarchies look
like local disks, so the same lookup paradigms work for them all.


Now some discussion of approaches to solving the problems.

Approaches:
Problem 1:
   Andy Goldstein tells me the real problem here is a byte field in a
lock value block that's full. This field is the hint that tells VMS where
in the bitmap it's currently allocating space. It is only a hint,
but currently is interpreted as the block within the bitmap.

  If it were interpreted as the "block" of size 1/256th of the bitmap
size, it could point within a larger bitmap. (Bitmaps already are
permitted larger than 64KB.) The changes needed to INIT and MOUNT are
comparatively trivial; while VERIFY (analy/disk) might need work as
well, if the block interpretation, init, and mount were altered, there
could be latent support for larger cluster counts on disk which would
permit investigating what else might need to be fixed up. My expectation
is that very little would need it. The other fields have been promoted to
at least word size, which should hold VMS for a number of years yet.

  This can only be done within Digital, as the logic is within the kernel
and not readily replaceable by any third party. (The idea of handling
the block as always 1/256th of the whole bitmap size was Andy's; I had
thought of a simpler scaling factor but his idea is better.)

  We won't have a chance to see what else might be affected without
fixing the XQP up somehow first, though, and init/mount. It can/should
be latent that this is there till some attempts to use it can flush out
other possible affected code.

Problem 2 and 3:
   This problem can I think be addressed as a third party might, but some
aspects are more tractable if dealt with within VMS itself.

   The key to the solution is to provide a two level directory system (that
is, with two sets of directories), and a file access restart which would
be used as a somewhat more general version of what HSM offers today for
inswap, but would permit full soft links. (The restart should be triggerable
from the ACP XQP interface and should be able to effect a channel reassignment
temporarily to another device, with channel restoration on close; this
could be used to handle the originally designed softlinks within VMS and
more.) 

   Given what I have already implemented (while an ISV), which includes
an open/close intercept that can give a single level softlink via an
I/O restart, I believe this can be done without having to touch the VMS
kernel or the XQP. Something that gets a look at attempted opens prior
to the current XQP open operation can perfectly well read a flag about
a file and restart an open somewhere; my code does this, but only to
one level of indirection. Doing more would require a small extension.

   The XQP QIO interface also has access to the operations that enter
a file into a directory on the local disk. If something operating at
the level just above (or "at the beginning of") the current XQP
interface were to call a database manager to look up the file in a
multi-volume directory (and my code is calling a database manager
of sorts from such a point), that lookup could be made to find where
a file really was, or it could "decide" where to put the file, and
an open restart operation on the channel (possibly moved to another disk)
could then access or create the "real" file and enter it into a
directory on the single-volume directory structure as at present. But
note that the multi-volume directory could cross volume bounds freely
and can be constructed to scale well for large directories.

  The directory access routines could be built into something that would
work more or less like an ACP as proof of concept; the idea would be
to take some ordinary, user-mode database access and use it to store
records with full pathname (in a pseudo-tree with all device MFDs stored
in a "root" directory, for purposes of composing the paths, perhaps),
file IDs, storage device, file ID on the real storage device (there
need to be two because a per device file ID is likely not to be unique
systemwide), and maybe a few other tags. Get one or more usermode processes
that contain a real DBMS, so that this "directory" is basically stored
in some efficient and scalable structure (say, a B-tree). It's
not that difficult to insert a call to such a program ahead of real
lookup. When the directory finds an existing file, the per device
file system (e.g. XQP) is called by resetting the channel to the
desired device, filling in FIB with the correct per-device FID, and
adjusting ref. counts, and letting the I/O proceed. (This is about what
I do to implement partial softlinks; it's quite fast.) You have to
intercept close operations to put the channel back also; I'll discuss
details if anyone gets to that point. Lookups would ordinarily use
the grand path directory lookups, not per-device ones. When a per
device path is used, though, the grand master system can concoct
its pointers to ensure all is consistent. You intercept all file
structured devices (or at least the disklike ones) to do this, so
users don't address some new device...just one of the old ones. (The
intercept of course must ensure it works cross cluster, but most DBMSs
for VMS have this already built in.) On a new file create, the intercept
creates its pointer, finds a device to put the file on, and calls the
XQP to create the file and open it as needed. (On close it would put
the user channel back also if it changed.) 

  In other words, this can be done with a code layer that would live
above, but separate from, existing file systems. If you're willing to
have file size constrained to the volume it starts on, that can be all
you do, and code mods to file systems or to the VMS kernel are nil.
If you want to allow file extensions cross volumes, you'd want to have
information about where all the pieces are stored someplace, possibly
however in extension header tags somehow, and the MAPVBLK routine would
need, perhaps, some extension to "notice" that it has mapped a header
that points to an area on another disk, and that restarts the file open
operation on that disk. That would mean an upcall to this layer
that manages such pointers...perhaps an IRP of some distinctive function
that could be noticed by the intercept could communicate this.

   An "extra directory" layer like this would of course be able to handle
controlling file open/close. It is a short step to allowing open to be
restarted generically. This is used in Unix for automounters, for handling
mount points, and for fully general soft links. I think that any directory
layer added should not only perform well and scale well (better than ODS2
anyway) but should permit a file to be tagged to mean "open the file
this points at", either by a tag within the file or by a return convention
to tell the open processing that it must restart the operation (possibly
finding a new tag now). The latter seems preferable.

  The result of building such a thing would be that users could address
not each disk, but a "disk farm" (or some number of them), in which
disks could be added without penalty and backed up individually or moved
and merged/unmerged by operations that can readily be designed. They
would look like single file structures as far as the user was concerned,
but would underneath be composed of as many or few underlying structures
as desired. Since the database for the user's directory access would
be the top one, it must be built for speed & scalability. The underlying
disks can be what they are; large directories would tend to be spread
across volumes, and basically all file access there would be by file
ID. An intermediate layer could of course apply any desired heuristic
to spread huge directories out on underlying disks as well, so they
would be able to function across a much larger range than had been
previously done.


  I will add too that similar techniques can be used as yet another way
to make files across networks or files on storage systems look like
they were online. DECnet currently pulls in remote directories when
they are needed, and remote files, or creates them, with FAL, and
that is reasonable for access provided extreme speed is not needed in
the opens. A layer such as I envision could however also be used to
build what amounts to cached directories on a local store, and keep
them around somewhat longer than DECnet does. It is also possible
that the file access agent to be called might not be an XQP, but might
be a network object, or might be a storage management system (my idea
being also that a user might have some "environment settings" that
specify how old in a storage system they want to look, how many
levels deep, what kinds of filenames, etc., so that the "filesystem"
they see of such a sort might be kept limited in size.) So if the
storage agent used is kept as a separately callable abstraction,
it is easy to plug in such agents of any desired form, be it local
disk, jukebox, storage library system, network using DECnet, network
using TCP/IP, network using NetGhod-Knows-What, etc. This would also
put this kind of network dependence strictly BELOW RMS, into its own
software layer whose complexity can more easily be kept manageable.
Fundamentally it would deal with getting streams of data, generally
disk blocks at first (at any rate), and allowing record formatting
agents like RMS further interpret these. The file access methods would
tend to use local disk caching and efficient bulk transfer protocols
to move data to these caches. (This would take on a somewhat AFS like
structure in that way; that has been a system of proven speed and
efficiency in bandwidth use, so this should not be bad.)

   What is sketched is a starting point for discussion, not an end point.
It seems clear though that something of this sort would be a useful
thing for customers (in that lots of disks could be treated as single
huge file structures, and directory accesses would become faster)
and might be a fruitful approach generally (in that little or no disturbance
of existing kernel or storage agent logic is needed).

Problem 4. 
   Consider for a moment a jukebox of disks. (Leave out whether they're
WORM or R/W; they may be DVDs before long.) Software to make these
all look like online units all the time exists, but access to a different
disk takes vastly longer than access to any local one. Similarly, if
you have a solid state disk, access to it is faster generally than
to normal rotating disk. Also, if you have local vs. remote storage,
access rates differ. Yet OVMS treats them all alike and has no provision
for handling the differences. Volume sets are designed for disks that
are all online at equal speeds. Ditto stripesets, shadowsets, and a
few more. The system above might consider speed in allocating devices
but handling time preferences is not easy. HSM systems can of course
be used to migrate often used files, but don't ordinarily migrate
directories. (The system above in full glory would do this, but
I have another idea too.)
   If a way to make storage speeds less unequal, where large inequalities
exist, can be devised, many of the issues involved in handling these
kinds of situations can be handled.
   My suggestion is simple. Build a disk cache that lives on a disk.
Use it to cache data from slower disks onto faster ones. The advantage
is that the cache need know basically nothing about file structure and
it can be large...gigabytes of cache are cheap if using a single
disk to cache things off a jukebox of hundreds or thousands of CDs or
of r/w or worm disks. Since the cache storage medium is non-volatile
and automatically readable the same clusterwide, too, synchronization
is needed only when the cache block to underlying block mapping has
to be changed. This may (and I'd hope will) be easier to do and less
frequent than the cache invalidation needed for ordinary cache in
memory.

   The design (as I worked it out about 2/1995) permits a simple scheme
that needs little internals hackings. Basically, you capture I/O
requests as they come to a device, and if they are in the cache
you remap them to the caching device and let them be handled there
by the "real" drivers. The mapping structure would conceptually all
be in memory (though a "paged" one could be used too). If the I/O
(r/w i/o anyhow) isn't in cache, a server is sent the IRP so it can
get the data into cache (possibly flushing old stuff) and requeue it
when data is present. The server could even be user mode code for the
most part. Now, the jukebox "disks" are being access trapped here and
addressed individually, so the server needs a way to get I/O to and
from the real underlying disks bypassing the trap, so the intercept
code either has to free the slow disk UCBs (clear the busy bit)
to send IRPs across, or it has to have another way to get at the disks
without waiting for busy to finish. Either method can be done; the
server might use a second ucb which would have a start-io that called
ioc$initiate (or step2 successors) instead of exe$insioqc and did
its own synchronization to ensure proper operation. Clearing busy
would be even simpler, with the IRP dequeued and handled by the
server till requeued. The server would be able to access databases
of what disks were to be cached and could then flush cache blocks
to the backing store and load more. When the mapping of cache to
underlying device blocks changed, of course, this would have to
involve a global invalidation of the mapping and reload after
it was regenerated. (To some extent a "paged" map might be better
here since the invalidate might not have to affect any other cluster
node, depending on access patterns. The map is envisioned as being
cached-device, LBN range, and handled in big chunks of maybe 100
blocks each, more or less according to need. It might want to be
adjustable too.)

   Such a cache if on a rotating disk would tend to hold directories
and most often used files from a jukebox of, say, CDs or other disks.
Thus, directories would tend to be scannable very fast; ditto index
file blocks. Many files would be there quickly as needed. This is also
a neat way to have solid state disks "just happen" to contain in
cache the hottest files. Handle the cache selection as LRU and this
falls out. And you can put anything on top, even full HSM systems
which now will need to bother with directories less critically.
Frequent stuff would be most available, regardless of file structure.
Even if the slowness of a disk were because it were highly fragmented
for read (as in the original Spiralog design), this sort of system
could make the actual access much more nearly contiguous (depending
on settings).

   If the system were set to "flush" the cache to disk now and then
consider how this might work on WORM. Set it to write to cache first
and then to backing store (a preference the server could have) and
directories & index file would tend to stay cached all the time. If
delete is disabled (and writing on existing files also perhaps)
(easy; alter FDT routines for IO$_DELETE and maybe IO$_ACCESS) data
could be flushed now and then from non directories to a WORM, and
a flush of the rest might be set to be very infrequent; it might
not be bothered with on dismount possibly, at least not every time.
When indexf and directory blocks were flushed, as long as it is not
done many times in a WORM lifetime, badblock revectoring could be called
on the WORM to allocate the new data to new blocks on the WORM each
time. This could possibly let you get away with an almost totally
transparent use of ODS-2 or other "normal" file structures which
work across clusters on WORM media. This would please customers
(among whom I've heard a number of complaints about non functionality
across clusters of the solution DEC sells.)
  






Appendix: Some other documents on these issues I've written in the past.

From: Glenn C. Everhart
Re: VMS Futures

The Problem:

If VMS is to be considered as a viable system for use in corporate critical
servers in the future, it would seem appropriate that it be viewed as the
leading example of certain aspects of the craft of operating system design.

There are two key technologies needed for a future server which are
incompletely addressed by the industry at present, but which will be of
critical importance in the future. These are storage management and
security (taken very broadly in both cases).

By storage management I do NOT mean simply easily used GUI based tools
to allow storage to be moved. What I am thinking about is the full or
very nearly full automation of the totality of storage, including archived
storage, in environments which must scale to very large, distributed
ones with multiple partial management domains. (By "multiple partial management
domains" I am trying to express the notion that different departments within
a company will have detailed information about some users, but neither
want nor be able to use full information about others, nor be able to get
it, YET want to be able to allow some information sharing by category of
user. This is a very common situation in corporations with large networks,
and most deal with information sharing far worse than Digital, and with
many fewer resources.)

A moment's thought will show that this has tentacles which reach into
network management, disaster recovery, system management, and security.
If information assets cannot be readily obtained from a corporate server
complex, and managed without immense amounts of work, it hardly pays to
invest in one.

The other key technology is security. VMS has a good reputation as a
secure system thanks to a lot of work by some very good people. However,
it has been lulled (like much of the rest of the industry) into the notion
that the military security model, rainbow books, and so on, are appropriate
to the problem of commercial security. This model was developed to express
in machine terms the military classification rules, however, and has been
weakest in precisely the areas which are most important to commercial
(and indeed to practical military) use. To be specific, the "need to know"
principle works well when a human brain is the filter. Its application to
computing has been translated into a requirement allowing access to
particular files to be limited to particular individuals or groups. This
in practice is vastly broader and less useful. This broadness has caused
many vulnerabilities to insider losses and to "social engineering" attacks
by outside threats on systems by way of insider access.

Notice too that if a large organization cannot ensure security of its
key information in a large shared server, it can hardly afford to invest
in one.

Current Situation:

A: File system

With respect to file system handling, the SpiraLog filesystem represents
a notable advance, in that it begins to allow a site to relax the
limitations of files having to exist on one of potentially very many
volumes. Its ability (in principle at least) to migrate storage from
one logical volume to another is moreover quite useful in certain forms
of nearline HFS. However, it is yet to be seen whether caching can be
used to overcome its disadvantage with respect to the more standard
file systems in read speed.

In automated backup and handling of archival storage, there are useful
point products as well.

Handling of optical storage, CD-R, and the like is not well integrated
however. At present, the third party solution being used cannot work
in clusters for some applications, breaks the MSCP server, knows nothing
of failover, uses a special driver, and still won't work at all on VMS
versions past 6.2. There exist other third party solutions which
overcome these problems, but Digital won't sell them. Modifications to
the Digital disk drivers will permit robust access to optical media for
the first time in VMS 7.1, however, so that at least the basic device
access will be possible using Digital file systems. (I have Spiralog running
on optical in my office using these drivers.)

The current HSM product also offers some ability to integrate multiple disks,
though it suffers from failure to interoperate with Spiralog and on ODS2
it leaves file headers online.

The plans I have heard for ODS-5 will address the need to expand filenames
to deeper depths, longer names, and more characters legal. However, they do
not address increasing the number of clusters permitted on a single volume
beyond about a million, and do not handle at all any issues of the
difficulty of managing large numbers of entirely separate file structures.
This may however change sometime.

In summary, there is some good base technology deployed, which addresses
parts of the problem space, but it is not well integrated, and the lack
of integration is the worse for the fact that someone wanting to obtain
an integrated solution must buy many small pieces and hope they fit together
without too many seams. The VMS OS itself is not advertised widely as
having a soup to nuts solution because it lives in so many pieces, which
might be better connected.

B: Security

In the security space, OVMS offers good ACL and group based security and
has done a good job of extending its coverage to all security relevant
system objects, so that subverting a well managed VMS system can be quite
difficult and risky to an intruder.

The security is extended by allowing identifiers to be associated with images
so that protected subsystems can be realized...provided that they are
in executable images, not interpretive languages. 

This means that, to a degree, one can regulate who may access files, and
with what images, provided the images are made protected subsystems.
This is a highly useful capability.

It is somewhat vitiated by the fact that one must use ACLs to accomplish
this, and ACLs are not particularly easy to use or maintain. Beyond that,
they don't handle combined selections very well. There are a number of
tools which have been placed on the DECUS VMS sigtapes to attempt to
manage ACLs, or just locate files with ACLs, indicating the problem is
one perceived by our users, but not solved by them.

If beyond this one tries to set up ACLs that can regulate more than
one item at a time, one discovers they grow very quickly to unworkable
size. (If one wants to regulate users access by, say, time of day and
by username, without assigning multiple accounts to each user, the
ACL can grow by the product of number of users and number of times.
More complex checks on environment (e.g. by identifiers conveying
location) per user can lengthen the ACLs more.

The result is that in practice ACLs are able to allow a user access
to a file or deny it, but no more.

User authentication in VMS is basically as good as anyone's, but it should
also be added that many terminal protocols and the X protocol are not in
general encrypted, and "wiretapping" attacks remain a problem for VMS
as they are for other OSs. There has been some very good work done to
address this, but the current off the shelf product shows little of
it. The fact that TCP/IP authentication is typically weak to nonexistent
has not helped, though this is also an opportunity for any system that
can address authentication end to end without noticeable delays.

It should be added that encrypted disk storage is addressed mainly by
a public domain cryptodisk in the VMS space. It is fast and reasonably
strong, but remains a free third party addon. (Commercial encrypting
disks have basically not sold at all.)

In summary, VMS security is a differentiator of VMS from systems widely
perceived to be less secure (e.g. Unix systems, or anything using NFS
as its storage paradigm...NFS has protocol weaknesses which make it
basically a lost cause to secure). However, practical security in an
environment where the main (90% plus by many estimates) threat is from
insiders who either exceed their authority on their own or are duped into
revealing information is not all it could be. VMS can list that it can
control access to objects by user, but the Unix vendors make the same
claims, so that the VMS difference at the moment is its quality of
implementation. It could readily be differentiated by additional abilities
the others can't claim to duplicate.

Solution Space

A: File systems

The ideal that could be provided would be a large storage space whose size
would appear infinite. In this space, old material would migrate to slower
archival storage without manual intervention, reappearing if requested,
and leaving no bits of rubbish behind when gone finally. Current material
would be safecopied to backups periodically, again without intervention (save
to change media), but made available at need by the act of attempting to
locate it. Facilities to undo erroneous erasures, or to archive erased
information, would be present. The storage would not be sensitive (save
in retrieval time) to where the data was, and storage networks might span
very large geographical areas with reasonable performance. 

This sort of apparent structure would be constructed using local file
systems and a meta-layer of glue which can be designed so that the storage
space provided remains a SINGLE namespace accessed uniformly.

To accomplish this, I will propose some design ideas. My assumption is
that a local filesystem...even Dollar...is likely not to represent all
local storage, and certainly not all global storage nor all backup storage
as it stands. Given the HSM package, SLS, and Dollar, one can build something
like a piece of this sort of system now. Doing it all is beyond current VMS
technology.

The first thing I would do (beyond removing directory tree depth limits and
enlarging filenames) would be to add some syntactic sugar to the file
systems so that they could understand a unix-type "root" structure consisting
of all devices with filesystems on them. This would allow device traversal
via path changes and would be immediately useful for some sites with no
added mods, but is in itself a minor thing.

Next I would add a restart capability to file open and create operations
so that they could be redirected by the system, and support using it
in the way soft links in unix are supported. (I have implemented a subset
of this capability in my HSM package, but not generically.) I would
use the capability to allow (not require, mind) identification of parts
of a namespace by volume name, and set up a file resolution daemon which
initially could interact with the operator to get volumes mounted, or
with a robot to do the same, where they were not mounted already. I would
implement soft linking with names so that cross device linking could be
readily handled, and so that directories could be included in this
linking scheme invisibly to the user. However, it would be possible
to specify a part of the namespace in which one might create files
and have them created there, with exact destination chosen by the
system, for whatever parts of storage were selected. There would be
a hook to allow i/o duplication via special drivers also, with
destination wherever the user selected. This piece of the namespace
would be chosen systemwide, and there might be several, since the
files would need to be tagged for lookup in a master directory, or
directory entries tagged. Initially at least I would tag the files since
a mechanism for doing so exists in all VMS filesystems and allow
an external lookup.

I would introduce the option of an access daemon such that where
a disk or disk set was not online when access was needed (but where it
was mounted), the I/O would be suspended and redirected to the appropriate
place. (Again, the I/O redirection I did for Virtual Branches (a
package that lets you treat a disk jukebox as a set of always available
disks) is a subset implementation of this kind of thing. I would extend
the ideas so that anything using VMS I/O would be able to use such a
facility. This means that network or robotic access to online storage
would be a first-class part of the I/O architecture, not a third party
addon only, and not required to use special switching drivers.

To support undeletion and delete archiving, the package I wrote last
year (DPS, Directory Protection system), will do about everything
anyone could want. It intercepts io$_delete and changes the delete
operation semantics, and can detect and react to volumes becoming full
via automatic archiving or final deletion or both.

Automatic migration is best driven off a tool that runs either when
space is needed or periodically. By hooking a disk's io$_create
or io$_modify entries this is easily driven. I would use HSM techniques
to migrate files as needed to nearline storage, keeping a database with
file information in it to permit fast access to full data. By driving
an HSM package off space being needed, it is possible to migrate
older storage offline readily. In a Spiralog type system, moving the
tail of the log to nearline storage may be even simpler. (I would
suggest using an on-disk disk cache system ...I designed one last
March...to make access times reasonable for such access if one uses
a robotic jukebox. Spiralog offers some interesting possibilities of
trying to keep the current head of the log on solid state disk, but
a cache system on disk could be even more beneficial on solidstate
and could allow read access to benefit. The point is that the cache,
in such a case, can hold gigabytes, and thus read hit rates can approach
100% much more closely than has generally been the case before.)

Now, current HSM systems (including Digital's and mine) leave file
headers on disk. A truly infinite store would not do this...disk can
fill up with them...but would allow them to migrate offline too.

One way to handle this is to insert a hook in directory opens such
they cause another access daemon to be notified. This access daemon
would have access to a real, honest-to-God database, with access
times that grow as log(number of entries), which might tell the
daemon in effect "this directory has other entries that were
shelved, but which have had their headers deleted". A profile, most
usefully per user, might be used to limit selections so that a user
might tell the system "I want to see stuff that's a year or less old,
no *.LIS files, no *.obj or *.exe files, descend not more than 5
levels deeper in directory".

When this happens, the daemon goes and gets pointers to these files
and generates a directory with softlinks to some very special device
in it pointing to the files (or subdirectories) included. It
builds its own scratch directory (or perhaps arranges to logically
append its scratch directory) pointing to the new stuff as well as what
was on disk. Now should any of the new files be pointed to, the
special device will arrange an inswap of the file header, followed
if needed by an inswap of the file, to a normal disk and restart
of the open to get it. (The temp structure could be torn down when
the directory was closed. In ODS2 this would be when the FIB goes
away.)

This general scheme will allow file headers to be done away with and
real transparent access to be initiated.

The foregoing would become part of the base file system (or at least
part of the server offering) accouterments so that VMS would be seen
to have near-infinite storage ability if you hang enough of almost
any kind of storage on it. (I'm presuming that outswap to disk is
understood by the HSM in use too.)



B: Security

I think the best start at explaining what is missing here is to refer
you to the manual for my "Extended Access Control Facility" (EACF) which I
will attach as an appendix here. In a strict server situation, there would
have to be file access authenticators running at client workstations
also...supplied by Digital...which would communicate with the facility
whenever an open was taking place on someone's behalf.

The point to be made is that it should be possible to do selection of who
can access a file based on

* username
* where the user is
* when the access is made
* what program is used (even if interpretive in nature!)
* what privileges are enabled (to let you discriminate AGAINST
		privileges)
* whether a user knows the password to the program
* What kind of access is requested (r/o vs r/w vs delete)

as well as the current categories. It should also be possible to
ensure that
* A file/program being open has not been tampered with
* A program does not see high privileges if it is desired it should not
		(this keeps you out of trouble with Trojan horses)
The foregoing amount to pro-active virus and Trojan containment schemes.

Further, it should be possible to hide files (or quietly open different
ones) rather than just refuse action when a break occurs with policy, to
find what is the likely motive; notice should be given too in all
such cases (to the system security people) that such action is going on.

The checking so added should be possible for any of a business' or user's
critical data files. By doing user authentication talking to authenticate
daemons on users' sites (I'm thinking of NT boxes here, or even Windows
or the like ones so long as the protocol gets encrypted to keep the
curious out and be sure traffic is what it is supposed to be) these
access checks can be done below the level of apps, so that they don't need
to be altered to benefit. Of course, one should publish the authentication
protocols so that apps can use them too, but unless the underlying storage
is protected first, protecting pieces of it within an app is useless.

The authenticators here should be designed to work across a network, and
should supply "classes" of users as well as individual information. In a
system I put on the sigtapes a few years back, done for General Electric,
I defined classes "non-employee", "non-US-citizen", "short-timer", and
"security-risky person" to be network wide. I would expect some larger
set, but starting along these lines, to be used. (These days, the
qualifier "Over-21" might be useful!)

In this way, when information was accessed whose master control was remote
at least the qualifiers would provide a measure of network wide uniformity
in access control. Authenticators might be contrived to hook into
whatever network access mechanisms one wanted, but could be made to
query others too, so as to allow a superset of what, say, NT alone
can do. Personally I believe this would be a good thing, particularly
with single signon, to avoid intramural network spoof problems. Where
dealing with a company's really critical data (and especially when writing
it!) some extra controls would seem worth having.

I should add that when I have spoken with bankers at DECUS about having
a system that allows them to let local clerks get at their ledgers
via the ledger system but that can keep them from copying things or
accessing it at all without a separate password, they have been
most anxious to hear more. Normal security is generally OK for most
of their files, but certain of their database files are VERY sensitive
and left more open than they really like because it's the best they
are offered. 

The protections described here (and in the EACF manual) are offered
nowhere else, so far as I know. There are a few products for VMS that
claim to offer realtime notification of unauthorized access, but
I've talked to the companies involved. They can't act until the access
or delete has already been sent, get in a race with the XQP to see if
they can $FORCEX or $DELPRC fast enough to prevent damage, and do NOT
always win.

You'll note that the protection here is rather different from mandatory
access controls. The major commercial issue is in need to know, or
need to access. This can be more finely tuned by the kinds of checks
described here than mandatory controls. (It's easier to manage a
database, too, than to manage ACEs all over the place.)

The environment at Digital is not typical of its customers in some
respects...after hours access to sensitive stuff was MUCH more unusual
at GE or RCA or Siemens than it would probably be here. Also, the
variety of images used to access more important files may be larger
at Digital than the others, on average. These facilities were however
the sorts of things that could have prevented a number of incidents at
those companies had they been available.

Security is a hard sell, of course, but it's getting easier. I'm
inclined to think that the sorts of features I'm discussing can be
added at relatively low expense. They'll still be unique and a
differentiator of VMS from other systems.

Costs:

Much of this depends on how much you build in house vs. buy from
others. My guess is that the magnitude of the work to accomplish the
new items here is on the order of 1 to 2 manyears, presuming too that
the Spiralog efforts continue and are fruitful.



The following is the user manual (kind of old but gives the idea) for
EACF, a part of my Safety package. The menus are not in final form but
the technology has been in existence since about 12/1994.


Appendix: 
EACF User Manual


Extended Access Control Facility (EACF)
 USER MANUAL

This manual is intended for use by those who will set up EACF and use
it to control file access on a system.


Introduction:

EACF introduces a number of new capabilities to VMS. This manual is
designed to:
   * explain what these are and what their intended use is, and
   * give information on how to mark files to achieve added capabilities.

Overview, or, Why EACF:

Work any length of time in a computer facility and you hear the stories:
someone reading another person's mail, some clerk telling a friend
how much the boss makes, some systems programmer typing out a document
that contains long range company plans and was for Vice Presidents
only. People are often curious and occasionally malicious. Regardless, the
information loss can create serious problems, especially if it falls
into the wrong hands, or lands you in court because of some breach of
privacy. 

Technical means to control this are often lacking. If you can only
allow or disallow read/write access to a file by someone, as is the
VMS norm, then (for example) the payroll clerk can edit the payroll
database. Doing this with the normal payroll application, at his
terminal during the day, might be perfectly OK. However, should
he be permitted to access it with, say, COPY? Someone developing
new software might need access to system areas to install his
latest versions. Should he, however, have access also to the technical
plans? 

As these questions make clear, it is desirable for the system to
limit access to sensitive files. By doing so, it gives useful warning
when someone's curiousity gets out of bounds, prevents loss of
information, and reduces organizational exposure to anyone giving
in to temptation to sell company secrets, by making those secrets
inaccessible outside of authorized limits.

Extended Access Control Facility is designed to prevent loss, damage or
unauthorized disclosure of data by controlling disk file access. Most 
security breaches ultimately involve tampering with disk storage. Others
such as login breakins are already dealt with fairly well by VMS if set
up properly. However, VMS doesn't make it possible to regulate what can
be done with highly sensitive data at all closely.

The difficulty of allowing access with certain programs to certain files,
for certain people and times, has led to many incidents of VMS users
browsing areas where they are not authorized by policy, but in which they
have access because their privileges work in a coarse grained way. Often
these are innocent curiousity. Sometimes they are industrial espionage.
With most (some estimates are as high as 98%) of security breaches coming
from "insiders" exceeding their authority, this exposure is a problem in
any organization keeping data online which competitors shouldn't get at.

A number of products claim they prevent unauthorized file accesses in VMS.
Unfortunately, all of them currently on the market, with the exception of
this one, work by accessing data from the VMS AUDIT facility. The problem
with this is that anyone even moderately familiar with VMS knows that the
Audit facility keeps track of security violations, and there are numerous
ways to shut it off tracelessly, including code from public domain sources.
This renders the strategy of these packages ineffective.

What is just as bad, these products find out that an unauthorized file
access has happened from Audit only after the fact. At that point, all they
can do is (attempt to) stop the image or its process, since the file access
has already happened, and often already succeeded. (Where the access is a
delete, the file is already gone.) As a result, they require that a policy
be in place to punish offenders after the fact, since they may have already
done some damage. It's hard to be relaxed about a breach in such a case.

EACF, on the other hand, lives in the VMS file system and prevents
unauthorized file accesses from happening in the first place. It is not
circumvented by privileges, works whether the Audit facility is up or not,
and can ensure that a protected file is NOT accessed where this is not
authorized. Thus, the innocent browser cannot have done any harm, and may
be treated with less rigor than might be needed if damage or data loss may
have occurred. Also, while other packages have to use obtrusive measures
like process deletion to stop accesses, EACF can silently hide files if
that is desired. Where a real violation IS happening, this can be much more
effective in finding out what is being attempted than stopping a process.
EACF of course does generate logs of any detected violations so that notice
of them is preserved.

EACF is uniquely valuable in controlling access by privileged users.  Many
sites have people who have elevated privileges, but who should not be able
to access payroll, grade files, company plans, etc., and who are not
prevented from this by VMS. EACF can protect against exactly such access
(and even discriminate AGAINST people with various privileges if you like)
and gives extensive controls over privileges and their uses above those VMS
has.

Finally, EACF is simple to use. You mark files for various accesses with a
menu oriented utility which selects how a file (or set of files) should be
marked, or which will display what markings exist on a file. All security
relevant information is present at one time in an easily understood format;
you do NOT need to hunt through all disk file ACLs to find it, if it exists
at all. No normal VMS facilities are lost, either; VMS security controls
continue to work normally, though EACF may in various ways alter the
outcome of their tests in understandable ways. (An example of this might be
that if you attach an identifier TWIT to a script, that script may fail to
open some files which have ACLs prohibiting access to identifier TWIT.)

If your business depends on any data on your VMS systems and your operation
is more than a VERY few people, you NEED EACF.

Capabilities:

EACF offers the following abilities:


* You can specify a password for file access. This password must be entered
	before access is attempted, using an EACF utility for the purpose.
	Then if the file password has been entered, and other EACF checks
	satisfied, the file may be opened (or deleted). Otherwise, access
	is denied. The password can be anything at all, and the tokens left
	in user space to indicate the entered password are also process
	dependent, so others cannot use them. The password can be changed
	at any time using EACF, but allows control via a separate piece of
	knowledge if this is preferred to control by time, username, etc.
	This facility has long existed in the mainframe world. Now it is
	available for VMS. While it is less secure than encrypting the file,
	this method also cannot cause a file to be lost forever if the one
	who knows the key dies or forgets it. Also, changing the file 
	password is quick and easy, unlike changing its encryption, so that
	temporary access can be given and revoked at need.

* You can control access by time of day (in one hour increments). You can
	use this facility to prevent file access during "after hours" times
	where there is no reason to allow them to be accessed, or
	"open up" files during periods where normal controls are deemed
	adequate. The access modes for each hour can be:

              Y - Access permitted provided all other security
                      checks pass.
              N - No access (except to users on backup list)
              U - No CHECKS, let everything by EACF checks on this
                      file during this hour. (DEC security remains
                      in effect.)
              P - Check privileges against max privs only
              R - Readonly access only, by anyone; r/w opens are
                      rejected. Other checks may deny access
                      totally as well.
              X - Allow r/o opens, test others normally.
              D - Disallow access by ANYone (even the backup account)

	(Note that provision for a backup account exists, and it can be
	forced to permit only read-only file access, but this facility
	allows further restrictions, operating per file.)

	Notice that this means that writing to a file can be disabled
	whenever one likes even if reading is left permitted. If there is
	a suspicion that some databases may be tampered with off hours,
	this gives a simple way to protect them. The privilege check
	mode is also of particular value, since it can disallow use of
	elevated privileges which might otherwise be used to bypass normal
	system protections by industrial spies or other evil doers.

* You can control WHO can access a file (with both permitted and forbidden
	usernames), WHERE they can access the file (with permitted and
	forbidden terminal names), and WITH WHAT they can access the file
	(with both permitted and forbidden image names). 
	This is the major advantage of EACF. You have, for usernames,
	terminal names, and image names, two lists each. You can specify
	username strings that are acceptable or that must be rejected,
	or can use * to indicate all, or & to indicate to skip a test.
	Likewise you can specify permitted or forbidden terminal names
	(access port names are tried also if they exist), or image names
	that are permitted or forbidden. These are handled with the normal
	VMS wildcard conventions (except that in addition a string with
	& in it means to skip the test). Thus, to say that a given file
	cannot be accessed with the COPY image one could add "*COPY.*"
	to the forbidden image list. (Note: "Access" as used for EACF
	means either open OR delete access. Renaming a file does not
	have any effect on the controls.

	The beauty of this is that all the EACF parameters can be
	controlled in one spot, without huge ACLs and unmanageable
	machinery to fiddle with system logicals or identifiers on the
	hour. This permits really fine grained access to your files.
	If you have an accounting person who works on the payroll
	file, you can authorize access to PAYROLL.DATA during working
	hours, from that person's normal terminals, but forbid access
	after hours, or from terminals like RT* or NT*, and you can
	ensure that the file cannot be accessed with, for example,
	COPY or BACKUP, or perhaps can only be accessed from *PAYROLL*.EXE*
	if only a given suite of programs with such names should be
	used for access. The access lists are all variable length,
	comma-separated lists, so considerable flexibility is available.
	Finally, you can prevent access using privilege (see below)
	to this data, so that people who have privileged access to
	your system, yet have no physical access to the computer,
	cannot simply turn on privileges and read the data to their
	heart's content.

* You can prevent files from being accessed with more than a given
	set of privileges. You do this by attaching a maximum privilege
	mask to a file and if the process trying to open or delete
	the file has any privileges not in this mask, access is forbidden.
	The default mask forbids access using BYPASS privilege. This is
	highly useful against curious privileged users (and any intruders
	who gain privilege) since it means that files can be protected even
	where local conditions force operation with many privileged users,
	or where there is concern that intruders may gain privileged access
	to the machine. The tags that cause files to be checked in this way
	can be kept in kernel space as well, so that they are safe from
	being seen even by privileged users.

	NOTE: This and other checks are applied in addition to the
	normal VMS checking. Hence even a file owner is affected by
	this checking.

	Many sites have people who must do development and need to be able
	to use INSTALL or other privileged functions, and whose privileges
	cannot be revoked without impairing their ability to work. However,
	most of these people have no need to be authorized to see, e.g.,
	the payroll file. Conventional VMS offers NO way to prevent such
	access, and other file "security" packages work by watching the
	output of the AUDIT utility, which can be silently and untraceably
	shut off by publically available programs whatever someone is
	running to listen to AUDIT. As a result, there has been, until now,
	NO way to block access by these people to sensitive data left on
	normal disk files, whatever file protection is attempted. EACF, due
	to its residence within the file system, works whether AUDIT is
	turned on or not, and prevents these folks from using the file
	system to access such data. EACF offers the ability to either block
	access altogether, or to block write access only, and cannot be
	confused by command obfuscation, nor does it care whether the
	access is by program or by console command. Someone who has used
	privilege to change his username can be blocked by finding that the
	privilege he needs to continue to access his system tables denies
	file access. Thus even common tricks one might play to fool an
	access system into permitting access can be thwarted readily.

* You can hide files. That is, if access is denied, you can arrange that
	instead of giving the user a "privilege violation" message, EACF
	can arrange that another file (on ANY disk) can be opened instead
	wth no indication to a user exceeding his authority that anything
	unusual has happened. EACF will of course generate logs of the
	denied access, as it does in all cases of such, but the intruder
	will not know that his access attempt has failed. Therefore, you
	can arrange that he will open FAKE_SYUSUAF.DAT or FAKE_PAYROLL.DAT
	instead of the real SYSUAF.DAT or PAYROLL.DAT. This allows you to
	safely allow him to leave evidence of his intentions (e.g. let him
	try to enter system accounts, or see whose pay records he wants to
	see or tamper with) without exposing your real business data to
	tampering. In effect, the unauthorized user CANNOT SEE the real
	files, but "sees" other files which pose as them. In some cases
	this can be FAR safer than an error message, which will alert a
	person exceeding his authority or an intruder that he has tripped
	some protection barrier. Rather, it seems to pose no resistance to
	tampering, but gives complete resistance, while permitting evidence
	of what is attempted to be accumulated.

* You can attach a cryptographic checksum to the file. At any attempt to
	open the file, this checksum is compared with that of the actual
	data, and the file will not open if they do not match. This is used
	for detection of file tampering and means that if someone should
	tamper with a file, it cannot be used in such a state.  This
	tamper-proofing works for any file type, and is not limited to
	executables. However, it does require that the file be read in its
	entirety to compute the checksum before an open can be done, so it
	is most effective for relatively short files. The most common
	examples of such are DCL scripts used by privileged users, which
	are frequent targets of attacks on system integrity. Certain file
	types (with very long maximum record sizes) cannot be fully checked
	in this way, but these are quite rare; the system provides for
	checks of files with records far larger than the longest
	permissible DCL line. If only a partial check is possible, EACF
	will check what it can. This facility can be used to guard many
	critical files against tampering which might otherwise be a first
	step in breaking into a system. Note too that the file size is used
	in the checksum, and this is tested first, so that anything that
	has changed the file size will fail the check very quickly. 
	Where a file is over 100 blocks long, the script asks for
	confirmation before marking it for this check.

* You can attach a replacement privilege mask to a file which will replace
	all process privilege masks when the file is executed. This can be
	attached to any file, no matter how executed (so that interpreted
	scripts can be tagged too).  This is normally used to ensure that a
	script or image never runs with elevated privilege. If you attach a
	privilege mask to, say, a DCL script or image you got from
	somewhere, and the mask contains no privileges except perhaps
	TMPMBX (and maybe NETMBX), then whenever it runs, the process'
	current and authorized privilege masks will be replaced with masks
	having only these privileges.  (The privileges are restored on exit
	or when the file is otherwise closed.) Thus if any Trojan code
	would normally be triggered by the script finding that it had some
	elevated privileges, it would never find such, even if run from
	privileged accounts. Also, since the process lacks privilege while
	the script or image is running, it cannot abuse privileges that
	might otherwise be available to it. (The authorized mask is reset
	to keep Trojan horse applications from using $setprv to negate the
	privilege lowering.)

	You should note that the UIC is not altered, so that implied system
	access via a low UIC would not be affected by this. Also, if you
	use this facility to raise privileges, be sure to allow E (execute)
	access to the file only and check the file carefully.  (See below
	for a fuller explanation.) This facility can be used to attach
	privileges to DCL command files, but can open security holes if
	this is not done very carefully. Remember: if the file is opened,
	the opening process' privileges are affected. When used to reduce
	privileges, the facility provides a pro-active measure to inhibit
	viruses and other covert code from running.

	Usage Note:

	Privilege, base priority, and identifier modification poses few
	if any risks where these facilities are used to LOWER access
	privileges. In this case, someone opening a file which has
	these side effects only reduces his access. (This might even
	be handy for privileged people wanting temporarily to assume a
	nonprivileged identity without using another account.) However,
	any RAISING of privileges or access rights should be done this
	way only with great care. Anyone permitted to open a file marked
	in this way can in principle gain these added privileges/rights.
	In this case, other facilities of EACF should be used to prevent
	access to the file in undesired ways. For instance, by allowing
	E (execute) access but not read access, and enforcing this with
	a max-privileges mask that prohibits it from being overridden,
	and perhaps further limiting username and image allowed to do
	the opens (prohibit *DCL*, for example). In this way security
	needs can be served while privileges can be more completely
	controlled than vanilla VMS allows.

* You can attach identifiers to a file. These identifiers are granted on
	file open, revoked on file close. This can be used for any
	convenient purpose, though again they tend to make sense most for
	files which are executed. If used for data files, it is wise to use
	other EACF facilities to narrow the exposure these have. If for
	example you have an identifier BUS_PLAN that grants access to
	certain files, you can "attach" that identifier to an executable
	or script and allow it to access them. (DEC has something similar,
	but it won't work on interpreted code such as DCL scripts or
	interpreted languages of any kind. With EACF, you can apply
	protections regardless of language choice.)

* You can set the base priority of a process so that when a file is open,
	the base priority is replaced by the one attached to the file. (The
	priority returns to its old state at close.) This can be used to
	ensure that known "CPU hog" programs run with lowered base priority
	even if run interactively, or that some programs run with higher
	priority where this is desired. The first case is likely to be more
	useful. In some shops, elaborate systems to ensure that long
	compilations (e.g., large Ada programs) run in batch have been set
	up. This facility means that the CPU intensive programs do not gain
	extra priority from running interactively if they are set up with
	a base priority lower than normal. Likewise, some editors or screen
	intensive applications may benefit from higher priority, and will
	not cause problems so long as they don't go into a compute loop.


				NOTE

		The three above modifications are flagged with
		cryptographic authenticators so that the tags on
		the files cannot be altered or moved to other
		files. A key that is used on loading EACF for
		each disk is also used in the computation, so
		such keys cannot be entered successfully unless
		one also knows this value. It is essential to
		system security that these key values be kept
		secret, and is desirable that the tagging program
		be available only to systems personnel. These
		modifications do not require any user-mode code
		to run, so impose negligible overhead.

* You can set EACF to have a set of "exempt" images which are permitted
	full system access. Access to these images can however be controlled
	by EACF. This permits EACF to avoid causing problems for disk
	defragmenter programs and the like, which must have a global
	view of the disk, without imposing large user burdens. Because
	access to these exempt images is still controllable, they can
	be prevented from causing security holes.

		It is also possible, for example, it is possible to have a
	copy of BACKUP which is allowed to have global file access, BUT it
	can be specified that this BACKUP image can be accessed only by
	certain users, at certain terminals, at certain times, with certain
	privileges etc., and perhaps accessed only with certain identifiers
	which might be granted by attaching them to a backup script which
	grants the identifiers. The full panoply of EACF controls is
	available to screen who can use this copy of BACKUP. Meanwhile,
	another copy of BACKUP.EXE might be the one most users can access.
	The specially privileged copy would typically be accessed using a
	different verb, and could be used for making system backups.

		This facility is essential to permit programs like disk
	defragmentgers to have global system access without interference or
	to allow system wide applications like backup to be run without
	interference when done by authorized people at appropriate places
	or times. Up to 32 image pathnames can be exempted. 

		In addition, an image is provided which can be used by a
	process to exempt itself from EACF checking or reenable it. This
	image must be treated with extreme care, but can be used for complex
	jobs where system maintenance is needed.


* There is provision for a "privileged user" list (empty by default) which
	can be used to give certain users broader file access, limited
	only by certain time codes and privilege mask tests (and DEC
	access controls), should such be desired. A supplied privileged
	utility can also be used to exempt a process from EACF controls
	where this is wanted for whatever reason. This is referred to in
	the menus as the Backup Account. It can also be prevented from
	write access to files while allowing read access.

* You can arrange for EACF to execute an arbitrary set of commands whenever
	access to a file is denied. The username, PID, terminal, and file
	ID are available to the commands (plus whatever else you care to
	make available). This can be used for mail or pager notice to
	systems people, for suspending the process, or for starting a
	keystroke monitor like SPY to record the process' activities, or
	for ANY other desired purpose. This takes place BEFORE the file
	can be accessed.

DISCUSSION

EACF is useful in relatively friendly environments, where it allows the
	limits of authority to be defined in the machine, so a privileged
	person who may be inclined to browse in public directories will be
	prevented from thoughtlessly printing private files with a message
	that he has exceeded his privileges, or in much more hostile ones
	where tampering or industrial espionage (which often uses the
	services of insiders who have been paid off to make (or have been
	tricked into making) a quiet copy of the payroll, or business plan,
	files) might be a risk. In either case, it is rather easy to limit
	access to sensitive files (or to directory files holding sensitive
	files) to certain people, and to remove the opportunity, and
	temptation, to disclose other data from people who might otherwise
	do so for whatever reasons. If, for example, your system manager
	should be curious about a file named "NEW_PRODUCT_PLANS.DOC" and
	tries to look at it, isn't it better that he be unable to do so
	(even though he may have BYPASS privilege) if company policy means
	he really shouldn't have access to it? Isn't HE safer with a system
	that will just block access rather than an alternative system that
	can't block access, but will get him in trouble after he DOES
	access it? Conventional VMS lacks the ability to do a detailed job
	of limiting access to such areas, so inadvertently this kind of
	thing happens a lot. With EACF installed, it is easy to have the
	machine understand, and enforce, the limitations of access company
	policy and safety demand. Thus, inadvertent as well as intentional
	access is controlled, and data access can be managed properly for
	the first time.

	EACF protections are in addition to normal VMS ones, and a person
	must be able to open a file to mark it for security. Henca all
	normal VMS group access provisions can be used to control which
	files a given group can mark.



WHAT IS HAPPENING TO FILES MARKED


You mark files in EACF using a menu-oriented file marking script, though
a command line interface exists if it essential to use one. This marking
results in an application ACE (the file mark) being attached to the file
and in the EACF database being updated to reflect desired actions on the
file. In addition, EACF startup can be set so that the server is passed a
list of file IDs which will be inspected. This list may contain any
number of IDs per disk and can be created from simple

$ DIR /FILE_ID 

commands. The list is a sequential file each line of which contains a file ID
in the form (n,s,r) where n is file number, s is sequence number, and r is
relative volume number. For example the line

LOGIN.COM;85         (9493,12,0)

will do fine.

When a file is marked like this, one uses a /FILENUM:sequentialfilespec.typ
switch to tell the server where the list of IDs is, and the server will then
check extended access even if the file mark has been damaged or removed from
the file. A file mark must have been created in the first place, so that the
EACF database will have been filled in, but if a mark has been deleted, this
will prevent such deletion from interfering with EACF checks.

There is an option in the menu system which checks that all files in the EACF
database correspond to files with correct file marks and will regenerate
these marks if any are missing. It also warns the operator of any files whose
marks have been deleted, since this may represent someone having attempted to
get around the EACF system's protection. Any unexplained cases of marks
disappearing should be investigated as security breaches. If this happens,
also, it indicates which files should be in the kernel-based lists, so that
they will remain protected even if their marks are deleted entirely. There is
no mechanism for removing files from the kernel based lists except a reboot
and restart of EACF with a different, or no, /FILENUM switch on the command
to start the server process.

The EACF setup script asks, for each disk selected, a security key (used
as part of the authentication for all ACEs that change identifiers, base
priority, or privileges) and a file containing these file IDs. Either,
both, or neither may be selected. Note too that directories are files too,
and can be protected, with the usual caveats. However, if the file
[000000]000000.dir is protected, EACF protection criteria can prevent
a volume from being mounted. 

HOW TO SET EACF UP


Pre-Installation:

Before installing EACF, you should prepare for the installation by
making sure that there is enough nonpaged pool available for the
dynamic needs of the system. These vary somewhat but can be adequately
approximated by increasing the SYSGEN parameter NPAGEDYN by the quantity
(10000 * (number of EACF daemons you will set up + 1)). Insert a line 
something like the following

ADD_NPAGEDYN=30000

in your MODPARAMS.DAT and run AUTOGEN. You may of course have adequate
nonpaged pool already (many systems will) in which case no modifications
are needed. SHOW MEM/POOL will show this.

Your system disk must have about 5000 blocks free to install EACF, though
if the EACF components are placed on a separate disk, system disk space
required for the final resident components is under 500 blocks.

Installation:

EACF is installed by VMSINSTAL. From the system account, use a command

$ @SYS$UPDATE:VMSINSTAL EACF010 distribution_medium:

to do the installation. The script will ask where EACF programs and
scripts should be stored, and most EACF components will be placed
there. This can be anywhere on the system, provided that the disk is
mounted in your startup script before EACF is started. These components
will occupy around 2000 blocks on a VAX, double that on an AXP, so
ensure the disk chosen has room enough. When running the setup script
after installation, the area chosen for the EACF components should
be specified as the area for EACF executable images.


Post Installation:

EACF comes with a setup script which is to be run after the VMSINSTAL
script has installed the package. This script (JT_SETUP.COM)
generates a startup script which should be run from SYSTARTUP_VMS.COM
or SYSTARTUP_V5.COM at boot time. The script generated will start one
access server which will be attached to all disks in the system, and
enough intercept driver units to control all disks selected. It is
perfectly legal and permissible to have many EACF servers, each handling
one or more disks, however. This allows faster processing of security
markings in some cases. The script offers an option to select the
starting intercept driver unit number, so that it can be run more
than once, and will arrange to load a server for all disks in each
run. Normally it starts with intercept unit 0, but if you connect three
disks, you can run the setup script again and start at intercept unit
3 (since the prior run will have used 0, 1, and 2). Then with this run,
another server will be started. It is up to you to ensure that disks
allocated don't overlap.

JT_SETUP is fairly easy to understand. Its initial menu looks like this:

    EACF SETUP  18:11:55

   --> *Set area to hold EACF database files
        Set start intercept driver unit number (now 0)
       *Set area for EACF executable images
        Done this menu, process disk selection
        Remove a disk from an existing EACF configuration
        Set images which are exempt from EACF (e.g. defraggers)
        Quit, do nothing

 ( Asterisk means that a value already exists for the item.)

It allows you to specify where EACF database files are kept (for
the series of EACF intercept driver units you configure). Each
run of JT_SETUP will let you configure one or more disk units,
each of which uses a JT intercept driver, with its necessary
data areas. Normally this starts with JTA0: and goes up in
units, but if you want multiple servers you can run JT_SETUP
once for each server and set the "start intercept driver unit number"
item to the start unit number, which should be one higher than the
maximum number of disk units configured so far. One JT unit can
control only one disk unit. Asterisk implies that values already
exist (perhaps from kitinstal) for these selections.

The "Set images which are exempt from EACF (e.g. defraggers)" item
allows image device:[directory]filename.type;version strings to
be specified for images which are exempt from EACF controls. Note that
any such images can bypass all EACF actions, BUT you can and should
use EACF to control access to these images.

When you select the "Done..." item, you enter a menu to select disk
units to configure for this EACF server. It looks like this:

 EACF Configuration
                 Disk Selection
    Use arrows to move to selection. Use RETURN to select.

 End disk selection
 _ARISIA$DKA700: VMSSYS
 _ARISIA$DKB0:   USRBAK
 _ARISIA$DKB400:  DWU
 _ARISIA$DCA0:   XRP07
 _ARISIA$DCA2:   XRP06
 _ARISIA$DCA3:   CDI
 _ARISIA$DCA4:   XTRA
 _ARISIA$DCA5:   CDTPL
 _ARISIA$VDB0:   TEST1
 _ARISIA$VDB1:   TEST2
 _ARISIA$DKB200:  <unmounted>
 _ARISIA$DKB300:  <unmounted>
 _ARISIA$DCA1:  <unmounted>
 _ARISIA$DCA6:  <unmounted>
 _ARISIA$DCA7:  <unmounted>
 _ARISIA$FQA0:  <unmounted>
 _ARISIA$FQA1:  <unmounted>
 _ARISIA$FQA2:  <unmounted>
 Type H for help.                              Currently on item 1 of 151


where mounted disks appear first on the list, followed by all other
disk-class devices on the system. The display will scroll if more disks
are there than will fit on a terminal.

Once you select a disk, you must enter a number of parameters for
that disk. This is done in another sub-menu which looks like this:

    EACF DISK SETUP  18:19:45

  -->   Enter security key for disk
        Enter file of file IDs to load for kernel marking
        Examine only kernel-marked files for access tests
        Set to automatically kernel-mark all files on load
              (This requires generating the file list at boot)
        Generate filelist database and set to use it
        Act as if ALL files were kernel-marked (use if most are)
        Quit, do nothing.
        Done this menu



The security key for the disk is the secret string (of any reasonable length)
which is used to validate EACF actions that alter privilege, identifiers, or
base priority. It must match the marking on a file or these will not be
altered (which prevents random people from creating such markings). Therefore
this key should be well chosen and the EACF startup file carefully protected
from browsers. Note that when a marking on a file is created which resets
privilege, grants an identifier, or resets base priority while that file is
open, a file security key is asked for. That key must match the volume key
entered here or the changes to process priority, identifiers, or privilege
are not made. This prevents unauthorized people from gaining privilege even
if they are permitted to mark their own files with EACF.

The other options are pretty self-explanatory. If you can afford the
time to create a file of IDs to load at EACF start, it makes the
system MUCH harder to avoid and allows the option of only examining
marked files, which reduces overhead on nonmarked files and is therefore
a good idea too. Whether to use item 4 or item 5 depends on how often
your files that need to be examined change. Item 4 (automatically kernel
mark all files on load) generates a complete list of all files that are
marked at every boot. Item 5 generates a list when you select it and
uses that, rather than generating the list at every boot. This can suffice
for fairly static lists of files to mark.

Once you finish with the menu, it will create EACF_STARTUP.COM
and EACF_LOGIN.COM which are used to start EACF and to define some
symbols needed for convenient use. EACF_STARTUP.COM should be run
from systartup_v5.com or systartup_vms.com (depending on version of
VMS) at boot time.


WHAT TO MARK

You will want to mark files (and perhaps directories, though remember
that files can be opened by file ID without going through directories)
which are important to your business with EACF. Transient files should
ordinarily not be marked. 

Critical system files should be marked. These include SYSUAF.DAT,
SYS$SYSTEM:NET*.DAT, and the startup scripts (so people can't easily
find out what nonstandard things you're running), as well as
any other .COM files usually run by SYSTEM. Be sure, however, to allow
access BY the SYSTEM account to these.

You should also use EACF to regulate the access to the files in the area
you choose for your EACF data files. The files in JTD$DB and GCY$CM areas
hold your security database and must be protected against access by
unauthorized people. It is also often a good idea to access protect
the file marking utility also (filename JTAUTHMAINT.EXE) so that only
authorized people may mark files. Limiting access to the *.GDB files
to the image JTAUTHMAINT.EXE (and including the whole path) is a good
idea. Also, the scripts used to start EACF up, which will contain
the volume access keys, should be protected against access too, so 
that users cannot enter file markings that alter identifiers, etc.
If you use the SETUP script, the file to protect is

SYS$MANAGER:EACF_STARTUP.COM

and it should be treated and protected at least as well as SYSUAF.DAT.
Note that the EACF setup script will, by default, protect the EACF
databases against access from images other than EACF's images and
BACKUP. It is recommended additional tailoring be done however.


HOW TO MARK FILES

You mark files in EACF by using the mark_files script JTMARK_FILES.COM.
The MARKFILE command does this.  (This presumes EACF has been installed and
generally presumes a privileged account to assure the ability to tag files
appropriately.)

A brief note first: permitted and prohibited usernames, imagenames, and
terminal names are matched with string matching, so that to allow access by
BACKUP.EXE, for example, you would use a string like *]BACKUP.EXE;*, since
otherwise it won't match the full path name which is available for matching.
Therefore most of the time your selection strings will contain the *
character, and the file marking script will issue a warning message if this
is not present. If your selection, for example, IS the complete pathname for
an image, or the complete terminal name (or access port name) for an image or
terminal respectively this is not a problem. In any case, your selection is
used. If however you want to (dis)allow access by, say, COPY, you can do so
by excluding *COPY* more compactly. There is a limit in total size available
for this information, so conserving space is often a good idea.

Now to actual use of the script.
In fact there are two scripts. One is named JT_SEL_MARK.COM and uses a
fullscreen frontend as a selection front end. With this script, you set your
default directory to the tree where you want to select files and run the
script, using the arrow keys and spacebar to select/deselect files. When you
are satisfied with the selections, type G to mark the selected set of files.
The selection menu is the same for both scripts. The MARKEACF command runs
the fullscreen - selection markup script.

The other script (jtmark_files.com) asks for a file specification explicitly,
which may contain wildcards.

The start of that script looks like this:

$ MARKFILE
File(s) to mark (? for dir):

At this point you select a file (or a wildcard file specification, in
which case you will be marking each in succession). The selection could
for example be:

$ MARKFILE
File(s) to mark (? for dir): VDB0:[EVERHART]ND.TXT

[Alternatively one can just use a command like

  $ Markfile vdb0:[everhart]nd.txt

to accomplish exactly the same thing.]

At this point, if the file exists, you are presented with a menu which
looks like this:

      Extended Access Control Facility - File Marking
            File VDB0:[EVERHART]ND.TXT;7
    -->  Edit max-privs-to-open mask (current: DFFFFFFFFFFFFFFF)
         Set base priority override (current: None)
         Edit permitted user list (*)
         Edit forbidden user list (None)
         Edit permitted term. list (*)
         Edit forbidden term. list (None)
         Edit permitted image list (*)
         Edit forbidden image list (None)
         Edit backup account list (None)
         Edit time-use profile (YYYYYYYYYYYYYYYYYYYYYYYY)
         Edit replacement priv mask (when file open) None
         Set backup user access [Readonly/Read-Write]  R/W
         Set file tamper check
         Set file password    Set AccFail-action cmd @GCY$CM:JTACTION
         Enter security key
         Set conditional softlink (None)
         Grant identifier on open/revoke on close ( )
         Quit, make no changes
         Done, update file controls now
         Perform database maintenance commands
         Apply selected mods to all wild files


In the above case, the settings are the defaults. The max-privs-to-open
mask excludes access with BYPASS privilege but allows all others.

If we select the first item (by pressing <cr> (the RETURN key)) the
system shows another menu which looks like this (showing all but the
BYPASS privilege selected:

       Privileges Permitted For Opening File  Privilege Set Up

  -->   Done setup    * MOUNT       * U05        * U25
        Set all priv  * OPER        * SECURITY   * U26
        Clr all priv  * EXQUOTA     * U07        * U27
      * CMKRNL        * NETMBX      * U08        * U28
      * CMEXEC        * VOLPRO      * ACNT       * U29
      * SYSNAM        * PHY_IO      * U10        * U30
      * GRPNAM        * BUGCHK      * U11        * U31
      * ALLSPOOL      * PRMGBL      * U12          Set NO mask
      * DETACH        * SYSGBL      * ALTPRI       Make no changes
      * DIAGNOSE      * PFNMAP      * U14
      * LOG_IO        * SHMEM       * U15
      * GROUP         * SYSPRV      * U16
      * NOACNT          BYPASS      * U17
      * PRMCEB        * SYSLCK      * U18
      * PRMMBX        * SHARE       * U19
      * PSWAPM        * UPGRADE     * U20
      * SETPRI        * DOWNGRADE   * U21
      * SETPRV        * GRPPRV      * U22
      * TMPMBX        * READALL     * U23
      * WORLD         * U04         * U24


At this point we can toggle any privilege setting by selecting it
and when done select the "Done setup" item.

The "Set base priority override" item, when selected asks the
question:

Enter base priority for process with this file open:

which can be answered by a priority between 0 and 31. If outside this
range, or if nothing is entered, the priority setting is left
unset, which means that EACF does not alter it.

The next 7 items:

         Edit permitted user list (*)
         Edit forbidden user list (None)
         Edit permitted term. list (*)
         Edit forbidden term. list (None)
         Edit permitted image list (*)
         Edit forbidden image list (None)
         Edit backup account list (None)

all lead to similar menus. If we select the "Edit forbidden user list"
item, for example, we get the following:

           Edit Forbidden User List
        Current user list: None

        * is wild card, & means omit this test


     --> Add users to list
         Remove users from list
         Replace list
         Quit, leave user list alone
         Done with this menu

If the string when you are done contains a '&' character, the test
corresponding to that menu is skipped. * on the other hand is a wildcard
character. In several lists, a message is issued if no * is seen, since
image paths or terminal lists will often contain them if they are to perform
as expected.

Now if I select "Replace list" and add the value GUEST, the question and
response look like:

Enter user list: GUEST

When we select the "Done..." item, we return to the top level which
now looks like this:

      Extended Access Control Facility - File Marking
            File VDB0:[EVERHART]ND.TXT;7
         Edit max-privs-to-open mask (current: DFFFFFFFFFFFFFFF)
         Set base priority override (current: None)
         Edit permitted user list (*)
    -->  Edit forbidden user list (GUEST)
         Edit permitted term. list (*)
         Edit forbidden term. list (None)
         Edit permitted image list (*)
         Edit forbidden image list (None)
         Edit backup account list (None)
         Edit time-use profile (YYYYYYYYYYYYYYYYYYYYYYYY)
         Edit replacement priv mask (when file open) None
         Set backup user access [Readonly/Read-Write]  R/W
         Set file tamper check
         Set file password    Set AccFail-action cmd @GCY$CM:JTACTION
         Enter security key
         Set conditional softlink (None)
         Grant identifier on open/revoke on close ( )
         Quit, make no changes
         Done, update file controls now
         Perform database maintenance commands
         Apply selected mods to all wild files

If we now want to control when this file may be accessed, we select the
"Edit time-use profile" item and get the screen following:

      Time Usage Permission Editing (by hour).
      Current Profile   YYYYYYYYYYYYYYYYYYYYYYYY
                        0         11111111112222
                        012345678901234567890123
    Key: Y - checked acc.   N - No access except bkp  D - no access at all
         U - Unchecked acc  P - Priv checks onlt      R - Readonly access only
         X - R/W chks only

       --> 0-1AM         12N-1PM    Quit this menu, no changes
           1-2AM         1-2PM      Exit, apply changes
           2-3AM         2-3PM
           3-4AM         3-4PM
           4-5AM         4-5PM
           5-6AM         5-6PM
           6-7AM         6-7PM
           7-8AM         7-8PM
           8-9AM         8-9PM
           9-10AM        9-10PM
           10-11AM       10-11PM
           11-12AM       11-12M

To explain the key:
 Y means normal checked access, checking all security criteria.
 N means No access, except that the "backup" account may access the
	file.
 D means no access at all, to anyone.
 U means Unchecked access. The extra EACF tests are skipped.
 P means only the test that allows open if the process has only
	privileges also in the "Max-privs-to-open" list/
 R means Read-only access. Opens for r/w access are denied.
 X means no tests for readonly opens, extra EACF checks only for
	R/W opens to a file.

Each letter selects an access code for one hour of every day.


If I want to disallow use (except backup account use) of the file from
2-3AM, I select that hour and enter N. The selection question is
as follows:

Give code for this hour access:: N

If I know that between 9AM and 3PM the file should be open to all there
with no added checks for reading, but I want to prevent writing except for
users allowed by the rest of EACF, I can use the X code for those hours.
When I have done so the screen looks like:

      Time Usage Permission Editing (by hour).
      Current Profile   YYNYYYYYYXXXXXXYYYYYYYYY
                        0         11111111112222
                        012345678901234567890123
    Key: Y - checked acc.   N - No access except bkp  D - no access at all
         U - Unchecked acc  P - Priv checks onlt      R - Readonly access only
         X - R/W chks only

           0-1AM         12N-1PM    Quit this menu, no changes
           1-2AM         1-2PM  --> Exit, apply changes
           2-3AM         2-3PM
           3-4AM         3-4PM
           4-5AM         4-5PM
           5-6AM         5-6PM
           6-7AM         6-7PM
           7-8AM         7-8PM
           8-9AM         8-9PM
           9-10AM        9-10PM
           10-11AM       10-11PM
           11-12AM       11-12M


Notice the Current Profile now shows X access codes between 9AM and 3PM
(and the cursor has been moved by hand to the exit item.) Now we can
exit this menu and see that the time access profile is set.

Where the file is not executable, a replacement privilege mask makes no
sense for it, but if we wanted to set one the screen would look like
this:

       Privilege Mask to Replace all priv masks  Privilege Set Up

  -->   Done setup      MOUNT         U05          U25
        Set all priv    OPER          SECURITY     U26
        Clr all priv    EXQUOTA       U07          U27
        CMKRNL          NETMBX        U08          U28
        CMEXEC          VOLPRO        ACNT         U29
        SYSNAM          PHY_IO        U10          U30
        GRPNAM          BUGCHK        U11          U31
        ALLSPOOL        PRMGBL        U12          Set NO mask
        DETACH          SYSGBL        ALTPRI       Make no changes
        DIAGNOSE        PFNMAP        U14
        LOG_IO          SHMEM         U15
        GROUP           SYSPRV        U16
        NOACNT          BYPASS        U17
        PRMCEB          SYSLCK        U18
        PRMMBX          SHARE         U19
        PSWAPM          UPGRADE       U20
        SETPRI          DOWNGRADE     U21
        SETPRV          GRPPRV        U22
        TMPMBX          READALL       U23
        WORLD           U04           U24

The privileges selected will replace all a process' privileges which
has the file open, until the file closes. Note the "Set NO mask"
item means that you don't want privileges to be modified when the
file is open. . Note that both current and authorized privilege masks
are replaced. This prevents anything from setting privs with the
$setprv service, since the authorized privilege mask is replaced
also.




If we want to set a file password, we select the menu item and get
a prompt thus:

Enter file password: Thisisaprettylongpassword

The password will not echo, and you are asked for it twice to
ensure you had no typing mistakes.

We can enter a security key; this will allow privilege, identifier,
or base priority changes to work if it matches the one used for the
disk in the setup script.  This prevents random people from marking
their files and altering privileges.
  The question asked looks like this:


Enter security key:

Again, the security key does not echo and is asked for twice to
ensure it is read properly.

	Hiding Files

Also we can arrange that another file be opened if the access is refused.
suppose we want EACF to force an open of VDB2:[EVERHART]LOGIN.COM in the
case access is denied. We select the "Set conditional softlink" item
and get a prompt. With our reply it looks like this:

Enter device:file to open if access denied: VDB2:[EVERHART]LOGIN.COM

	Action Commands

You will notice the menu item "Set AccFail-action cmd" here. This
is a command which can be executed whenever someone tries to open the
file and is rejected. This command runs in a spawned procedure and
can be any command at all. If the command is terminated with a "\"
character, it is executed with no arguments. Otherwise it is passed
5 arguments, namely the username of the rejected accessor, the terminal
name, the file ID (as two hexadecimal longwords), and the PID (Process
ID) of the process whose access was rejected. An example routine named
JTACTION.COM is furnished which just sends email to SYSTEM when such
an access is attempted. Any desired commands are permitted, though
you should be aware that access is delayed till the command finishes.
Hence any long actions may be best placed into SUBMIT commands.
This mechanism can be used to provide for prompt notice of attempted
file access by whatever means are convenient, to suspend offending
processes, or whatever else is desired. Note that the commands can
be different for each file, permitting tailored responses. The default
is no such commands are executed unless you enter one. The current
selection, if any, is displayed on the menu. It is suggested that options
such as turning on session recording via a SPY type program could
be one useful action to enter here.


	Automatic Identifiers

If we want an identifier on the file while it is open we can grant one
with the menu item. Suppose we enter the identifier FIELD. The
prompt and response look like:

Enter identifier to associate with file: FIELD


When done with all changes so far (including setting a priv mask on
open that has TMPMBX and NETMBX in it and nothing else) the menu looks
like:

      Extended Access Control Facility - File Marking
            File VDB0:[EVERHART]ND.TXT;7
         Edit max-privs-to-open mask (current: DFFFFFFFFFFFFFFF)
         Set base priority override (current: None)
         Edit permitted user list (*)
         Edit forbidden user list (GUEST)
         Edit permitted term. list (*)
         Edit forbidden term. list (None)
         Edit permitted image list (*)
         Edit forbidden image list (None)
         Edit backup account list (None)
         Edit time-use profile (YYNYYYYYYXXXXXXYYYYYYYYY)
         Edit replacement priv mask (when file open) 0010800000000000
         Set backup user access [Readonly/Read-Write]  R/W
        *Set file tamper check
         Set file password    Set AccFail-action cmd @GCY$CM:JTACTION
        *Enter security key
         Set conditional softlink (VDB2:[EVERHART]LOGIN.COM)
    -->  Grant identifier on open/revoke on close (FIELD)
         Quit, make no changes
         Done, update file controls now
         Perform database maintenance commands
         Apply selected mods to all wild files


If we select Quit (to abandon the marking) or Done (to perform it) we
eventually get another prompt:

File(s) to mark (? for dir):

and can either enter a filename or a return to exit the script.

Note that if you have not entered a security key, the selection of base
priority, replacement priv mask, or identifier will not be effective. The
security key must be entered, and must match that entered for the volume at
EACF startup to be effective. This protects against unautorized file markings
causing security problems. If any of these are entered and no security key
is entered, you will be warned and given the chance to go back.

The "Apply selected mods to all wild files" is used only where the file
specification given is a wildcard one, and means that the markings just set
up for the first file will be applied to all files in the wildcard
specification. If this is a huge number of files it may take some time. If
you have marked files in the fullscreen front end, this option will apply the
selected controls to all files you selected. Thus you select files, then set
up access controls, and then apply the access controls to the whole list for
simplest use.

The "Perform database maintenance commands" option selects various
EACF maintenance operations. The menu it brings up is the following:

           Extended Access Control Facility
                MAINTENANCE COMMANDS

        Delete this entry
        Remove all database entries for now-deleted files
        Create listing of all files marked
        Repair deleted ACEs (ANY such are a security issue!)
        Change or delete a string in all lists
        Done with this menu

"Delete this entry" means delete the currently selected file's ACE
and database entry, removing it from EACF protection.

"Remove all database entries for now-deleted files" means to go
over the whole database and delete any entries referring to files
which have been deleted. This should be done periodically to ensure
that the database file is not cluttered with old information. It is
handled in this way so it will work even if files are deleted with
EACF disabled. (If the system logical GCY$DELMNT is defined to be
"OK", then EACF will delete database entries when files are deleted to
reduce the need for this maintenance.)

"Create listing of all files marked" will create a sequential list
file of all marked files in the database in the format

Filename (fileid)

suitable for the /FILENUM command's input. This can be used for any
desired purpose and is created as JTD$CM:OP<device>.GDB_LIST. The
DCL symbol JTA_LISTFILE also is set to the filename value. The intent is
that this file is used to set up the map of all marked files used
in the /FILENUM switch (see above).


"Repair deleted ACEs" goes over the database and checks that the
correct EACF ACE (file mark) is in place on each file. Any files whose ACEs
have been deleted are possible evidence someone was trying to access
a file EACF denied access to. Since the ACEs are marked hidden
and privileged, this generally means a privileged user. Warnings
are output to the OPCOM log and the user terminal if this is seen.
The ACEs are then reconstituted. Running this item is a good idea
also; it should be done periodically to ensure that protected files
STAY protected. Files attacked in this way are protected anyway if
they are in the list supplied to the /FILENUM:file-list-name.typ
command. If they were not, they may have been compromised and should
be checked carefully. If they were, EACF protection was not lost, but
it is likely someone tried to break the EACF protection.
************************************************************
These kinds of events represent security related attacks by privileged users.
They must be taken VERY seriously. EACF output to the operator log will give
more information about any denied accesses if the /FILENUM feature was used
so that protection was not actually lost. Also this will show potential
earlier attempts to access the file from before the ACE was deleted.

"Change or delete a string in all lists" is used, for example, to
remove a username from all lists in the system requiring only one
command per disk. It globally replaces one string in the lists with
another. It should be recalled that since this is literal, if you
are, for example, renaming a terminal server, you must replace
the selection strings actually in the database, which are likely
to be of form ",*name*" rather than plain strings. The menu front
end is such that the "`" character (accent grave) cannot appear
in either string. The underlying utility can use any delimiter. However,
this character is illegal in all the selections likely to be there.
The string changed cannot be null. However, it is possible to add a
user to a group by replacing, for example "USER1" by "USER1,USER2".
This can be useful.



	PERFORMANCE ISSUES

Highest performance at runtime is obtained by using the maintenance
commands to create a list of all files in the database and loading
this into the EACF database in kernel mode, selecting the option that
tells EACF only to inspect files marked in kernel. In this mode,
essentially the only files which incur more than a few extra instructions
overhead at runtime are those known to have markings on them; the rest
are quickly passed over. If however the number of files so marked
exceeds around 10,000, it is probably best just to let EACF inspect
all files for markings. As long as there is any header cache at all
in the system in that case, the extra header access done to read the
ACL will only use the cache, and not cause another disk access. Also,
the disk containing the EACF databases should ideally be as fast
as practicable (and remember: EACF should also be used to protect
those database files from tampering). Finally, the EACF servers can
profitably be given higher than normal priority, and should be. They
are I/O intensive when used, and system responsiveness will be improved
if the EACF servers are run at software priority greater than the normal
4.

Even if EACF is checking all files, system access time will not be
noticeably affected save for those files which have extended checks,
and in those cases the checking may add time. The longest operation is
the file integrity check. This check requires, if the file length has
not changed, that the file be read completely before opening by the
user, to ensure that it has not been tampered with. This is reasonable
for short scripts and the like. It should be done with caution for
30000 block images. (If file length changed, this is noticed at once.)

Other operations take negligible time.

EACF can be set up with as many servers as desired, from one for all
disks in the system to one per disk. Basically, the more servers there
are, the more opens with EACF checks can run in parallel. The servers
are however activated only where it is known there is a file marked
which needs server access. ACEs are read from a kernel thread, of
which there can be many active at a time. Accordingly one EACF server
is likely to suffice for most systems.

	USAGE NOTES

To show some of the uses of these capabilities, let's take an example
situation.

Let's say we have several files whose contents are confidential (say,
a business' general ledger) which need to be manipulated by a few
programs which we'll call GL, AP, AR, IN, and PR for short. What we
want to avoid is that anyone browse these files.

One way to do this is to create an identifier (call it GLG) and set
up ACLs on all the files which grant access to the files to holders
of identifier GLG, but deny access to all others. Then we arrange
that each of the 5 images has the GLG identifier granted when it is
run (the programs are set for E (execute) access only) and now the
programs in question can get to the files.

IF nothing else in the system ever gets the identifier GLG, and IF nobody
uses privilege, we're done.

Note however that there are two disadvantages:

1. The control is scattered around, so you need to do considerable
	poking to find that nobody else might have the GLG identifier
	normally, and you must set the ACLs on all files that the
	GL, AP, AR, IN, and PR access.
2. Anyone with privileges can override the controls.

Note that you can fix objection #2 by prohibiting access to any process
holding READALL or BYPASS privileges. Ensuring that nobody has the GLG
identifier requires separate examination, however.

If it is now decided that access at night to these records should be
prohibited, EACF can simply be set to disallow access during those
hours to the files, or possibly to the GL, AP, etc. programs that
access them. Notice that with pure ACLs this is quite awkward, and
blocking privileges is impossible. With EACF, it's easy.

However, there is a neater way to accomplish this.

On each file to be accessed by the GL, AP, AR, IN, or PR programs,
	have a permitted image list containing these image names, and
	a forbidden image list of * (everything) so that only those
	images can access the files in question, regardless of privilege.

Note that now no identifier needs to be created, and the files used
by the programs are clearly marked so that it is obvious by looking
at them using the marking script that only the selected images have
access. Moreover, this cannot be overridden with privileges.

Notice too that preventing access at night to this data is done in
the same records, and in these cases, you protect the confidential
data, not necessarily the programs (which might be run at night on
test data). All the security relevant controls are visible in the
file marking display.

   It may occur to some to ask how the data gets backed up now.
One can of course set certain times of day for uncontrolled access
to the files in question. A simpler way is to have a special copy
of BACKUP that is permitted unlimited system access.

The CLD for such a copy, assuming it is called "SYSBACKUP" would be
as follows (using the VERB utility to extract the original BACKUP
command language definitions):

define type ENCRYP_VAL
   keyword NAME
      value (type=$outlog)
   keyword ALGORITHM
      value (default="DESCBC",type=$outlog)
   keyword VALUE
      value (list)
define syntax BACKUP_LIST
   parameter P1 , label=INPUTS , prompt="From"
      value (list,type=$infile)
   parameter P2 , label=OUTPUTS , prompt="To"
      value (list,type=$outfile)
define type MEDIA_FORMAT_TYPE
   keyword COMPACTION
      negatable
define verb SYSBACKUP
   image SYSBACKUP
   parameter P1 , label=INPUTS , prompt="From"
      value (required,list,type=$infile)
   parameter P2 , label=OUTPUTS , prompt="To"
      value (required,list,type=$outfile)
   qualifier ANALYZE
      nonnegatable
   qualifier ASSIST
      default
   qualifier BACKUP
   qualifier BEFORE
      value (required)
   qualifier BLOCK_SIZE
      value (required)
   qualifier BRIEF
      nonnegatable
   qualifier BUFFER_COUNT
      value (required)
   qualifier BY_OWNER
      value (type=$uic)
      placement=local
   qualifier COMMENT
      value (required)
   qualifier COMPARE
   qualifier CONFIRM
   qualifier CRC
      default
   qualifier CREATED
   qualifier DELETE
   qualifier DENSITY
      value (required)
   qualifier ENCRYPT
      value (list,type=ENCRYP_VAL)
   qualifier EXCLUDE
      value (required,list,type=$infile)
   qualifier EXPIRED
   qualifier FAST
   qualifier FILEMERGE
      nonnegatable
   qualifier FULL
   qualifier GROUP_SIZE
      value (required)
   qualifier IGNORE
      value (required,list,type=$infile)
   qualifier IMAGE
   qualifier INCREMENTAL
   qualifier INITIALIZE
   qualifier INTERCHANGE
   qualifier JOURNAL
      value (type=$outfile)
   qualifier LABEL
      value (required,list)
   qualifier LIST , syntax=BACKUP_LIST
      nonnegatable
      value
   qualifier LOG
   qualifier MEDIA_FORMAT
      value (required,type=MEDIA_FORMAT_TYPE)
   qualifier MODIFIED
   qualifier NEW_VERSION
   qualifier OVERLAY
   qualifier OWNER_UIC
      value (type=$uic)
      placement=local
   qualifier PHYSICAL
   qualifier PROTECTION
      value (list)
      placement=positional
   qualifier RECORD
   qualifier RELEASE_TAPE
      nonnegatable
   qualifier REPLACE
   qualifier REWIND
   qualifier SAVE_SET
      placement=local
   qualifier SELECT
      value (required,list,type=$infile)
   qualifier SINCE
      value (default="TODAY")
   qualifier TAPE_EXPIRATION
      value (required,type=$datetime)
      placement=local
   qualifier TRUNCATE
   qualifier VERIFY
   qualifier VOLUME
      value (required)
   disallow (ENCRYPT.NAME and ENCRYPT.VALUE)
   disallow (DELETE and RECORD)
   disallow (DENSITY and MEDIA_FORMAT)

Now one adds the path to this copy of SYSBACKUP.EXE (a copy of BACKUP.EXE)
to the list of exempt images in EACF. (The correct full path is shown
in SHOW PROCESS/CONTINUOUS when the image is active for the process being
shown.)

In so doing, now, the SYSBACKUP verb can be added to DCLTABLES and
used for making system wide backups. Note too that EACF can be used
to control access to the SYSBACKUP.EXE image, permitting access to
known people at known places and times. It is possible, too, to deny
access to anyone with privileges to the SYSBACKUP.EXE image and have
EACF grant the privileges itself (so that the "maximum privileges to
open" mask would have only a few privileges, and the "replacement
privilege" mask might have all privileges set). [One can also just use
a VMS INSTALL to install the image with privileges.] In this way it can be
made exceedingly difficult for anyone to use this system backup
utility for unauthorized access. It also makes it fairly simple to allow
an otherwise unprivileged operator account to perform these backup
duties without granting the account privilege. By limiting the
ability to use the program to a NONprivileged user, you make it much
harder for anyone to gain "spoofed" access.

The security model supported here is much richer than the default VMS
model, and while it can be enhanced, it covers a great deal of ground
in controlling who has access to what files.

Hopefully the above discussion is helpful in describing some of the ways
in which EACF facilities can be used.



Appendix - Extra Odds and Ends

(The intercept driver is also able to be set to prevent file opens
where a volume is allocated or mounted privately. Normally such volumes'
files are private but the privacy can be overridden in normal VMS with
the SHARE privilege. This option allows the volumes to remain private
even so. Note however that this can mean a print spooler will not have
file access, and may inhibit other actions, so that this is desirable
only some of the time. Access to disks mounted /system or /cluster is not
affected. This operation mode is not currently supported by the setup scripts
however, as it is somewhat esoteric. One sets the 2048 bit of the function
mask to enable this. One also can enable fragmentation reduction by setting
the 2 bit. By setting the 262144 bit, one tells the driver to only examine
opens of files that have had their file numbers loaded into memory. This may
sometimes be wanted for performance reasons. These and some other more esoteric
functions are built into the driver and may be used as needed.) 






o:     SYSMGT::CLABORN
Subj:   RE: YATR: Microsoft Professional Developers' Conference
Enter your message below. Press CTRL/Z when complete, or CTRL/C to quit:
Thanks for the info, George.

Many of these sorts of tricks MS has can be done rather straightforwardly
in VMS, or could be added in with a few (yes, I believe few) definitions
and bits of code. I already have, demoable RIGHT NOW on my workstation,
an app that, aside from doing softlinks, file hiding, extended security
and reliable undelete (by making deletion mean "via a wastebasket") on
VMS, also can be set to fire up a command file of your choice with
any file open.

We could rather straightforwardly also make the internet look like a set
of local disks. I've done some bits like this, but if you think about
how DECnet DAP access works, the netacp interface gives the tool to
allow this.TGV wrote one to talk over IP (that's how phase/IP works)
and we could also, or just use something that translates DAP into
IP requests and use theRMS support for DAP-speaking devices (set
dev$v_dap) and make it look like a pure local disk.

With a bit more hackery (using extensions of the softlink stuff I have
working perhaps) something could be set up that looked like a local
disk, but worked more like a cache of whatever has been asked for on
the net, with expiration dates a possibility too. My package has inswap
technology integrated (it's an HSM too). So does DEC HSM, though I don't
know if it's as flexible as mine (which doesn't care what file structure
it uses). That is the underlying kind of technique you can use to
grab files remotely and make it look local...or at least is one such.
The nice thing about using inswap type techniques is that the files
can really be moved to local storage for awhile, then when they expire
the pointers can be left around. (Takes a little more work for directories
but they could be handled too.)

Most of this would be mightily simpler if there were just a simple way
to tell the XQP to restart an open. I have implemented this in a partial
way, enough for what I wanted, but can see that the fully general
extension is feasible, and can well be done as a third party kind of app
even, ie, without disturbing either RMS or any XQP/ACP/other filesystem
structures, or the VMS kernel. What I am using for softlinks is SO damn
close I wish I had thought it through a bit more when designing it so's
to make a fully general solution. Now it'd take a few weeks hacking and
testing to be sure it worked...heavily on the testing part....

I had thought of this kind of stuff as useful for large storage
archives, so you don't need to teach a whole new search language
to people. (And it IS useful for that.) Still, DEC has had distributed
filesystems longer than anybody, and I'd suspect M$ is using an
approach something like what I outlined above, though it may well be
far less integrated, and hiding it behind a pretty frontend. It
would not be that hard to do it in VMS, with no pretty frontend
required; it could be as goddam seamless as you like.

Glenn Everhart



From:	MOVIES::MOVIES::PALMER "OpenVMS Engineering, UK - DTN 824-3349  02-Jul-1996 0942 +0100"  2-JUL-1996 04:57:59.09
To:	STAR::EVERHART
CC:	PALMER
Subj:	RE: Cache in vms

Hi Glenn,

Many thanks for sending me your ideas. For the first version of the cache,
the intention is to provide a high performance VBN cache that solves many
of the problems of VIOC (e.g. no caching under write sharing, fixed max
sizes etc etc) as well as providing new features for ODS-2 like
write-behind caching. The default behaviour needs to be write-through
though except when RMS is already doing deferred write.

To provide the features we have committed to, an LBN cache is not an option
as we need knowledge of files. The cache is being designed to require the
minimum amount of file system support with a view to possibly implementing
a smaller LBN cache from the same code subsequently.

The idea for a disk based cache is very interesting, but I think its
outside the scope of the work we are looking at here I'm afraid.

Thanks for sending me your thoughts. I appreciate you taking the time.

Julian


From:	STAR::EVERHART     "Glenn C. Everhart 603 881 1497"  1-JUL-1996 16:19:00.01
To:	movies::palmer
CC:	
Subj:	Cache in vms

    Hi there...

    Just some ideas for cache.

    1. If you track back thru window blocks you can find what file an
    LBN access is for (if it's open). Assuming you cache only virtual
    I/O, this allows you to get data for files open only for read
    in the cluster. If you get an MSCP request to read a block from
    some other node, this can occasionally be a win when several
    systems are booting at once or otherwise accessing a common file.

    2. Where you have very slow devices (e.g. tape filesystems, opticals,
    jukeboxes, even HSM files) it can be a moby win to have a disk cache
    ON ANOTHER DISK. You have to synch the block map when it's changed
    from anywhere, but data access does not have to be synch'd since
    it can be automatically visible. This can also be handy in that
    it gives a simple way for directories on the slow disks to
    be cached, with gigs of cache, w/o taking much real memory, and
    regardless of whether all files are. Kinda happens automatically.

    I have a design (and early beginnings of code) for this that I did
    using intercept techniques before coming to DEC.

    The cache should be LBN oriented, so any file structure can use it,
    and can also give a simple way to put cache of a, say, spiralog
    disk onto an electronic one.

    Arrange to do periodic flush back to real disk and not bother
    with writethru (since the data goes to nonvolatile storage on
    the cache!) and you can use it to write WORMs, using badblock
    revectoring on the WORM to do remapping of what is overwritten,
    and doing it only rarely.

    Also very handy for making jukeboxes full of disks look like one or
    two disk volumes (almost) for a pc...

    Another thing it might permit would be a kind of soft directory
    scheme that would in essence allow one directory structure to cover
    many disks using a soft-link technique, while keeping the disks'
    individual filestructures intact. That's a bit off the exact topic,
    and would need a somewhat different intercept to do right, but
    it would mean that someone could treat a disk farm as a single
    structure except for backup, when it could be individual ones.
    Volume sets are almost this way, but they lack viability as single
    disks.

    Maybe you'll want to know more, maybe not...

    Glenn C. Everhart
    dtn 381 1497
    VMS Engineering





From:	STAR::EVERHART     "Glenn C. Everhart 603 881 1497"  2-JUL-1996 09:03:15.40
To:	MOVIES::MOVIES::PALMER
CC:	EVERHART
Subj:	RE: Cache in vms

Thanks for the "Ack", Julian. The first point was a bit of a nicety...
basically you still cache VBNs, but can recognize reads from MSCP.
You might think about this: I've suggested to Eleanor already (RMS group)
that some way (per process?) to force the RAH/WBH bits on for RMS, maybe
even as crude as setting them on the way in, would be a general performance
win, and cheap to do...saves both code path length and time. (We had some
i/o walkthru here where I suggested it and she agreed this would be a
good thing.)

The disk based cache should not get too far in the back of your mind.

Operation is something like this:

Normal startio gets redirected to tell a server about the request for
r/w, and you provide an alternate path to real disks that works like
vddriver (I can send this to you if you don't have it) but calls
ioc$*initiate instead of insioqc.

The server looks in a memory data structure when it gets the message
about work to be done and ensures the data is in cache, or uses its
special "on the side" interface to get it there and write out whatever
was there; it'll cache biggish (~100 blocks maybe) chunks of disk space.
There also needs to be an interlock in the intercept to ensure that
normal I/O is not started while the server's running, since it will
also have to do periodic cleanup, and when the server modifies
the in memory page of the mapping structure it must coordinate globally
to be sure this is cluster coherent. Data r/w does not need this
coordination since it's to nonvolatile store and the map structure
needs to be on nonvolatile disk always. (I'm reciting this from memory...
original docs are at home...so don't be too critical of the order of
presentation please.)
Actually, the intercept driver itself (example intercept driver available
on request in source too) will handle the mapping if the in memory page
of the mapping structure tells it where the cache data is. All it needs
to do is (after all) to clobber the LBN in irp$l_media and point
irp$l_ucb at the right device and get the right locks if it lacks
them; the example intercept shows this. Then it can just queue the
IRP off. I envision several disks served by one cache.

You'll note the beauty of this is that the server does all the fancy 64
bit mapping and so on; once it gets data to cache, a user read is restarted
pointing at the disk cache (backed by your memory cache perhaps, or even
its own special one if one needs...) and restarted. NO funny mapping
tricks needed, and you use a documented interface to read your data.

There is a hook in post-processing now to speed that up...it'll need to
be stolen now and then...so that it can take place all at IPL 8, skipping
the IPL 4 interrupt and fork interrupt that can otherwise be needed;
you set the IRP fastio bit and the finipl8 bit (new for 7.1) together
to do this, then clear them in your intercept code (actually restore to
prior context) and continue with whatever, then do a real post...

By using a server to do the dirty work of the disk cache update, you see,
most of the difficult kernel work is simply bypassed. The server
could in principle work in kernel, of course, but it gets harder to
do; running in a process means simple $qiow can be used for underlying
disk access (or possibly $qio if you're sure the synch can be handled
correctly by your own interlocks, doing the wait in the virtual disk
path instead of the "real" disk path). You need to take some care
about the busy bit, since dkdriver sets/clears it nonstandardly, and 
may want to do some special stuff with it to ensure you don't cause
trouble while error processing on the real disk is going on. This
shouldn't be all that hard.

Note too that write through is NOT needed for this to the real disk...just
to the cache disk.

It was and is my thought that this could turn out to be even more useful
with Spiralog than with ODS2 since it would tend to mean that on disk
reads of the fixed locations would tend to be near the r/w heads of the
cache disk. With one stroke you solve some of the Spiralog caching
problem as well as that of many other devices, and provide a real serious
benefit to buying DEC's solid state disks, which currently are pretty
hard to use in practice (I've heard from customers).

I figured you should have some details...

One other hack for an LBN cluster cache is to break the disk into ~10k
pieces and lock each BTW. Gives a manageable number of locks to cluster
coordinate. A low level driver piece of code can readily examine the
appropriate lock and go ahead and use a cache if valid and no conflict
exists. For writing it'd have to grab the lock first, if not held
already.

The less fragmented the disk is, the more useful this is. One trick I've
used for several years is to trap io$_modify and when a file is extended
I alter the extend quantity as follows:
1. Boost the quantity by some fraction (usually 1/4) of the file size
2. Ensure it isn't more than 1/8 of free space on volume at the time
3. Maximize with a user settable maximum (default 100k blocks)
4. Optionally test that the extend quantity is over some user set minimum
	or bail out
5. Ensure that the quantity chosen is at least as large as the original
	request (VITAL!!)
6. If chosen, every Nth time (default N=1), and if the extent is not marked
	contiguous or contig best try already, set the CBT bit to get some
	of the chaff out of extent cache.

This DRAMATICALLY reduces fragmentation and speeds ODS-2 up a lot,
particularly on large files. I'll send you code to try it yourself if
you want...I use this on my workstations here,using an intercept. Takes
maybe 20 lines of code in the intercept to do the real work. If
you have this in, of course, demos like the one at DECUS may go the other
way, since the trips "to the well" to get disk blocks become fairly rare,
but I think the result will please you.

So much for today's attempt to be helpful...
Write if you have questions.

Glenn Everhart
star::everhart
dtn 381 1497


% ====== Internet headers and postmarks (see DECWRL::GATEWAY.DOC) ======
% Received: from mail13.digital.com by us2rmc.zko.dec.com (5.65/rmc-22feb94) id AA10815; Sun, 24 Nov 96 21:59:32 -0500
% From: EVERHART@Arisia.GCE.Com
% Received: from Arisia.GCE.Com by mail13.digital.com (8.7.5/UNX 1.5/1.0/WV) id VAA00032; Sun, 24 Nov 1996 21:53:16 -0500 (EST)
% Date: Sun, 24 Nov 1996 21:53:45 -0500 (EST)
% To: GCE@Arisia.GCE.Com
% Message-Id: <961124215346.62@Arisia.GCE.Com>
% Subject: vms-futures.txt