Path: news.mitre.org!blanket.mitre.org!agate!newsgate.duke.edu!nntprelay.mathworks.com!howland.erols.net!psinntp!vitro.com!alpha.vitro.com!vaxs09
From: vaxs09@alpha.vitro.com (John Briggs)
Newsgroups: comp.os.vms
Subject: Re: vax/vms security holes
Date: 24 Dec 97 14:58:23 -0400
Organization: Vitro Corporation
Lines: 86
Message-ID: <1997Dec24.145823@alpha.vitro.com>
References: <67qh67$60i@usenet.srv.cis.pitt.edu>
NNTP-Posting-Host: alpha.vitro.com

In article <67qh67$60i@usenet.srv.cis.pitt.edu>, jamst84@vms.cis.pitt.edu writes:
> hello,
> I am doing a paper on security holes in the vax/vms OS. Actually I am doing it 
> on all Operating Systems, but I have a lack of knowledge as far as vax/vms 
> goes.. any exploit or security hole explained in detail would be greatly 
> appreciated.. this has nuthing to do with hacking.. I am doing it for a paper.
> thanks

Well, it's old enough to be common knowledge now.  Should work on
versions of VMS prior to 4.0.  That's circa 1985.

You find a program that the system manager has installed with privilege.
The notion of "installing with privilege" is something akin to the Unix
SETUID bit.

VMS maintains a list of "known" images.  This list is maintained in a
secure manner and can only be updated by trusted individuals.  When an
image is installed with privilege, it is put on this known file list and
associated with a set of privileges.  When anyone runs this program, they
gain the use of those installed privileges as long as that program is
running.

The theory is that the program is written carefully so that it will
only do what it is supposed to do.  So any extra privileges that are
held while the program runs can't be abused.  And the system manager
will take care to only install trustworthy programs.

There are a number of standard programs supplied with VMS that are
installed with privilege by default when the operating system boots.
Some are required so that the system will function properly.

For instance, in current releases of VMS, SET.EXE is installed with
CMKRNL privilege.

So far, so good.


Now, VMS also supports the notion of shareable runtime libraries.  These
are called "shareable images".  You compile and link your program into
an executable image format that includes your code and includes pointers
to these support libraries on disk.  The pointers work by file name.

When a program image is run, the relevant shareable images are
automatically mapped into the process address space along with the
program image itself.

By default, shareable image files are located in SYS$LIBRARY.
(SYS$LIBRARY is a "logical name" indicating a particular directory
on the boot disk).

In the case of SET.EXE, it is linked against LIBRTL.  When it is
activated, LIBRTL.EXE will be pulled in automatically.

It is sometimes useful to be able to change the run time libraries.
For instance, in a software development arena someone might make
a change to a shareable image and want to test some code against
the modified shareable image without affecting anyone else.

This desire can be accomodated with logical names.  Either the entire
SYS$LIBRARY logical name can be redefined to reference a different
directory.  Or the individual shareable image library name, e.g. LIBRTL
can be redefined to reference a different file.

Logical names can be redefined at the process, group or system level.
Process logical names can be redefined with no special privilege.

Now you know enough to initiate the obvious attack.


This bug was well embedded in the operating system and it took the major
release at 4.0 to root it out.  The fix was to rework the logical name
system and the image activator.

DCL was enhanced so that privileged users could easily create inner mode
(privileged) logical names.  This was via the /EXECUTIVE_MODE qualifier
on the DEFINE and ASSIGN commands.

The logical name translation mechanism was enhanced so that it was possible
to do a logical name lookup that would ignore outer mode (unprivileged)
logical names.

The image activator was enhanced so that, when activating a privileged
image, it would go into a "paranoia mode", refusing to map shareable
image files using untrustworthy logical names.

	John Briggs			vaxs09@alpha.vitro.com