From:	MERC::"uunet!CRVAX.SRI.COM!RELAY-INFO-VAX" 19-AUG-1992 04:18:38.58
To:	info-vax@kl.sri.com
CC:	
Subj:	SUMMARY: Checking for VMS debugger.

A couple of weeks ago I posed the question "How can I determine if
running under the VMS debugger?". For the curious, the reason I wanted
to know this was so that I didn't do a lib$signal(SS$_DEBUG) when the
debugger was already there. I initially queried if checking the
exception handlers was a good way to check for the debugger.

Thanks to the following for their input:

diewald@virrus.zko.dec.com (Jeff Diewald)
bnebga@ciba-geigy.ch (Gary Nebbett)
jnelson@gauche.zko.dec.com (Jeff E. Nelson)
MILLER@TGV.COM (Bruce Miller)
carl@sol1.gps.caltech.edu (Carl J. Lydick)
ESM@SLC.SLAC.STANFORD.EDU (Ed Miller)


To summarise, the best way to check for the debugger is to walk the
shared image list and look for the image DEBUG, which is the debugger
kernel.

For those of you wanting to do this, here are the basics: The image
list for the images currently mapped into P0 is pointed to by
IAC$GL_IMAGE_LIST. (Link with SYS$SYSTEM:SYS.STB to define this).
Each image control block in the list consists of a forward and
backward link, 3 less interesting longwords, and followed by a byte
counted string in the next 40 bytes (ie 1 byte indicating the length
and then the characters making up the name of the activated image).
[There is other stuff as well]. For the specifics see section 26.1 of
the VMS Internals book and Figure 26.3.


An alternative, possibly easier, and more documented approach is to
make use of the six arguments VMS always passes to any invoked image
in EXE$PROCSTRT. This is documented in CLIDEF -- see also section
27.6.3 of the VMS Internals Manual and Figure 27.6 for a pretty
picture.  The sixth argument, CLI flags, contains a flag indicating
whether the debugger was requested - the information I am after. This
solution though more documented than the image list has disadvantages
that ruled it out for me. To get this information you have to modify
the top routine of your program. This can be tricky if you are working
in VAXC as that "fixes up" your argument list just before you enter
main() -- you can get round this by not having a main() routine and
just giving the top routine first to the linker, but this is not very
attractive. More importantly if your program is invoked though a
foreign command then CLIFLAGS is not set -- this meant that this
approach was a no no for me.


Selected excerpts from the replies I received follow.

]>>Jeff Diewald said:

]The easiest way to do this is to walk the image list to see if any of the
]Debugger images are present.

]You don't want to be looking at the $SETEXV handlers; we play some games there
]that could confuse you.

]	Jeff Diewald			EASYNET:   VIRRUS::DIEWALD
]	Debug group			INTERNET:  diewald@virrus.zko.dec.com
]	Digital Equipment Corp.


]>> Gary Nebbet said:

]The debugger modifies certain entries in the system service dispatch table. 
]The slot for sys$exit is changed from something like

]	chmk	#0038
]	ret

]to

]	jmp	@#share$dbgssishr+0b68

]The value of the opcode (chmk(0xbc) = nodebugger, jmp(0x17) = debugger)
]provides the information you need. The P1 copy of sys$exit dispatch table 
]entry is located at 0x7ffedf40 (the opcode appears at 0x7ffedf42).

]The following short program run with and without the debugger demonstrates
]the behaviour:

]main()
]{
]        int *x = 0x7ffedf40;

]        printf("%0x %0x\n", x[0], x[1]);
]}


]>>Jeff Diewald commented:
]While this may be true for the release Gary is working with, I wouldn't 
]guarantee it at all for any future revs.  In fact, we've changed how our 
]system service interception (SSI) works - which makes this unreliable.
]You cannot count on this to tell you if the debugger is present.  It's 
]even worse if other programs that get activated do SSI as well.  In short,
]this is a bad idea for a general solution.


]>>Jeff Nelson commented:
]Not always, and we aren't the only ones to do this. This is really not a
]dependable way to figure out if the debugger is active. Jeff Diewald's
]suggestion of walking the image list to look for an image named "DEBUG"
]is the best option I can think of.

]Of course, you'll have to decide how close you want to monitor this
]information: just because the debugger isn't active when the program checks
]doesn't mean that it can't be activated later: the user can type CTRL/Y
]DEBUG at any time.

]-Jeff Nelson
]-DEBUG developer
]-Digital Equipment Corporation
]-Internet: jnelson@gauche.zko.dec.com


]>>Bruce Miller said

]Ahem.  The debugger is not the only application that modifies the System
]Service.

]I wrote a system service monitor that replaces the system services.
]Maybe you could check for that lock that the debugger uses...


]>>Gary Nebbett said:
]The option of a sys$enqw with a flag of LCK$M_NOQUEUE on the user mode
]resource DBG$ALIVE_<pid> followed by a sys$deq if the lock is granted
]seems like a simpler solution than searching for a debugger image -
]beyond knowing the name of the resource, no VMS internals knowledge
]is needed.

]Is there any reason to believe that this technique is more susceptible
]to fail following a VMS upgrade than the activated image list search?

]This lock was probably introduced with VMS V5.0 at the earliest, but 
]the debug shareable images may have changed at the same time - at VMS
]V5.4-2 the debug shareable images in the activated image list are called
]DEBUG and DBGSSISHR, whereas DBGSHR (or perhaps DEBUGSHR) may have been 
]present in earlier versions.


]>>Jeff Diewald said:
]There are probably several different Debugger artifacts you could look for
]and check, if you really wanted to write twisted and unreliable code.  The 
]suggestion above is flawed because that code underwent some change in the 
]recent past and may be eliminated entirely in a future release.  (Recent 
]changes require a more sophisticated mechanism than the DBG$ALIVE_<pid> lock.)

]I strongly disagree with the statement that "no VMS internals" knowledge
]is needed for either of these solutions. (and any other ones depending on
]esoteric debugger artifacts)  These solutions rely on undocumented and
](in both examples cited) unreliable mechanisms not easily visible to 
]system programmers.  They take internals knowledge of the Debugger to 
]verify.  

]On the other hand, walking the image list is not at all unusual.  It's
]a "published" interface in that you can get to the include file on the
]system where it's defined - and that you can go to the VMS internals book
]and see how to do it easily and quickly.  It's not something that's going
]to change.  The system has to know the names of the images it's dealing
]with.

]If you're going to try to solve this problem, then you need to know a little
]about VMS internals.  The image list is the easiest, most straightforward,
]and by far the most reliable mechanism to use.  Keep It Simple - use the 
]image list.

]The lock was introduced in the 5.0 timeframe.

]As for the debugger images, DEBUG is the debugger kernel.  It's the part of
]the debugger that lives in the same process as the user program.  DEBUGSHR
]is the Debugger main.  It has all the UI, the command interpreter and the
]mechanisms for controlling the kernel.  It's always there - has been since
]5.0 as well.  There are other images like DBGSSISHR (system service 
]interception code), but they aren't always activated.  We've added a few 
]images recently, but VMS relies on the names "DEBUG" and "DEBUGSHR", so 
]they aren't going to change any time soon.


Thus the shared image list seems the way to go!
But there is an alternative that may be OK for some applications,
(see my comments at the top for poissble restrictions).


]>>Jeff Nelson said:

]Actually, I think there's a better way, assuming you can modify the
]mainline code of the program you want to check. All programs have six
]arguments passed to their mainline. The last is a "cli flags"
]argument. This contains a bit which indicates whether or not the
]program was activated with the debugger.

]Look in SYS$LIBRARY:STARLET.REQ for the string $CLIDEF. A code example (in
]Bliss and not tested) looks like this:

]----------------------
]module is_debug(main=main) =
]begin
]library 'sys$library:starlet';
]external routine lib$put_output;

]routine main(cliargs : ref block[,byte]) =
]begin
]local
]    cliflags : long;

]cliflags = .cliargs[cli$l_cliflag];

]if .cliflags[cli$v_dbgtru]
]then
]    lib$put_output(uplit byte(%ascid'DEBUG is TRUE'))
]else
]    lib$put_output(uplit byte(%ascid'DEBUG is FALSE'));

]ss$_normal
]end;
]end eludom
]----------------------

]This way is better because it's faster than walking the image list,
]and it's more "public" in that the $CLIDEF structure is documented in
]STARLET, but the image list structure isn't (it's not even in LIB).


Finally, there was a much simpler solution that might be useful to some:
]>>Carl Lydik suggested:

]Well, if you want to know that for debugging purposes, the easy way
]would be to have a global variable that your program initializes to
]zero but into which you deposit a non-zero value at the beginning of
]the debugging session.


Thanks to all involved for your input.
--

    __o         
  _ \<,_       Craig A. Marby    marby@layla.harvard.edu
 (_)/ (_)      
~~~~~~~~~~