Article 152376 of comp.os.vms:
In a message of Sat, 03 Aug 1996 21:04:54 GMT, received on  4 Aug 1996, 04:33
Jon Derman <jderman@cris.com> wrote to Info-VAX@Mvb.Saic.Com

>There is a web page that regularly posts memos that many of my users
>need to see.  Typically, one would access the memos by going to a page
>that contains a table-of-contents (in an html table format).  You look
>at the table and see if there are any new memos posted.  If there are,
>you choose the link to that memo.  The links are in a standard format,
>like  memo9601.htm, memo9602.htm, memo9603.htm, etc.
>
>I would very much like to automate this process and be able to
>automatically d/l new memos whenever they are posted and forward them
>to all my users that need to see them.  I figure it would be much
>easier if the memos were posted to an FTP area rather than a web page,
>but no such luck.  So, in lieu of FTP...
>
>What I would really like to do is set up an automated process using
>the VAX, that regularly checks the table-of-contents page and looks
>for anything new, goes to the link, and saves it in a file.  Or
>perhaps it would be easier to forget about looking at the
>table-of-contents and just save everything using some kind of wildcard
>logic (i.e., memo96*.htm).
>
>I would then forward any new memos to my mailing list.
>
>Do you think I can do this?  Would I be able to set up some kind of
>script with Lynx, perhaps?  Any suggestions would be appreciated.

Funny you should ask this.  I just hacked a little script last Friday. 
The key part is to get fetch_http from the OSU HTTP server, it is in the
[.base_code] directory.  Check http://www.eng.ohio-state.edu/soft/ for
the OSU server.  But as you only need fetch_http.c I've put it up here
on 

    http://www.sternwarte.uni-erlangen.de/ftp/michael/fetch_http.c
     ftp://www.sternwarte.uni-erlangen.de/pub/michael/fetch_http.c


Compile as

$ cc/pref=all fetch_http
$ link fetch_http

This works for me on AXP/VMS + DECC + UCX 3.3.  YMMV.

My simple DCL script, which I submit every midnight, reads like this:

$ i = 0
$loop:
$ i = i + 1
$ if i .gt. 9 then goto end
$html_loop:
$ now = f$cvtime()
$ fetch_http http://www.bbcnc.org.uk/schedules/auntwsi'i'.wir.html -
  s1:[www.bbc]'i'.html
$ status = $status
$ file = "s1:[www.bbc]''i'.html"
$ size = 512*( f$file_att( file, "EOF" )-1 ) + -
         f$file_att( file, "ffb" )
$ date = f$cvtime(f$file_att( file, "cdt" ))
$ if date .lts. now .or. size .le. 0 .or. status .ne. 1  then goto html_loop
$ purge 'file
$ goto loop
$end:
$ submit/after=tomorrow 'f$env("procedure")



Hope this gets you started.

Michael

--
Michael Lemke
Sternwarte Bamberg, University of Erlangen-Nürnberg, Germany
(michael@io.as.utexas.edu or ai26@a400.sternwarte.uni-erlangen.de)