INFO-VAX Sun, 23 Mar 2008 Volume 2008 : Issue 165 Contents: Re: Divining the full pathname of a file, all logicals translated Re: Divining the full pathname of a file, all logicals translated Re: Hitachi 500G SATA drive in a PWS 600au Re: RIP Arthur C. Clarke Re: RIP Arthur C. Clarke Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Tripwire on OpenVMS Best Practices? ---------------------------------------------------------------------- Date: Sat, 22 Mar 2008 18:42:41 -0700 (PDT) From: Rich Jordan Subject: Re: Divining the full pathname of a file, all logicals translated Message-ID: <7040efff-2b8f-43f4-90e7-56f86a223bd7@q78g2000hsh.googlegroups.com> On Mar 20, 5:16 pm, AEF wrote: > On Mar 19, 5:41 pm, Rich Jordan wrote: > > > > > I'm looking at ways to generate the full "low level" pathname to a > > file with no logicals, concealed, terminal, or otherwise involved, and > > in standard format. > > > So far the way that seems to work closest to what I need using only > > VMS provided tools follows. THis is (currently) a DCL command > > procedure searching multiple directories (using F$SEARCH with a > > wildcard, and generally with a logical in the search term): > > > - F$SEARCH returns the filename. > > - The return from F$SEARCH will already have translated non- > > terminal logicals; for example in the SYSTEM account searching for SYS > > $LOGIN:LOGIN.COM returns SYS$COMMON:[SYSMGR]LOGIN.COM;12 > > > - Take the output of F$SEARCH and feed it to F$PARSE with > > argument "NO_CONCEAL" as the parse type. > > > This returns: > > > NODE$DKA0:[SYS0.SYSCOMMON.][SYSMGR]LOGIN.COM;12 > > > Thats workable but I really need it to be normalized as > > > NODE$DKA0:[SYS0.SYSCOMMON.SYSMGR]LOGIN.COM;12 > > Why, pray tell? If you *really* need it this way, just subtract all > possible combinations of square and angle brackets. (I think that's > pretty safe, but unanticipated situations and upgrades can break > manual parsing in general.) I assume you're writing about files on an > ODS-2 volume, right? You might get into trouble doing this on an ODS-5 > volume, but I'm not sure. > > > > > I can put in conditional code to clean up the f$parse output but I'd > > rather find a way to have the clean pathname returned. Is there a way > > to do this using normal lexical functions? > > > ===== > > > One other alternative seems to be to use one of the FID to NAME > > programs; since I can locate the file in question I have its FID and > > can run it through a program (seen online here in COV, also perhaps on > > freeware) and get the full raw pathname back. That would be > > acceptable too but if anyone's done it and has recommendations I'd > > appreciate hearing back. It would mean many thousands of image > > activations running the FID to Name program. > > Peter Weaver answered this. But why is it so important to remove the "] > ["? > > AEF > Currently ODS-2 and virtually certain to remain so. The full pathnames are being recorded and although VMS utilities handle the various permutations of divided directory paths properly, a prospective program that uses the archives may not, especially if its used or needed by people unfamiliar with VMS filename syntax. I want to make the archived info as uniform and straightforward as possible to avoid any future problems. If I have to deal with the divided directory paths and multiple directory path delimiters I will but it complicates the code; I'm just seeing if there's a better way than doing it manually. Apparently there is but only in later versions of VMS for DCL, or by using a called program. ------------------------------ Date: Sat, 22 Mar 2008 21:29:29 -0700 (PDT) From: Hein RMS van den Heuvel Subject: Re: Divining the full pathname of a file, all logicals translated Message-ID: On Mar 22, 9:42=A0pm, Rich Jordan wrote: > On Mar 20, 5:16 pm, AEF wrote: > > On Mar 19, 5:41 pm, Rich Jordan wrote: > > > > I'm looking at ways to generate the full "low level" pathname to a > > > file with no logicals, concealed, terminal, or otherwise involved, and= > > > in standard format. I hear that and believe it to be a flawed requirement which will lead bad decision. concealed logical are defined for a reason. Respect that. Specifically you do NOT want to go back down to the physical device... IMHO of course. What if the DKA controller is gonzo but DKB works fine? What if you restore the data to a different place? Maybe to a DGA or DRA? I guess you could restore to $x$DGA1234: [DKA100] and then define a (concealed! :-) logical DKA100 to point to that huh? > > > =A0 =A0 =A0NODE$DKA0:[SYS0.SYSCOMMON.][SYSMGR]LOGIN.COM;12 > > > =A0 =A0 =A0NODE$DKA0:[SYS0.SYSCOMMON.SYSMGR]LOGIN.COM;12 > > Why, pray tell? If you *really* need it this way, just subtract all > > possible combinations of square and angle brackets. Just subtract the "][" if you have to. I uses to be one of the first to shout 'not clean: what about the "<>" directories', trying to show off. Nonsense. nobody uses that, and you know whether your target system uses them. It doesn't. If some application actually does use those, then it deserves to fail.... IMHO of course. > > > I can put in conditional code to clean up the f$parse output but I'd > > > rather find a way to have the clean pathname returned. =A0Is there a w= ay > > > to do this using normal lexical functions? An unconditional - "][" on the same line as the parse, right after it will do fine. > > > One other alternative seems to be to use one of the FID to NAME > > > programs; since I can locate the file in question I have its FID and I don't think that is reasonable. FID_TO_NAME is a 'best effort'. It does not know how the script foudn the file, it only tries to figure out how somene could find the file. To wit: $ dir /file/nohead/notrail/wid=3Dfile=3D40 sys$system:show.exe SYS$COMMON:[SYSEXE]SHOW.EXE;1 (37406,57,0) $ WRITE SYS$OUTPUT F$FID_TO_NAME("SYS$SYSDEVICE","(37406,57,0)") DISK$ALPHASYS:[VMS$COMMON.SYSEXE]SHOW.EXE;1 $ WRITE SYS$OUTPUT F$PARSE(F$FID_TO_NAME("SYS $SYSDEVICE","(37406,57,0)"),,,,"NO_CONCEAL") EISNER$DRA0:[VMS$COMMON.SYSEXE]SHOW.EXE;1 $ WRITE SYS$OUTPUT F$PARSE(f$search("sys $system:show.exe"),,,,"NO_CONCEAL") - "][" EISNER$DRA0:[SYS0.SYSCOMMON.SYSEXE]SHOW.EXE;1 > > > can run it through a program (seen online here in COV, also perhaps on= > > > freeware) and get the full raw pathname back. =A0That would be > > > acceptable too but if anyone's done it and has recommendations I'd > > > appreciate hearing back. =A0It would mean many thousands of image > > > activations running the FID to Name program. I made a program to do just that. Included below. The bigger performance overhead is more likely from back-tracing FID --> header: DID --> directory header.DID ... --> DID =3D (4,4,0) The program is in C, so its activation will be more work than a simple MACRO version. I almost started to recode. However: "Anything not worth doing is not worth doing well" > prospective program that uses the archives may not, especially if its > used or needed by people unfamiliar with VMS filename syntax. I fear you'll end up with something neither the OpenVMS folks, nor the new guard understands. my sample FID_TO_NAME program below. In the context I needed it, I had the FID pieces already. so it takes those as seperate arguments. Wouldn't take too long to make it parse '(x,y,z)'. $fid_name :=3D=3D "$my_tools:fid_to_name " ! ** CHANGE ** : $ file_name=3Df$fid_name(FILE_ID_DEVICE,file_id) ! V8 $!7.3-2 fid_to_name 'FILE_ID_DEVICE' 'file_id_low' 'file_id_mid' 'file_id_hi' file_name Cheers, Hein. /* ** fid_to_name.c Hein van den Heuvel, 2007 ** ** Usage: Define as external DCL command and pass filespec as param. ** Result: DCL Symbol defined with filename as value ** $fid_to_name device index-lo sequence index-hi symbol */ #include #include #include void set_symbol(char *symbol, char *value) { void lib$set_symbol(); struct {int len; char *addr;} symbol_desc, value_desc; symbol_desc.addr =3D symbol; symbol_desc.len =3D strlen(symbol); value_desc.addr =3D value; value_desc.len =3D strlen (value); lib$set_symbol ( &symbol_desc, &value_desc); } main (int argc, char *argv[]) { char device_name[128],file_name[256]; int i, s, lib$fid_to_name(); short fid[4]; struct {int len; char *addr;} device_name_desc, file_name_desc; if (6 !=3D argc) { printf ("Usage: $fid_to_name device index-lo sequence index-hi symbol\n"); printf (" index-lo : fid-word-0, 16 lsb bits for file-id\n"); printf (" index-hi : fid-word-2, 8 msb bits for file-id plus rvn byte\n"); printf (" Symbol : DCL symbol to be defined with resulting filespec\n"); exit (16); } device_name_desc.addr =3D argv[1]; device_name_desc.len =3D strlen(argv[1]); file_name_desc.addr =3D file_name; file_name_desc.len =3D sizeof file_name - 1; fid[0] =3D atoi ( argv[2] ); fid[1] =3D atoi ( argv[3] ); fid[2] =3D atoi ( argv[4] ); s =3D lib$fid_to_name ( &device_name_desc, &fid, &file_name_desc, &fid[3] ); file_name[ fid[3] ] =3D 0; set_symbol ( argv[5], file_name ); } ------------------------------ Date: Sat, 22 Mar 2008 18:34:32 -0700 (PDT) From: Rich Jordan Subject: Re: Hitachi 500G SATA drive in a PWS 600au Message-ID: <7c90233b-a8fe-485d-ac2d-2bd6b6f7b9e5@a1g2000hsb.googlegroups.com> On Mar 21, 6:07 pm, kuhrt.nospa...@encompasserve.org (Marty Kuhrt) wrote: > Hi all, > > My wife works for an online photo company and they use a lot of > disk space. They are currently up to about eight petabytes of > storage. The windfall for me is that they get rid of a bunch of > working disks fairly often so they can upgrade. She just brought > home a Hitachi 500G SATA drive for me to play around with. > > Drooling like Homer Simpson over a doughnut, I thought it would > be neat if I could put this in my PWS 600au. So I ask the group; > is there a PCI SATA HBA available for this machine? If not, is > there a way, with an adapter, I could hang a few of these off one > of my current SCSI HBAs (like a Qlogic ISP1020 SCSI-2)? > > Any suggestions, or working examples, would be appreciated. > > Thanks, > Marty I've never used the Acard SCSI-SATA bridges but I have had no trouble at all running with the Acard UltraSCSI-IDE bridge card on several 7200RPM IDE drives (Seagate and Western Digital 200GB) on my PWS600au. They get one error on startup and then run perfectly for as long as the system stays up. They are a bit slower than native UltraSCSI drives though. ------------------------------ Date: Sat, 22 Mar 2008 20:56:21 GMT From: "PL" Subject: Re: RIP Arthur C. Clarke Message-ID: "Phillip Helbig---remove CLOTHES to reply" skrev i meddelandet news:fs0772$h89$1@online.de... > In article , "PL" > writes: > > > One of my big fans, I got a copy of a Discovery program when he unveils fake > > ghost pictures using > > Clarke was one of YOUR big fans? > Yes, what about it? ^P ------------------------------ Date: Sat, 22 Mar 2008 20:58:53 GMT From: "PL" Subject: Re: RIP Arthur C. Clarke Message-ID: "Phillip Helbig---remove CLOTHES to reply" skrev i meddelandet news:fs0772$h89$1@online.de... > In article , "PL" > writes: > > > One of my big fans, I got a copy of a Discovery program when he unveils fake > > ghost pictures using > > Clarke was one of YOUR big fans? > Ah, sorry, it came out a _bit_ wrong, sorry, I meant the opposite of course ;-) ^P ------------------------------ Date: Sat, 22 Mar 2008 11:17:20 -0700 (PDT) From: Hein RMS van den Heuvel Subject: Re: Too many files in one directory (again) Message-ID: On Mar 22, 9:56=A0am, s...@antinode.org (Steven M. Schweda) wrote: > From: Hein RMS van den Heuvel > > Mark Hopkins whisperred to me > > "1) A directory extends by the maximum of 100 blocks or 1/2 the > > current =A0 file size (I just looked in the code). =A0I'm pretty sure th= at this is > > =A0 rounded up to the next cluster factor > =A0 =A0That sounds good, but as I showed before, I wasn't seeing anything > bigger than that 26384 - 26272 =3D 112, which is the smallest N* C_F >=3D > 100. =A0No sigh of any "1/2 the current file size" here. I thought the description a little odd, so that's why I quoted it. It must be the minimum or 100 and 1/2 current, rounded up to cluster size. > > Steven>> it should also (now) be using 127 > > for the multi-block count ("fil_rab.rab$b_mbc"), and using two : > =A0 =A0The RMS parameter code here (which I use everywhere) will use any > non-default SET RMS parameters set by the user instead of its own > internal defaults. =A0(And it's all open-source.) Super! btw... beware of the plumbing: pipes/paren(thesi)s. Compare $SET RMS/IND/BUF=3D123 $PIPE SHOW RMS =2E.. Process 0 | 123 $ PIPE SHOW RMS | SEARCH SYS$PIPE Pro,"|"/MATCH=3DAND Process 0 | 0 $PIPE ( SHOW RMS ) =2E.. Process 0 | 0 > =A0 =A0For the record: > IT $ pipe t ; vmstar xf zip.tar ; t : > IT::_RTA3: 08:25:11 =A0 (DCL) =A0 CPU=3D11:18:33.07 PF=3D1344 IO=3D225,889= ,468 MEM=3D247 That's a big number. 1189 IOs per file. 75 IOs per second. If we assume that IOs are synchroneous and that the job does not trigger parallel IO, then we should discount for 11 hours CPU time used and come to 100 IO/ second That's not great, but not unreasonable either for a single disk randomly accessed with short IOs. The directory expansion shuffle does NOT explain that number. Best I can tell that should acount for 'only' 170 * 594 =3D 100,000 IO. - number of shuffles =3D 19000/112 =3D 170 (right?) - average IO per shuffle =3D 2 * (19000/2) / 32 =3D 594 (1 read, 1 write, many times. right ?) So pre-allocating would not have helped much/any, other then taking away rampant speculation. The only hint we have to explain the high IO rats is the Dir Data Attempt Rate @ 6700. Apply those 70% hits and that leaves a potential for 2000 IO/sec. Was that a quick worst-case picture, or over minutes ? Unfortunately the MONITOR output sorely lacks in the space. Plenty of room on that screen! T4 would help there. The displayed XFC stats do not help explain that 2000 number back to 75 IO/sec > So, it took only about 35 hours on a (newer, faster) VMS system to do > what took about 35 minutes on the HP-UX system. And that's what hurts and needs to be pointed out. Thank you for the topic! Hein. ------------------------------ Date: Sat, 22 Mar 2008 12:01:03 -0700 (PDT) From: AEF Subject: Re: Too many files in one directory (again) Message-ID: <8daf75d4-5b67-4494-a478-d8912c1e771d@a70g2000hsh.googlegroups.com> On Mar 21, 2:00 pm, JF Mezei wrote: > Richard B. Gilbert wrote: > > Unix does not store the directory entries in ascending alphanumeric > > order! The VMS developers, rightly or wrongly, assumed some degree of > > sanity in the users. > > So when I do an "ls" in Unix, it does an in memory sort of the directory > before listing it ? It must, I would think. > > I guess it was a case of providing better performance for DIR vs CREATE > while Unix provides better performance for CREATE vs DIR Long ago I asked about the point of keeping directory entries in alphabetical order and Carl answered it by saying it speeds up finding a given file in a directory because one can then do a binary search. So it's more than just DIR. Also, it's more than just CREATE: DELETE, RENAME, COPY, etc. AEF ------------------------------ Date: Sat, 22 Mar 2008 15:59:32 -0500 (CDT) From: sms@antinode.org (Steven M. Schweda) Subject: Re: Too many files in one directory (again) Message-ID: <08032215593213_2020CE0A@antinode.org> From: Hein RMS van den Heuvel > I thought the description a little odd, so that's why I quoted it. > It must be the minimum or 100 and 1/2 current, rounded up to cluster > size. That would seem to agree better with reality (if not with optimality). > [...] > So pre-allocating would not have helped much/any, other then taking > away rampant speculation. I did only see that multi-second pause when the directory allocation grew, which was not often enoug to account for much. > The only hint we have to explain the high IO rats is the Dir Data > Attempt Rate @ 6700. > Apply those 70% hits and that leaves a potential for 2000 IO/sec. > > Was that a quick worst-case picture, or over minutes ? > Unfortunately the MONITOR output sorely lacks in the space. > Plenty of room on that screen! T4 would help there. It was a snapshot, but very typical. Whenever I looked at it, it looked pretty much the same. > > So, it took only about 35 hours on a (newer, faster) VMS system to do > > what took about 35 minutes on the HP-UX system. > > And that's what hurts and needs to be pointed out. > Thank you for the topic! Always glad to complain. Now I'll just settle back (hold my breath?) and wait for something to appear in the release notes. SMS. ------------------------------ Date: Sat, 22 Mar 2008 17:38:22 -0400 From: "Richard B. Gilbert" Subject: Re: Too many files in one directory (again) Message-ID: <47E57C4E.2000504@comcast.net> Steven M. Schweda wrote: > From: Hein RMS van den Heuvel > >>I thought the description a little odd, so that's why I quoted it. >>It must be the minimum or 100 and 1/2 current, rounded up to cluster >>size. > > > That would seem to agree better with reality (if not with > optimality). > > >>[...] >>So pre-allocating would not have helped much/any, other then taking >>away rampant speculation. > > > I did only see that multi-second pause when the directory allocation > grew, which was not often enoug to account for much. > > >>The only hint we have to explain the high IO rats is the Dir Data >>Attempt Rate @ 6700. >>Apply those 70% hits and that leaves a potential for 2000 IO/sec. >> >>Was that a quick worst-case picture, or over minutes ? >>Unfortunately the MONITOR output sorely lacks in the space. >>Plenty of room on that screen! T4 would help there. > > > It was a snapshot, but very typical. Whenever I looked at it, it > looked pretty much the same. > > >>>So, it took only about 35 hours on a (newer, faster) VMS system to do >>>what took about 35 minutes on the HP-UX system. >> >>And that's what hurts and needs to be pointed out. >>Thank you for the topic! > > > Always glad to complain. Now I'll just settle back (hold my breath?) > and wait for something to appear in the release notes. > > SMS. No, don't hold your breath. Just don't put too many files in a directory. ISTR that something like 100 blocks is as big as a directory can get and still work reasonably well. The only time I EVER saw a directory bigger than that was when some damned fool went on vacation and left something running that created 70,000 files in a directory; an incident I've mentioned earlier in this thread. If you REALLY NEED to do something like that, use a tool suited to the job. VMS is NOT such a tool!!!! If I were again faced with 70,000 files in a director, I think I would initialize the disk and restore from backup!!!!!!!!!! Yes, it's that bad!! ------------------------------ Date: Sat, 22 Mar 2008 18:11:13 -0400 From: JF Mezei Subject: Re: Too many files in one directory (again) Message-ID: <47e5842a$0$28124$c3e8da3@news.astraweb.com> Richard B. Gilbert wrote: > If you REALLY NEED to do something like that, use a tool suited to the > job. VMS is NOT such a tool!!!! In the early days of VMS, VMS was used in isolated islands. Software was designed for VMS and ran on VMS. People wrote their own software. And during the 1980s, Digital developped a LOT of software/applications that were optimised for VMS. Today however, the owner of VMS has a very tiny portfolio of actively developped applications specific to VMS. New software generally comes from other platforms and must be adapted to run around the restrictions imposed by VMS. The fewer restrictions exsist, the better such software can run on VMS and the easier it would have been to port it to VMS. But lets face it, VMS is NOT KEEPING UP. Porting Mozilla to VMS was a great accomplishement, but they haven't kept up and VMS is way behind now. We can undeerstand why this is, and can't fault the engineers. But the end result is that VMS has lagged the industry with no change in sight that would allow VMS to catch up. ------------------------------ Date: Sat, 22 Mar 2008 16:47:31 -0500 (CDT) From: sms@antinode.org (Steven M. Schweda) Subject: Re: Too many files in one directory (again) Message-ID: <08032216473165_2020CE0A@antinode.org> From: "Richard B. Gilbert" > > Always glad to complain. Now I'll just settle back (hold my breath?) > > and wait for something to appear in the release notes. > No, don't hold your breath. Just don't put too many files in a > directory. ISTR that something like 100 blocks is as big as a directory > can get and still work reasonably well. The only time I EVER saw a > directory bigger than that was when some damned fool went on vacation > and left something running that created 70,000 files in a directory; an > incident I've mentioned earlier in this thread. > If you REALLY NEED to do something like that, use a tool suited to the > job. VMS is NOT such a tool!!!! VMS V8.3 isn't. Why shouldn't a later version be able to do better? This reminds me of all that "Zip -V archives are _supposed_ to be useless on non-VMS systems" advice I got in 2004. Sometimes it's practical to improve (fix?) the tool (making the world a better place for all). > If I were again faced with 70,000 files in a director, I think I would > initialize the disk and restore from backup!!!!!!!!!! Yes, it's that bad!! As I said, DFU DELETE cleaned out about 50k files in a few minutes. But hey. You're welcome to complain about the things which bother you. ------------------------------------------------------------------------ Steven M. Schweda sms@antinode-org 382 South Warwick Street (+1) 651-699-9818 Saint Paul MN 55105-2547 ------------------------------ Date: 22 Mar 2008 16:14:08 -0600 From: Kilgallen@SpamCop.net (Larry Kilgallen) Subject: Re: Tripwire on OpenVMS Best Practices? Message-ID: <1cHRMh4yPH82@eisner.encompasserve.org> In article <75da0787-9aa4-4eef-96eb-8d9ae220f1cc@s8g2000prg.googlegroups.com>, rfc2307@gmail.com writes: > Good Morning, I'm looking for any information on best practices on > what to monitor, that is Operating System specific on the OpenVMS > platform. Any information or lessons learned would be appreciated. Your use of the term "Tripwire" in the title is not clear, but for the body of your post, consider: http://www.ljk.com/ljk/checklists.html > Please reply to rfc2307@gmail.com Ask here, get an answer here. ------------------------------ End of INFO-VAX 2008.165 ************************