Fri Feb 21 13:02:09 1992 Message : #2835701 From: Steve Summit Address : scs@adam.mit.edu Group : NETCOMP.FolkLore Length : 713 words Subject : Re: Adminstration Horror Stories Msg-ID: <1992Feb21.194543.10478@athena.mit.edu> Posted: 21 Feb 92 19:45:43 GMT Org. : none, at the moment [enough "can you top this /unix compression ratio?" already...] A friend was once visiting the University of Hawaii and was given a temporary account on a Unix machine for the duration of his stay. While acclimating himself to the machine, he chanced to notice several of those charming little ::0:0::: turds in /etc/passwd, and dutifully reported them to the system manager, who blew up, insisting that those lines were in the password file for good and technical reasons, and that users should NOT be poking around in that file. Joe: "But it's world readable!" Admin: "I though I could trust users not to meddle where they're not supposed to. I don't have time to check the permissions on every system file. It was probably distributed that way by the incompetent people who wrote Unix in the first place." Joe: "Maybe you should fix it." (Smirk evidently too subtle for this blowhard to notice.) Admin: "You're think you're so smart, I will!" He had no sooner typed chmod 600 /etc/passwd than other users started coming in the door, wondering why ls -l had suddenly started printing numbers instead of user names... [Disclaimer: this is the way it's the most fun to tell the story, but since I wasn't there, I can't remember if the permissions actually got changed, and if so, if Joe had really goaded the guy into it.] Here's one I know is true: As an undergraduate, I didn't always have time for the more troublesome tasks that occasionally cropped up in the maintenance of this system I worked on (the same one with the "popcorn" disk drives). One day I finally had some free time, and sat down to: 1. rebuild the bootable removable pack I kept around in case of emergencies, which had stopped working, and 2. reformat the disk containing the root partition, which had a few bad blocks which hadn't gotten serious (yet). I didn't really feel like rebuilding the emergency boot pack, because it was a real pain, and at about the limits of my abilities at the time. (I had spent several days learning how to build it at all: compiling a kernel with different root, swap, and pipe devices; finding and assembling a boot block for the removable drive; etc.) So I figured, what the heck, I'll perform task 2 first. The low-level reformat takes about ten minutes. Five minutes later, while wandering around the room waiting for it to finish, my mind finally thought about the precedence which was lurking in what I had been absent-mindedly assuming to be two unrelated tasks. ("The word `bulldozer' wandered through Arthur's mind for a moment in search of something to connect with.") I didn't even make a mad dash for the halt switch; it was, of course, far too late. The really galling thing was that my whole reason for building the bootable removable pack in the first place had been my realization, seconds before initiating a previous reformat of drive 0, that the backup tapes I had made wouldn't do me much good without a bootable system with which to read them back on to drive 0. I learned a lot more over the next day or so: how to load in a system from the (damaged) 2.8bsd distribution tape, how to toggle in a bootstrap loader from the front panel, how to deal with the fact that the distribution kernel used different partition sizes than we did, why it was a bad idea to use non-default partition sizes... The next day I learned how to order a DEC magtape boot prom. (It didn't arrive until after I left. Years later, when I came back to help move the system to Princeton, there was still this little envelope kicking around with a prom in it which nobody knew what was for...) * * * Does anyone collect these stupid user/administrator/field service person stories? They're the best part of this newsgroup. Steve Summit scs@adam.mit.edu Thu Oct 15 14:04:17 1992 Message : #4671628 From: Erkka Sutinen Address : eps@siivu.oulu.fi Group : Usenet.alt.folklore.computers Length : 1073 words Subject : Re: WANTED: Unix administration horror stories ! Msg-ID: <1992Oct15.183747.24196@ousrvr.oulu.fi> Posted: Thu, 15 Oct 1992 18:37:4 Org. : University of Oulu In article <1992Oct12.080944.23519@jet.uk> djs@jet.uk (David J Stevenson) writes: > When this happened to a colleague (when I worked somewhere else) he restored > vmunix by copying from another machine. Unfortunately, a 68000 kernel does > not run very well on a Sparc... Uh. This reminds me of an long and hard day couple of weeks ago..... I was walking merrily towards my room while world was beautiful and sun was shining. I got the first hint that something may be wrong when I saw our sysadmin banging his head on a nearby door. I asked if something might be wrong, and he told me so while continuing banging. He had made backups of our old faithful, tolsun (sun3/160) and made a minor typo. Instead of tar cf /dev/rst0, he had written tar xf /dev/rst0. (Scripts?? We don't use any bleeding scripts!! They restrict creativity and make improvising impossible!) Oh well. And the tape happened to be an old backup from a sparc. So. No binaries worked, execpt that one could login as box. inetd had crashed (at which point it did is irrelevant). There was no active sessions for that system and there was no way to get in..... Lets take a break, have some coffee and think it over. There was a lighter side: all of the disks were mounted to another system via nfs, and that daemon was still working. List of the files overwritten was on a log file, and there wasn'y very many of them. Backups were on 8mm tapes and our only 8mm drive was on our server, but with nfs, that wasn't a problem. On the other hand we had lost /usr/bin, /usr/etc, /usr/kvm and /ucb. Ugh! I think I don't like this. Fine. Lets take everything back from tape.... Wait a minute... We had just installed new operating system, which had taken several days since additional upgrade hadn't worked due to lack of disk space and we had a rather unorthodox system running, which worked fine with our add-ons, such as appletalk box... And we hadn't make backups of this new system yet... Aww shit. We couldn't afford to lose this system, it had too much sweat already in it. Fortunately we had all the binaries of the upgraded system on our server's disks, and with nfs we copied everything to the faulty system. Blessed is the network, who rules our lives! Nothing happened. Nothing worked. Ah! We just have to restart inetd. Hmmmm. But how? Do not worry, this is Unix, full of possibilities! First cron, but the clock of that system was totally out of this world showing probably the time of Ouagadougou and we didn't know where was that! Ah! making some false login attempts will bring error messages to the console. Nothing happened... I guess syslog had died too.... When daemons are in agony, no program can stand without feeling some sympathy towards those who suffer so.... Oh no... But of course. Removing root's password.... Why are your brain cells sometimes so slow.... This made it.... But still..... Nothing worked... Ah, we had the upgraded binaries... Hm. Ahh. the sun's unupgrade option was used while upgrading the system, so we just reunupgraded the system. No problem. Everything is fine again... ?? Why doesn't that system work?? Why our server doesn't let anyone in... Awwww...Jesus Christ. What had I done!!! Unupgraded our SparcServer with sun3's binaries.... Awww shit and . At this point I was trying to find an good way to get rid of all of our computers.... Would anyone notice if I just dumped our systems to river and claimed that we have never had any computers.... No hope. Back to work. And at the same moment, the cron miraculously worked, and we had two inetd's running. No harm done. It is just a nuisance. No problem. Don't panic. Both of the systems are up and running.... Well aware of the fact that if there are any more mistakes, the systems will crash and they will not come up. We didn't even have our server's operating system handy (one disk in whole unversity...). I took the most recent backup of our server and carefully extracted the rpc.passwdauth and couple of other files from the tape, and lo... everything worked again. The reunupgrade procedure was made at this time at the right computer, and at our suprise it started to work again.... The world is a miraculous place......While ps again operational, we checked the daemons still running.... nfsd and getty were only systems that had stayed up in addition to the usual bunch (init etc...) At this point we thought everything was fine... Let's make the backup now and take a good care not to touch anything..... It worked.... Couple of days later I noticed that the restoring of our server's daemons had gone to a wrong directory and the mail hadn't been running for two days... No problem... Execpt that delivering two days worth of mail brought our server to its knees... Fortunately it didn't crash but it was ssslllllooooowwwwww for couple of hours.... .....and once again our heroes had defeated evil nazis and were heading home for meal....... Moral of the story: DON'T PANIC. There are several ways to handle unix as long as it is running. Do NOT try to reboot it. It only brings sorrow with it. Moral of the story 2: Double-check every command you give. If one of your commands has an error, it is the one which causes biggest damage. Moral of the story 3: Check the machine you are in, before you give any even possibly destructive commands... Networks can be a real nuisance you know..... -- ======================================#===================================== == Erkka Pietari Sutinen #Eke what availeth Maner and Gentlinesse eps@rieska.oulu.fi / sutinen@csc.fi # Without yow, benygne creature? University of Oulu. Finland # Shal Cruelte be your governesse? Dep. of Information Prosessing Science# -Chaucher Fri Oct 16 07:02:39 1992 Message : #4679040 From: Anthony DeBoer Address : adb@geac.com Group : Usenet.alt.folklore.computers Length : 211 words Subject : Re: WANTED: Unix administration horror stories ! Msg-ID: <1992Oct16.133943.24670@geac.com> Posted: Fri, 16 Oct 1992 13:39:4 Org. : Geac Computer Corporation In article <1992Oct10.010412.3448@waggen.twuug.com> broberts@waggen.twuug.com (Bill Roberts) writes: >My most interesting in the reguard was when I deleted "/dev/null". Of >course it was soon recreated as a "regular file", then permission problems >started to show up. I was once called in to save a system where most things worked, but the main application package being used on it hung the moment you entered it (leaving the system more than a little useless for getting things done). I poked around for awhile, verified that the application's files were all present, undamaged, and had the right permissions. The folks who normally used the machine had also discovered that all was well if root tried to run it. But nothing was visibly wrong anywhere. So, being a bit hungry by then, I took a break for supper, and about halfway through, the little voice at the back of my head that sometimes helps me said, "/dev/tty". Sure enough, somebody had chmod'ded it to 0644, and the application directed (or tried to direct, in this case) all its I/O through it rather than just using stdin/stdout like a sane normal process. -- Anthony DeBoer adb@geac.com | uunet!geac!adb | GEM: ANTHONY.DEBOER Fri Oct 16 20:02:59 1992 Message : #4685077 From: Ian Chard Address : chardi@p4.cs.man.ac.uk Group : Usenet.alt.folklore.computers Length : 413 words Subject : Re: WANTED: Unix administration horror stories ! Msg-ID: <1992Oct16.140521@p4.cs.man.ac.uk> Posted: 16 Oct 92 13:05:21 GMT Org. : Department of Computer Science, University of Manchester I had the dubious honour of administrating a 3B2/300. No manual pages, no nothing. Lovely. Anyway, back to the story. The company in question _survives_ because it has a mailing list with x thousand names and addresses (where x is a constant the value of which I have forgotten). The very nasty DBMS being used can remain anonymous as I don't really want to get sued into the Earth's core. Suffice it to say that there was no direct access to the database other than building a form and some code, and someone had asked me to delete one record (a previous sysadm had removed the "delete" option from the user form, along with all the source code, just before he walked out :-) ). I set about writing a very short program in the DBMS's 4GL which looked something like: use file 'fmlist' if c_name = 'Foo Inc' then delete; ...or similar. You get the idea. Unfortunately the DBMS didn't, and set about deleting every record. Thirty seconds later (the box had 16 terminals; our maintenance people recommended no more than 6) I realised that something was wrong and interrupted it. A quick query revealed that the world no longer contained anyone whose name starts with A or B. Damn. Oh God, damn, damn, erm, oh, damn, {face turns white} etc. I reach for the backup tapes. Oh God, damn, no, please, not that. The tapes were there. Unfortunately they were not numbered, and the somewhat hacked-together backup software we were using core dumped if it came across a tape that was out of sequence. Now there were six tapes, giving a total of... oh GOD! combinations. The users are beginning to jump up and down, asking why "routine maintenance" (all I could think of under pressure) was taking so damn long, and why I was looking so pale. Finally I hit the right combination at 2:15am the following morning. I thought I never wanted to see another backup tape ever again. And then, six weeks later, the disk crashed. I changed my mind very quickly. Ian. -- [ Ian Chard, Systems Integration | "Kryten, unpack Rachel and get out the ] [ University of Manchester, UK | puncture repair kit." ] [ Email: chardi@cs.man.ac.uk | -- Rimmer (un-H), Red Dwarf ] Fri Oct 16 21:03:39 1992 Message : #4685612 From: Rob Slade Address : rslade@cue.bc.ca Group : Usenet.alt.folklore.computers Length : 245 words Subject : Re: WANTED: Unix administration horror stories ! Msg-ID: <1992Oct17.000456.18891@softwords.bc.ca> Posted: Sat, 17 Oct 92 00:04:56 Org. : Computer Using Educators of B.C., Canada Hope this fits. I had a job one time teaching Pascal at a "visa school". The machine was a multi-user micro that ran UNIX. I have enough stories from that one course to keep a group of computer educators in stitches for at least half an hour. The finale of the course was on the last day of classes. When I showed up and powered up the system, it refused to boot. Since all the students' term projects and papers were in the computer, it was fairly important. After a few hours of work, and consultation with the other teacher, who did the sysadmin and maintenance, we were finally informed that the new admin assistant around the place had decided that the layout of the computer lab was unsuitable. (I had noticed that all the desk were repositioned: I thought the other teacher had done it, he thought I had.) The AA had, the night before, moved all the furniture, including the terminals and the micro. She did not know anything about parking hard disks. We knew now, that we were in trouble, but we didn't realize how much until we started reading up on emergency procedures. For some unknown reason, booting the micro from the original system disks would automatically reformat the hard disk. (The visa school refunded the tuition for all the students in that course.) Mon Nov 2 22:02:51 1992 Message : #4845067 From: Lars Peter Fischer Address : fischer@iesd.auc.dk Group : Usenet.alt.folklore.computers Length : 122 words Subject : Re: WANTED: Unix administration horror stories ! Msg-ID: Posted: 3 Nov 92 01:15:33 Org. : Mathematics and Computer Science, Aalborg University [ Our news server has been badly broken, so I'm resending this. Sorry if you've seen it before. ] >>>>> "Alan" == Alan Saunders (alan@spuddy.uucp) Alan> On his return, he decided to put what he had learned Alan> into practice, and changed the ownership of all files in /bin, /usr/bin Alan> to bin.bin! In a related move, a new computing center sysadmin here once decided that there was no reason for people to be able to copy system commands, so he did a "chmod a-r /bin/* /usr/bin/*". /Lars -- Lars Fischer, fischer@iesd.auc.dk | It takes an uncommon mind to think of CS Dept., Aalborg Univ., DENMARK. | these things. -- Calvin Thu Jun 17 20:03:59 1993 Message : #7333870 From: Doug Siebert Address : dsiebert@icaen.uiowa.edu Group : Usenet.alt.folklore.computers Length : 415 words Subject : Re: Worst Bugs (was Re: About the Internet Worm Msg-ID: <1993Jun18.015647.13167@icaen.uiowa.edu> Posted: Fri, 18 Jun 1993 01:56:4 Org. : Iowa Computer Aided Engineering Network, University of Iowa S947460@UMSLVMA.UMSL.EDU (Andrew Zaitsev) writes: >When I was experimenting with fork() for the first time, I managed >to (mistakenly) write a program that spawns a child and then dies. >After that child will spawn grandchild and die, etc. > >I ended up with an infinite process with constantly changing PID. >I couldn't kill it, because by the time I entered 'kill -9', >the PID I specified would be already non-existent. > >So, I wrote 'hunter' program that would track down that infinite process >and kill it. No luck again - it was spawning too fast, 'hunter' couldn't >get it. > >Then I gave up. I hopelessly looked up at my directory, saw 'a.out' >file in there, thought 'why would I need it now?' and deleted it. >Tada! Infinite process has stopped - it was reading a.out every time >fork() was executed! I did something similar but even stupider....I created a program which by accident did the fork/die thing moving through the process table, and I quickly hacked up a 'killer' program to track backwards through the pids and kill it for me. I got extra-clever and, since I was on a system running HP-UX where I had real-time processes available to me and could become root, and ran the killer in real-time. When it failed to kill the program I was mystified, but eventually figured out I had filled the process table with zombies and so I couldn't fork anymore so the chain was finally broken. Here comes the stupider part: I made a fix to my program to correct the problem that had made it fork endlessly, compiled it, ran....the OLD version by mistake. Unfortunately I did so from my shell that had real-time priority so now I had the same thing happening in real-time priority! Needless to say, the machine locked up tight as a drum! Luckily the process table again filled up and I was able to delete the evil offending program and run the fixed version which did in fact behave as I wanted :-) -- Doug Siebert | "I don't have to take this abuse Internet: dsiebert@isca.uiowa.edu | from you - I've got hundreds of NeXTMail: dsiebert@chop.isca.uiowa.edu | people waiting in line to abuse ICBM: 41d 39m 55s N, 91d 30m 43s W | me!" Bill Murray, Ghostbusters Tue Sep 21 18:52:31 1993 Message : #8575086 From: Wayne Hathaway Address : wayne@Auspex.COM Group : Usenet.alt.folklore.computers Length : 124 words Subject : Re: The Programmer's Handy Guide to the Languages (humor) Msg-ID: <18657@auspex-gw.auspex.com> Posted: 18 Sep 93 22:27:27 GMT Org. : Auspex Systems, Santa Clara > >Unix: > >% ls > >foot.c foot.h foot.o toe.c toe.o > >% rm *.o > > That should be: > %rm * .o > which is how most Unix neophytes manage to shoot themselves in the foot. Actually I almost never introduce a space like that; too unnatural. What I *DO*, however, is leave my finger on the shift key just a hair too long, so that the period is shifted along with the asterisk. This of course produces %rm *>o Pretty much the same joyful effect. :-( Wayne Hathaway Internet: wayne@auspex.com Auspex Systems Phone: 408-986-2044 5200 Great America Parkway FAX: 408-986-2020 Santa Clara, CA 95054 Tue Sep 21 18:54:08 1993 Message : #8575187 From: William Gladnick Address : wglad@nexus6.org Group : Usenet.alt.folklore.computers Length : 181 words Subject : Re: The Programmer's Handy Guide to the Languages (humor) Msg-ID: <1993Sep18.125403.16257@nexus6.org> Posted: Sat, 18 Sep 1993 12:54:0 Org. : Nexus 6 - New Orleans Public Access Unix Dino Dini (amiga@hosta.dircon.co.uk) wrote: : In <27bj41$287@nwfocus.wa.com> David Ruggiero writes: : >Unix: : >% ls : >foot.c foot.h foot.o toe.c toe.o : >% rm *.o : >rm:.o no such file or directory : >% ls : >% : Should have been : % ls : foot.c foot.h foot.o toe.c toe.o : % rm * .o : rm:.o no such file or directory : % ls : % : It is very easy to shoot yourself in the foot with UNIX. : Fortunately, it is also very easy to make a new one :) : -- : ---------------------------------------------------------------------------- - : Dino Dini Dini & Dini Productions amiga@hosta.dircon.co.uk : "The opinions expressed here *are* those of the company... I am the company." : ---------------------------------------------------------------------------- - Or how about: # pwd / # rm * .o rm: .o: No such file or directory # ls ls: command not found # -- ______ William Gladnick wglad@nexus6.org Thu Sep 30 07:05:24 1993 Message : #8753942 From: Patrick Mann Address : mann@munich.ixos.de Group : Usenet.alt.folklore.computers Length : 32 words Subject : Re: The Programmer's Handy Guide to the Languages (humor) Msg-ID: <28elkq$45q@munich.ixos.de> Posted: 30 Sep 1993 14:06:34 +01 Org. : iXOS Software GmbH, Munich, Germany Here's another interesting twist I was privileged to experience: rm -rf when executed on an auto-mounter mount point blithely goes off to rm all of the mounted directories... Patrick.