From: Ben@lspace.org (Ben) Subject: PFY: (7) Do not touch the machines Date: Tue, 03 Oct 2000 17:42:14 GMT The phone rings. This isn't a problem in itself as you might have normally expected. I pick it up. Perhaps in retrospect I should have gone out for a night of hard drinking. Either way I would have ended up the same in the morning; tired, unshaven and smelling like a yak's underarm. I'm at home. So, there I am at home on a Friday night. I've taken home one of the new laptops and am happily installing my little collection of 'special applications' including such gems as "URL Logger", "Dialup Details Logger" and my personal favourite, the "Low Bandwidth Desktop Snoop", useful when you've set their 56k modem to only work at 28.8. I like to make sure the phone's ring sounds nothing like the one at work. It stops me from getting nervous palpatations on the occasions I don't manage to divert my work line to the sex number. I don't think I'm the only person who enjoys the break from having to deal with tech problems. There have been a number of people who have called more than once to find out just where exactly their dongles fit in, or whether their hardware is compatible. It's sad really, but no-one's complained. I pick up the phone and wait for the person to identify themselves. People who ring me at home people understand what I do during daylight hours and why I don't confirm if someone's rung the right number till I know who they are and why they're calling. I once tried to never give out my home number out to people connect with me at work, but someone had the idea of making us fill out 'emergency contact details' forms. Naturally, being aware of the dangers of working with lusers all the time, I felt it might be in my best intrests to make sure someone could get in contact with a certain someone who knew how to disarm the traps on backup systems and stop the deadman's switch on the ring mains at my three previous jobs in case I actually pull through whatever horrors the lusers have inflicted on me. "Hello?" "Hello." I'm giving nothing away. "Is that you?" They're good. But I'm better. "Who?" "Oh it is you, I recognise the voice. This is the Admin Office." Damn. "We need you to come in. Now." The first answer that comes to mind is unspeakable, let alone unprintable. There isn't a character-set in existance that supports the kind of thoughts that have come unbidden to mind. "... Why?" I managed to choke out. It's Friday. Admitedly it's only 15:00, but given the hours I've put in this last week alone I thought it might be nice to stay late for once, not extend lunch until 17:00. "The server is telling people that there are S-C-S-I errors and now anything we save we can't get back. We've saved five or six important things now and we can't get them back. We were wondering what you could do because if we save anything else we'll be in trouble." I actually offer some advice. "Stop saving things to the server for the moment then?" I suggest. "Would that work?" she enquires hopefully. "It might just." I sigh, "I'll be in in a while." ... Around 16:30 I get into work. Most of the people have gone home for the weekend, which is good. I like to do my magic alone. I notice that once again the bike racks are full and that only the disabled ramp has available locking points. Being a Bastard, but not a complete bastard just yet, I decide that for once I'll not lock the bike over the end of the disabled ramp, (usually necessitating the occasional glance from the window to see if the Custodian is removing my locks with an angle-grinder) and instead I simply roll it through the freshly setting concrete a few times to expose the network cables into the building. Given the serious amounts of Cat-5E and 6 I scored off the installation of ethernet in the library I think I should be able to get the spare cameras I have connected to it in a matter of a few hours, giving me 100% coverage of the outside of the building in case I should have any unwanted visitors. With this cheery thought in mind I slide a few leaves over the grooves, lock the bike to something solid and wander into the building. The lift's on the ground floor so I come over all lazy and jump in. This lift is old and rickety and isn't even modern enough to have even a voice circuit in it. This hasn't stopped me though. I've installed a couple of relays on some of the more important signal wires and run a few wires up to the small linux box at the top of the shaft that runs the webcam which points north, towards the women's college. The RS232 interface is a wonderful thing. Bzzzzzzz-chack! Fourth floor. I wander out and typically find the offices empty. Amble into my room (having disabled the door knob current with an old Sony remote) and check to see if the technician's left her machine insecure. I find that doing things to her machine tends to be the best way to keep her on her toes and up to date with all the breaking security issues. I spend the regulation twenty seconds checking my machine for tampering and review the webcam trained on the machine to see if the technician's done anything. Looks like she's been too busy to try stuff. Jolly good. I log back in and try the nifty little port hack mentioned on the tech channel I frequent. Damn, she's closed her machine to it already. I power off her machine and then rapidly cycle it a few times. Maybe she'll learn to disconnect the power button and padlock her case. I wander back over to my machine and take a look at the server. Everyone seem to have left their files open. This includes the pay/assessment review files. The ones kept passworded and on the standalone machine. Looks like someone forgot that they're saved to Central, rather than my servers where I can get at them. It would be totally unethical and wrong for me to look at them, nevermind the fact that I could do irreperable damage to people's long term careers. I save a copy, on general principles. On the other hand, one can never tell when Central services might suffer a sudden, catastrophic, irrevokably damaging data-loss incident due to, say, a distributed denial of service attack during a maintenance and transaction cycle. You'd have to know that the vulnerable period was between 02:00 and 04:00 on a Sunday morning, and what exactly the buffer overflow was in the software was... The odds of getting exactly the right time would be staggering. Anyway, there are strange things afoot at the Circle-BOFH. Seems someone's been buggering about with the disks on the server, resulting in SCSI errors. I power down the server (over 278 days of uptime lost, someone will pay) and open the caddy box. Not only has one of the disks been dislodged in its caddy. The disk marked 'spare' is no long spare, it's gone. "!" sez I. and "?". After a few speechless seconds I come out with "%^&*". Unless I come up with something plausible soon I'm going to run out of special characters. There's no way the technician would have done this. Not only would she do a better job of hiding the loss with spares, she'd have told me as soon as possible as she values her job, and life. Who would have done this? I take closer look at the machine's casing. There's a boot mark. Someone has been kicking my servers. MY SERVERS. Goddamnit. I check with the technician using the phone number taped under her desk. Nothing untoward has happened while she was at work, but she did have to fix someone's printer and left the door unlocked for longer than normally advised (I chastise her over the phone by letting her listen to the sound of me writing with a squeaky indelible marker on the magnetic surface of her backup floppies). This oversight means it could have been anyone, including outsiders. I grep the logs for anything suspicious. One person has had a sudden increase in free space. I check his .bash_history and logs, there's a powerdown and reboot followed by "pico /etc/fstab" (no lie) which looks promising. I have motive and opportunity. Naturally I can't prove anything, yet. I fire up some mp3s while I think. The webcam's not caught anything as the servers are out of shot. Fuckit; he's to blame, he gets to pay. Random Violence without Proof 101, if you can't prove it straight away, do it anyway. I pop downstairs with the comms room key and switch him over to a small 10-base 5-port hub with an intermittent collision fault. I also reroute his telephone. From the recesses of the comms rack I pull out a black cable with an RJ-45 on one end and a power plug on the other. An etherkiller. I'm prepared to sacrifice one run of cat-5 and a PC, or at least, parts thereof. I've never actually used an etherkiller. This presents the perfect opportunity. Especially as we've just had a 440V spur installed. I set a small domestic timer switch and retreat. I connect up the PC after opening the case and confirm that, yes, there's a new SCSI disk in there. I make sure the thing is as isolated from the rest of the net as possible and wander back upstairs to repair the damage to the server and its files. Kicking back for thirty minutes of Unreal Tournament I then consider what else I can prepare for the next day's offensive against Mr Clumsy Boots. ... Next Monday morning I'm in at the crack of 11:59. Bleary-eyed and requiring coffee I stare at the technician long enough to get her to reveal everything she's done to my computer, and apparently the chair, too. Sometimes it's good to have not so much a 'thousand meter stare' but an 'I can see your P45' variation of same. I let her in on the results of the server foobar and do a quick Q&A on what she would have done in my place. There aren't many differences, except she would have seeded his files with abusive words (her own perl script) and tacked on some imaginative header fields to his email and news posts. I approve and fill in another little box on the "road to BOFH-hood" wall planner. I fire up a little application which watches the load on the 440V spur and wait. At around 12:30 there's a mometary rise which triggers a .wav of maniacal laughter as performed by Vincent Price. Down the stairs I hear a pop and a loud shriek muffled by distance. A few heartbeat calming minutes later the phone rings. The technician answers. "Hardware support and maintenance," officially one of our titles, really. "What have you broken?" "Erm, I was using my PC and the network wasn't working. I saved some files to my disk before coming up to get you to fix it and suddenly everything went bang. My monitor has gone black and there's smoke coming out of my case. I really need to get my report finished for three this afternoon or we lose out on the project. Come and fix it, or give me a new PC." I listen on the speakerphone and decide to bait the hook. "Have you done anything to your PC recently?" "No..." "... Because this sounds like an incorrectly connected hard drive situation." I continue smoothly. "When someone untrained adds a SCSI hard disk to a machine, they sometimes forget that connecting a Low Voltage Differential drive to an older HVD SCSI chain can result in an uncontainable backing up of voltage which, in addition to generating a massive EMP pulse which can wipe whole sections of a magnetic media device..." I pause, letting that line of bullshit sink in, "can also mean that large current surges ground out in other components in direct contact with that bus. I wouldn't be suprised if your network card is half-melted. No, don't touch anything. We'll be right down." I hang up. The possibilities at this point are almost infinite. It's dizzying. We leave the room, and head on down. He hasn't got a chance.