BOFHJournal - March 2005

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

February's Journal
April's Journal

31/03/2005 [17:20] I really should have gone home by now. However Elaine's going home by bus so I have a little while before she turns up, I think. One of the developers I'm building machines for gave me one back today to incorporate all the changes he's made into the build. This has taken slightly longer than expected due some oddnesses concerning Java, the UPS module I've added to the build and the firewall here which makes adding certain Perl modules which require connectivity a little harder to deal with. Still, I think I've got it all sorted now. I was going to re-kickstart it today but RedHat Networks was down and what with one thing and another I just haven't had time. Still, tomorrow should be good as I've got this machine to do in the morning and then hopefully another developer coming to see me to get the last of his stuff of that other server I'm going to be nuking. Tonight is Thursday, with all that entails, so I should head off and be home before Elaine is. 30/03/2005 [13:00] Although we went to bed fairly early last night we were both shattered this morning. I decided not to run to work as I needed to get in in good time and be able to deal with any issues which had arisen since I left on Thursday evening. Luckily nothing seems to have gone wrong and I think the UPS thing has sorted itself out, at least for the moment. Most of the morning has been spent sorting through email, doing NTP server updates and scheduling up2date on all the RHEL boxes I administer. I may get around to doing some actual planned work this afternoon. [16:10] Progress. I've put in motion the procedures which should cause the Windows administrators to call me when they're ready to install the new Java clients for talking to the MGE UPS. Hopefully they'll do this in the next day or so. I won't be able to get the client on to the numerous Sun boxes we have here until I've talked with the Solaris administrators here who know more about the specific Java versions we're required to run here. 29/03/2005 [30/03/2005 - 12:30] After one of the coldest nights in a while we stripped most of the insides of the tent out and packed them into the car. After heading to breakfast at the Longlands Cafe again (we go there every morning when in the area) and making some further purchases we drove over Winnt's Pass to the Chestnut Centre and spent an enjoyable few hours looking at owls, otters and other such creatures. Not quite the weekend we were after but enjoyable nevertheless. We're definitely going to go back (and elsewhere) with that tent as it's perfect for four people. Being able to stand up while camping and under cover is an experience I've not had since I was in niger tents in the Scouts. Unfortunately we had one casualty of the trip. Elaine woke on the Sunday morning and managed to wrench her neck badly because of the cold and sleeping in an odd position. After a trip to A&E in Sheffield that day we ascertained that there wasn't anything serious wrong but it was going to be painful for a while. Cue me forgetting, pointing things out and Elaine turning to see them and then shouting in pain. Back at the camp site after the Chestnut Centre we took down the rest of the tent, wiped all the mud off the bottom and drove home. I have to say that we were all extremely disappointed not to be able to climb at all. The walks we took at places like Mam Tor and Hope were extremely enjoyable and allowed us to stretch our legs, but weren't what we'd gone there for. I have a feeling that the next couple of months are going to see a lot better weather so we'll have a fair number of opportunities to introduce Elaine to the perils of proper climbing. 28/03/2005 [30/03/2005 - 12:00] Cormac and Steph got caught on sandbanks returning from releasing seals on Saturday. Consequently we didn't actually leave until well gone 17:00. As a result we arrived at Laneside campsite in the pitch black. Setting up a large tent for the first time in the dark when it's spitting and the ground is extremely muddy isn't a huge amount of fun, but we managed it and did it well. Unfortunately, the entire weekend consisted of rain most of the time so the rocks were unclimbable except by the terminally brave. We opted for some long walks and retail therapy in 'Outside' in Hathersage. 25/03/2005 [30/03/2005 - 11:40] For most of the morning I stayed in bed and tried to catch up on some sleep. After a while though I decided to get out of the house so cycled into town to get Elaine a sleeping mat for the weekend's camping. Other than that we did very little other than watch the weather reports. 23/03/2005 [24/03/2005 - 14:50] The day was kind of in two halves. The morning was spent trying to work out how to get the last developer and his mass of project files over from one server to the other. This currently doesn't seem to have a happy ending, but I'm hopeful. For the afternoon I was asked to return to my task of getting the MGE UPS to talk to all the servers in the this server room. We have thirty-nine servers in there (Solaris, Linux and Windows) and we need them all to listen to the XML Web Card thingy in the UPS. This requires a) firewall ports to be opened, b) software to be found on MGE's site, c) custom installers to be made which use SSL rather than plain HTTP to authenticate and d) the UPS not to throw a major wobbly when it realises that it's suddenly being asked to furnish SSL rather than unencrypted traffic. This is what happened as I was waiting for Elaine to turn up and help me home with the new tent which was delivered to work during the day. Luckily, as I was leaving the web interface for the thing seemed to starting to respond again. When I got home, before Cormac came to pick us up for climbing, I emailed my work colleagues, let them know the score and how to disable the new clients I'd installed if there were any problems. Climbing was pretty good although we all looked a little wiped out. I have tomorrow until Tuesday evening off, and I intend to relax, tidy the house and then go climbing in the Peaks between now and then. Photos to come plus some kind of climbing report when I get back. Promise. 22/03/2005 [16:35] Turns out that you can't give blood if you have a mouth ulcer. I only found this out when I had waited for the hour break the nurses were taking, the queue to move forward and for me to get my interview with a nurse. If I hadn't mentioned it I probably would have a small hole in my arm right now, but it seems that an ulcer could be virus related and viruses end up in the blood as it were. Still, I got to see a huge mechanical arm thing tearing down a car park next to the church I was attempting to donate at. Oh, something I forgot to mention yesterday is that Elaine won a bottle of whisky at the show on Saturday. I'm sure she/we will get around to drinking it some time soon. In other news I made an absolutely wonderful risotto last night. Pancetta and Borlotti beans with some shallots, garlic and a splash of vermouth in with the rice to give it some more flavour. Very nice. I think Elaine was only slightly more surprised than me at how it turned out. I'm looking forward to doing the next experiment next week. Of course it required a cycle to Tesco all the way from home, but that was part of the fun. 21/03/2005 [12:15] We bought a tent. OK, we bought a tent between four people. This made it very cheap and the chances are it'll only ever be used by the four of us at the same time. The show itself was huge. Lots of interesting things. Elaine bought some new rock boots. Come 15:30 of course Cormac was glued to the large plasma screen in the Wales Tourism Board area (which for some unlikely reason was right next to the Ireland stand) for the rugby. This seemed to draw probably the largest crowd next to the British Bouldering Championships. On the way back we got some shopping in and watched The Incredibles before Cormac sloped off without most of his stuff. Sunday was spent mainly getting housework done and playing FarCry (now that my new DVD drive has arrived). We did go to see Constantine at around 16:00 though, which rounded off the afternoon nicely. When we got home Elaine was interested in watching Calendar Girls, and against my better judgement I decided to watch it too. It's a wonderful film. As soon as I got in this morning I had to order a taxi to take me and a Sun engineer to another site to install a new fan tray in an E250. One of the fans had completely siezed it turns out. Bringing the databases and the operating system down didn't take long and I think all told it was offline for about fifteen minutes. Once back at the ranch I had a few problems getting the web application working with it again but a few kill -9 commands saw it right again. I may go to give blood this afternoon, depends whether I go to try a new climbing wall this evening or not. I really could do with getting a bit more in shape again. [15:15] I still haven't decided whether to go climbing tonight or go and give blood instead. In the meantime I've been given a new job; working out which machines are connected to the UPS via the network and a Java/XML client type deal and which aren't. First off is giving the UPS a DNS name that isn't in the zone we want to delete. I think I'll give blood and climb another time. 18/03/2005 [16:55] Today ahs been, on the whole, a good day. For most of the morning I was able to do some little bits of documentation which'll allow me to build the new live server more sensibly. I also was also able to tick off my first fully moved over project from the list as well as ticking another one off as never having needed to be on there. I went out for lunch with Shaun and a few others to say goodbye to one of the developers before coming back to help some of my other migrating developers shift across certain files before shutting down apache and samba daemons. The old development box is getting quite sparse in what it's serving out now, but it's still getting hammered. I have to wonder how it's survived as long as it has with what it's been forced to endure. Now that a Microsoft Access job has been moved completely from the development server to the live one (where it'll remain) we're very close to having only one project left on the box and the developer for that one is extremely willing to help with his move. I think I may have it all done by the end of next week. At that point I get to trash the crap install and do some major firmware upgrades as well as take out about four SCSI cards which aren't doing anything other than making the box slower to boot. This weekend a few of us are off to Birmingham to The Outdoors Show. If I don't come back with anything I will be Most Surprised. 17/03/2005 [13:30] Aside from a few logfile location and database related misconfigurations I've just managed to get my first signoff from a developer on moving stuff from one server to another. I haven't disabled his logins on the old box yet, but I may do soon. I also had to rename the new machine this morning when I realised that I'd spelt the name of the person I'd named it after incorrectly. Bah. Other than a bit more work on the new server (getting some more DNS A and CNAME records added and removed here and there) that's been about it this morning. I head off to the old workplace this afternoon to do what is hopefully the last bit of handover to the new guy there. It's great, I can already think of it as somewhere I know a lot about, but don't really have any connection with. I'm not even going to get stressed if he's started dumping files all over the place and making things untidy. It's Not My Problem any more. With it being St Patrick's Day today and me knowing an actual >50% Irish person, I'm going to the pub this afternoon/evening and pretend I don't have work tomorrow. 16/03/2005 [13:45] Happy birthday to my brother today. I'm sure he won't want to be reminded of how old he is today, so I won't say anything. I had an excellent meeting with another developer today about his part of the development server I'm porting across. He also made me feel a lot happier - on being told his project had two separate parts to it - when he told me he could probably talk to someone else for me and get it moved off completely. He's also decided to modify his current development environment on the current machine to mirror the new one that I'm creating, all without my prompting, which is great. I'm off for a technical seminar this afternoon and then home. 15/03/2005 [17:55] For most of the day I've been in an Esteem/Sun presentation day thingy at a nearby hotel. As I'm under the usual NDA all I can really say is that the speakers were pretty interesting and the food was OK. I left before the last talk so I could come back here and change a machine's DNS details. I took the opportunity to reboot the machine after putting some symlinks to shutdown and startup scripts in the rc chain. I was pleased to note that they worked flawlessly and the services I wanted brought back up came up and the ones I didn't (didn't exist yet) weren't tried. In celebration I bought myself a new DVD-ROM from eBuyer which should apparently arrive some time before Wednesday of next week. Oh, yes. While at the seminar thing this morning I met up with Julian who waxed lyrical about his almost completed new kitchen and I decided it was time to start planning for a new kitchen of my own. Now that I have some money saved up I think I can start making serious plans. Of course, I won't be paying quite as much as he is. Elaine's been out of town all today doing some much-needed clothes shopping. I'd imagine she's on her way home by now so I should probably pop off too. Tomorrow I have a meeting with another developer regarding another project on the development box I'm building. Hopefully what I need to move over won't be too hairy. 14/03/2005 [18:00] What a Friday. Not only did I burn myself out totally doing all kinds of mindbending things with the box I'm trying to configure, I then left at lunch time and spent two and a half hours in an MRI machine doing extremely intensive brain things. By the time I got out I was ready to drop off. That, coupled with the hard day/week along with the fact that some dumb student caused me to crash into them on the way to the MRI appointment and caused bleeding from knee and finger and by the time I got home I just wanted to collapse on the sofa and forget about the week. Luckily Elaine was on hand to soothe my fevered brow for a while. Of course I couldn't forget about the stuff I had to do until some time on Sunday afternoon. The weekend was mainly about Elaine cooking cake and stuff. That and getting my hair cut, finally. Cormac and Steph invited us out to see a Celtic folk rock band type deal on Saturday night which I had to leave early from to run home and check we hadn't been broken in to. Long story, don't worry about it. Of course the house was fine. Today has been pretty much as intensive as Friday, only for a complete working day. I'm shattered. On the plus side I think I'm very close to getting the first complete project moved from the old development server to the new one. I had the two developers who've worked on the project sitting at my desk today showing me where to change certain parts of the code to make it work with my new directory structure. Aside from some Samba fiddling and a firewall issue with getting CVS to work I think that's about done. I'm not in tomorrow (on a Sun demo thingy) so I won't know if we're ready to go until Wednesday. Other than that I can't think of much else that's happening at the moment. My parents are both finally moving off Windows 98 to XP and getting machines capable of running at a decent speed, my brother's out of hospital and seems fine, I finished Homeworld2 and I don't know why usernames with hyphens in them don't show up as anything other than a UID when you do a ps under linux. Anyone know? 11/03/2005 [13:55] Short day, short entry. Have been fiddling with Samba for most of the day as well as advising a colleague on Apache installations. I managed to get the web application I'm installing to talk to the database, which was a major fist-punching-the-air experience. I'm pretty sure I know what to do for all the other ones now. I made sure I got it all written down too. Off for an MRI experiement now. Then the weekend! 10/03/2005 [14:15] Today has gone really quite well thus far. I've managed to get one project theoretically up and running (setting aside things like incorrect SSL certificates and no Oracle client installation). As a result I should really have lunch some time soon. I slept like the dead last night, only waking once because I was far too hot. Elaine and I are in pretty good moods at the moment, I think it's the way things are going at work and at home, coupled with more relaxation when we're around each other. I've had or arranged meetings with three of the four project people who'll be using my replacement development box, managed to document most, if not all, of what I've done to make the box work as far as it does and am really, really hungry. I should have got my hair cut yesterday but missed my appointment due to work. The hairdresser was very surprised to get a call from me apologising. Now, food. 09/03/2005 [10/03/2005 - 14:10] Lots and lots and lots and lots and lots of fun and games with Apache2, Resin-2.1.16 (not 2.1.6 as I said yesterday) and configuration files. I think I spent most of the day umbilically tied to my headphones listening to MP3s in an effort not to go made and kill someone or something. I ran in in the morning without breakfast, worked through lunch and when I'd finally achieved a minor breakthrough, left straight for climbing. This meant that although I climbed like a demon, by the time we got home I was running very low on energy both in body and brain. 08/03/2005 [18:20] What a day. It's been one that I would prefer to forget, only I can't because everything I've learnt today I have to remember or I'll put myself back another day trying to do the same things again. I feel I haven't actually learnt anything today, more bounced from problem to problem, fixing them by luck rather than skill, or judgement. It's been very frustrating. For most of the day I've been getting a new development server ready to work in the same fashion as the old box it's being groomed to replace. Getting the directory structure in place was overly complex until I settled on doing one project at a time. Hell, working out which projects were still live was an issue in itself. I now know that there are probably four live projects on the server, rather than the seventeen that seem to exist. Once I'd settled one which project to try and port across first I had a go at getting some of the back end infrastructure into place. The Java SDK went on easily enough, and with a bit of fiddling so did Resin 2.1.6 (we may change to version 3.x in the future if things work). I decided to bypass the sticky-looking samba setup in favour of the apache setup (mod_ssl). This turned out to be an extremely irritating exercise. There are a lot of fiddling little changes between 1.3.x and 2.0.x. Suffice it to say that it's taken me in the region of two hours to get a working SSL setup going on the new box using the cascading configuration files that I have to use to deal with the multiple projects and their requirements for individual web servers. In the middle of all this I've been getting the machine a new IP and DNS details as well as a slew of extra IPs that it's going to be listening on too. I had one of the web developers come down to take a look at the SSL part of my configuration file and he couldn't see the problem. The problem was that there was IfDefine HAVE_SSL stuff in there, which really didn't help. I don't know if it needs to be there or not. Probably not given that when I commented it out everything began to work as it was supposed to. I'm going for the cleanest configuration files I can to make it as maintainable as possible. The more cruft I can clear out, the better. Anyway, now that I have a server which servers HTTPS in the special way I need it to I can start thinking about moving over the content (I have the developer looking at what can go, what can be moved to the new CVS server I've already set up, and what needs to be copied across), working on getting samba working for the people who need it, and then moving on to the next project. The other three projects are going to be much harder as far as I can tell. There's the Resin stuff, the Oracle intergration, the Perl modules and on top of all of that the java setup. Fun huh? 07/03/2005 [13:40] Yes! Fixed a stupid problem with a Dell PowerEdge 1750 using the megaraid rather than megaraid2 driver for its hardware RAID setup. I should really thank two guy called Andrew and Eric for two halves of the solution which involved PCI ID strings and creating a new netstg2.img cramfs image. This morning the networks guy and I moved the other mail server to another site to ensure that we have proper redundancy in the event of power failure. That was easy and didn't take long. I think the machine was down for an hour and nine minutes all told. Once that was done we took a trip to another site to clear away some old network cables in a room with no lights. That was torchtastic. Back at the ranch I took some time to work out why the server insisted on building with the wrong driver and thanks to some emails I sent out over the weekend and some Googling have now implemented the solution which is building as I write this. Of course this is the easiest part of the whole project, but it gives me a stable and monitorable base to work from. The weekend was most excellent. Saturday started very well, and then remained almost as good while we laid about the house doing not so very much at all. I took the time to get some games reinstalled on my PC and finally killed Homeworld: Cataclysm. Just like with Homeworld, once I'd take a break of a few months it was easy. Cormac and Steph took us out for dinner as thanks for letting them stay while the house was replumbed for heating. We went to a noodle bar type deal where the service was almost too quick, the food wonderful and the heating warm enough to chase away the chill outside. Back home we sat and watched far too much Scrubs before heading to bed. Sunday was fairly good with some cleaning, some tidying, some shopping and much too much game playing. Most of the evening was spent playing Homeworld2 (or The Sims 2 in Elaine's case). I don't think I stopped until some time around 12:30 this morning. We didn't get to bed until 01:00. 04/03/2005 [12:55] We've finally had the initial meeting of the 'what to do with the weird and clever system that's undocumented and needs replacing' project. Already two years behind there's not really a rush to get it done, more of a pressing need which would be usefully furnished by getting on with things right now. So, we have a first stage plan which is basically 1) get a machine up and running, 2) get the right software on there, 3) start consolidating the multivarious configuration scripts on the old development server into simple neat ones on the new one, 4) ask each developer to come and look at the machine and ask in a whining voice why their bit of stuff doesn't work correctly. The last part may take a while. Once that's 'done'. I get to make the whole thing scriptable and add it to the the build script, try it, fix it, try it again and then start on the live server version. I built a snowman this morning in the area behind work. Now that the snow's all gone it looks pretty good out there by itself. A couple of people have remarked on it, including one of the bosses who asked me to come back in. Still, all done now. 03/03/2005 [17:10] I think today was good. I know that decomissioning old machines is a good thing. Especially when they're old Windows 2000 machines serving DNS. So now I have a few more U's of space in the rack for my new servers. Other than that I've been tidying the server room a bit, getting one of my servers booted for the first time with an initial build script, working out why Big Brother was going nuts, trying to figure out a way to get HTML::Mason installed along with mod_perl for Apache2, PHP and mod_ssl, all without user intervention. I don't think it's going to be easy. I'm leaving some of the configuration to the people who want the servers so they can do what they think they need to do and I'll turn it into scripted actions at a later date. I hope. The machine I added to the backup actually backed up last night, which is helpful. I don't think wearing contacts and looking at this screen is one hundred percent good for my eyes. I may go back to glasses and wear contacts on the weekend only. Oh yes, we didn't go climbing last night because Cormac and Steph came over to have showers (new boiler ignited today) and in the end we wouldn't have climbed for very long once we'd arrived there. So, that's about it I think. Time to go home. 02/03/2005 [17:50] Kind of an exciting day today. While it started off slowly I did get to provide some entertainment to Julian who's languishing at home with nothing to watch. At lunch I went out with Shaun and my co-workers in Operations Support to a pub where much food was eaten after a bit of a delay. I've just realised that my packed lunch is sitting in my bag uneaten. I think I'll share it with Elaine on the way to the climbing wall this evening. Anyway, back from lunch (where we tried to set our working world to rights) I started unboxing one of the five new servers I have to set up for the Web Services team. Getting the IP addresses allocated and the rights ports open on the firewall is going to take a bit longer. Either way the consistency checking on the RAID array can happen while I'm away enjoying myself. I found an abandoned PowerEdge 1400SC in a store room a few days ago. For a few minutes I wandered around asking people who might know who it belongs to. I'd like to use it for some test installs and perhaps as a dedicated RedHat kickstart installation box (to take the load off one of the other machines). Even if I don't use it for that, it'd make a useful Proof Of Concept platform. Anyway, that's it for now, time to go home and prepare for climbing. 01/03/2005 [12:10] Today we've been learning about rsync and how we probably should have picked it up about a decade ago. The thing is that I've never had cause to use it before now. Still, a new command assimilated and ready for production use. I'm a big fan of --dry-run too. I spent from about 17:00 to 00:30 last night getting my home machine up and running with the new graphics card, a fresh OS installation and getting it back to the same state as it was before. I think it was about time I moved from that incarnation anyway, even Firefox downloads were starting to misbehave. I doubt I would have been allowed to spend so much time (I also forgot to eat) on it if Elaine had been around. I guess that's one of the few good things to come out of her being away at the moment. I'm very much looking forward to her return tonight. For the moment there's more modifications to be done to the build script for the CVS server (and future PowerEdge 2850 machines) to allow Big Brother to say hi to the RAID controller as well as getting the backup regime in place. [18:05] I've just spent the entire afternoon from 13:00 until about five minutes ago working out and then documenting the bespoke XML-based backup system we have here. I think someone was doing some bad crack when they came up with it. Don't get me wrong, it's ingenious, clever and generally impressive and I doubt I could come up with anything nearly as good, but it's so complex, badly documented and basically impenetrable. It doesn't help that about six other people have fiddled with it since it was implemented, and the person who created it has left for somewhere else. I'm pretty sure I have a handle on how it all works now and I can even add another machine to the regime and have it actually get backed up when it's supposed to, but it's been a labour. As a result I'm going to go home, make sure the house is tidy, and wait for Elaine to come home. I may even go and meet her at the train station as a surprise.