BOFHJournal - February 2005

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

January's Journal
March's Journal

28/02/2005 [13:20] This morning I got in nice and early to clear down the queue on the mail server I was going to move. So, at atound 08:00 I was in and looking through it for spam, bad emails and the like. Happily the machine was down well before the planned time. The problem arose when we de-racked the machine and found that it wasn't actually on rails at all and had been resting on the machine below it. No-one know where the racking kit for it was. I put in an order as quickly as I could but it wasn't going to help the move right then. Happily once we'd got down to the other site we found a shelf which we mounted in the right place and got the thing in place, connected and fired up. While we were about we got me another card with a swipe strip on it so that I can get into the other server rooms on the other sites without having to have someone else with me. This necessitated me getting a new card, which also meant I got to have an updated photo on it. I wish I'd shaved this morning. I'm back at work now, but my line manager isn't in today so my meeting about the hardest of the projects isn't going to take place after lunch. I guess this means me and a colleague can get an rsync-to-backup-server thing in place for the CVS server in place and running now that that's live too. My new graphics card hasn't arrived yet. I'm hopeful it will arrive before the end of the day. I spent the weekend getting my machine ready by shuffling data to disks which won't be touched by the reinstall. With Elaine away in Scotland visiting her father I've had the time (too much) to myself to get on with consolidating my data and preparing things. For some odd reason my machine in its current incarnation has developed an intermittent cross-browser with displaying JPEGs. I'm hoping it'll be fixed by changing graphics card and operating system.

25/02/2005 [15:10] It's been a bitty day. I didn't have much to do this morning so I solved the reason why my Dell servers sometimes hang when booting to serial console. Turns out that if you have serial "redirection after boot" on in the BIOS, grub and the BIOS fight it out until one of them suffers some kind of timeout. Sometimes this results in a grub lockup, other times eventually you get a grub console boot screen. I've turned it off on two of the four machines affected and should be able to do the other two machines when they're moved to their final resting places on the next two Mondays. Five more servers have arrived for me to set up. I'm not sure what they're going to be used for yet other than the web services guys are using them. Setup is going to be a bit more complicated this time around as they're going to be both live and development boxes. I like how things have been going here recently; each project is slightly more complex than the one proceeding it. At this rate I may be able to tackle The Big One without actually doing something utterly catastrophic. Elaine's off to her Dad's place in Scotland early tomorrow morning. I was hoping to spend the weekend playing with my new GeForce 6600GT which allegedly shipped yesterday afternoon... only Parcelforce don't have any record of it being with them yet, so I don't think it's going to arrive today. Most likely Monday then, which is OK as Elaine's not back until late on Tuesday. I think I'll go and unbox those servers and see which are which. Some have different amounts of RAM/CPUs/HDDs and I have no idea which is which. [16:05] Grammar aside, I know now and have labeled the boxes accordingly. Of course I can't do anything to them until I know what the developers want doing with them. Frankly I'd like to go home considering how tired I feel at the moment. I may actually leave at 16:30, just like the normals do.

24/02/2005 [11:45] Elaine and I decided not to go climbing last night. We were both absolutely knackered. I think it's the cold more than anything, and this snow which falls but doesn't settle. I don't know what the road conditions are going to be like half the time. Sometimes I think it's safer to run to work. We chatted about stuff, sorted things out and generally caught up on the last few weeks. It's been harder recently, with everything going on, to spend proper time with each other just being comfortable and relaxed. I know I haven't been quite doing things right and it's something I'm going to be taking steps to address. Among the things I need to do is to actually think less. Odd as this may sound this would be a major achievement for me and do a lot of good. Seriously, don't ask. Otherwise work is going well. I've implemented a CVSNT CVS repository on a spare box here and asked my sample developer to have a go with WinCVS as the client. It's really rather good and everything looks to be stunningly compatible. There's a huge plus to CVSNT which is that you can set repository compatibility such that one may behave like a CVS 1.11.x server and another like a bells-and-whistles CVSNT server supporting the latest and greatest switches, flags and malarky. I'm modifying my build script to remove CVS (and RCS apparently) and install CVSNT and associated scripts, configurations and whatnot. Pending the meeting this afternoon I may rebuild the actual CVS box to the new specifications.

23/02/2005 [12:15] I hate to admit it but I couldn't stay away. I realised that if I didn't get the BrightSTOR ARCServe Backup patch installed on the two servers running that software at my old place pronto, there was probably going to be a lot more for me to do (as the person with the most knowledge of the setup) if it wasn't done and the machines were hacked. As a result (and owing to the fact that I ran in in the snow this morning and didn't have a bike or a jacket) I got a lift from one of the consultants, who has a Porsche Carrera 2, across town to get it done. It didn't take long. I'm pretty sure the machines hadn't been hacked, but there were some Discovery Service errors in each machine's logs. Here's hoping. I've done all I can considering I don't even work there any more. I really don't want to have to start going back there to help the new guy get his head around things. For the moment though I'll contact Networks and make sure that the place isn't serving warez and porn to the world at large. At lunch time today we have a semi-obligatory 'brown bag' seminar. This is where someone from the department gives (what will actually be) an interesting seminar on what their group actually does. I think this is a pretty good idea as most of the time people here don't seem to have a great idea of what other people are doing. The brown bag refers to the fact that we get some kind of free lunch (TANSTAAFL) as it's over the lunch break. I really need some new headphones. These in-ear buds just aren't up to the job.

22/02/2005 [15:35] One of what I thought was one of the core development servers went down at lunch time. It's the other half of a live service that's one of my next projects here. It frightens the hell out of me as it's what can best be described as a maze of twisty little chroots, all alike. I currently have no idea how to adminster it, let alone recreate it in a new way on the transition server I have to play with. I've requested some time with my line manager to work out what exactly we know how to do and to read over what documentation exists but it's a little complicated at present. I think it's going to need to be simplified quite a bit before I go ahead and create the replacement system. Right at the moment I'm tired, my back hurts and I want to go home and hug Elaine for a while. Hopefully things will seem a bit more sensible tomorrow.

21/02/2005 [15:35] Cormac and Steph wre still with us with their house due to be replumbed for heating and stuff beginning next Monday. Most of Saturday was spent lying around and generally recovering from the week. Well, that and because Elaine and Cormac stayed up until 03:00 playing Burnout 3 on my Playstation 2. We toddled off to have a play with the kites in the afternoon but found it just a bit too windy so went for a walk in the nearby town instead and had a look at some architecture. Cormac and Steph went out for a meal while Elaine and I stayed home and ate some of the stuff we had left over in the house. Sunday morning was all about the biggest fryup in quite a while. Once that was over we went out to do some shopping. We managed to get a double Cat-5e socket for the final bedroom as well as a bread bin, duvet and some network tools from places all over town and still get back in time for the rugby at 15:00... which wasn't on because the BBC's site had it wrong and it's next Sunday. Everyone sat around eating cookies while I did some patching before realising that one of the sockets I had was for phone, rather than data. One wuick trip later and everything was sorted. We spent the rest of the evening watching more Scrubs episodes before going to bed under the new duvet we'd bought. Problem is that the duvet's material is very 'loud' when you rustle it (which you can't help). As a result we've taken it back today and will be getting a non-down one another time. Annoying, but sleeping's pretty tough when there's a racket in your earhole. Today I made sure that mail was working properly on one server and then downed the other one and rebuilt it. I had the same problem with the NIC/switch not autonegotiating quickly enough so dumped a sleep into the post-install build script before it got copied to the machine. Everything was fine on buld then. After a quick up2date and a reboot it was soon re-relaying mail again as if nothing had happened. I love it when a plan comes together. I took a trip into town at lunch time with the duvet, met Elaine and got it returned before heading to my old workplace to make sure everything was fine for my replacement's start date of the 28th. Now I'm back at work I'll be doing some work on documentation for the remainder of the day and then heading home.

18/02/2005 [13:55] I was sure it was later in the day than this when I decided I had time to write today's entry. Just goes to show that some days have less going on in them than others. Yesterday it turned out that Cormac's house needs a bit of work done to it with regard to heating and stuff and it can't be done for about a week. Consequently he and Steph are bunking at our place until such time as it doesn't feel like the inside of a tomb at Cormac's house. It should be a whole lot of fun. Elaine went to the dentist this morning (which necessitated her getting up early so that she got some time at work before the appointment, and hence I got up early too). Turns out she's got an impacted wisdom tooth that won't come out without a hospital appointment. Given that it doesn't seem to be bothering her overmuch at the moment that's going to wait for a convenient time. I've been bringing my documentation up to date this morning. I'm determined to try and do it in parallel with whatever work I do here, updating as required. Theoretically this should mean that there'll never be a lot to do at one time and in the event of loss I should be able to reconstruct any lost configuration files from the contents of the wiki. Finally fed up of VLC not working for me I've installed mplayer, which seems to work a whole lot better out of the box. Now all I need to do is work out some way of getting it to work with Apple's movie trailers site. Other than that I'll be doing a rebuild of the CVS server this afternoon to see if I can bring it back to its current status as a working repository machine with pserver (yes, I know it's insecure) enabled. I'm also in the market for a new graphics card. Given my budget I think it's going to have to be a GeForce 6600GT.

17/02/2005 [12:30] A colleague from the DBA group and I have been working hard on a CVS setup that'll work with my CVS server. He, and others are going to have to use WinCVS (a Win32 client) and it's not entirely flag-compatible with CVS 1.11 on my linux box. This is a tad annoying. In other news I think I may have found a solution to the networking issues with my new server which I was having problems with yesterday. I've yet to try a new rebuild as yet, I'll report back on that later once I've done one. [13:30] One of the developers here has had a small belated birthday thingy and brought in champagne, cake and orange juice. A Bucks Fizz in the middle of the day is probably a bad idea. Especially before I have my lunch.

16/02/2005 [17/02/2005 - 12:05] The guys from PC support came in this morning with the PowerEdge 2650 that I wanted to swap my 2850 for. They were in a bit of a rush, which suited me down to the ground. Not doing anything else right then I got it racked up (heavy beast) and set up the initial RAID 1 mirror. For some reason I couldn't set the hot spare from within the Adaptec BIOS, which was a bit annoying. I also updated the motherboard and RAID BIOSes as well for good measure. Kickstart wasn't so good. For some unknown reason it wasn't anywhere near as hands-off as the other machine had been. I had to intervene a few times. The rest of the build proceeded OK until first reboot, at which point the build script failed on the first few network transfers (a links --source and some rpm installations via http). I had no real idea what was wrong but put it down to either bad VLAN/firewall configurations or the fact that the machine was plugged into a gigabit port on a Cisco 3750 and the other machine (which built perfectly) was on a 100Mb port on another model (of Cisco). I suspected extremely slow autonegotiation after the NIC came up on the machine. While I had the machine installed I set up the hot spare using the afacli tool (now successfully (manually) rpm'ed in). A second build on the same box failed in a similar but slightly different manner. By the time I was in a position to talk to Networks they'd gone home, so I went to. Climbing was extremely good with some real advances made on some tricky routes. We were all famished by the end so had some really greasy food and then went to bed.

15/02/2005 [18:40] OK, today has been one of those days where you achieve a lot, things go well and then at the end you make a fool of yourself on two international mailing lists. One of those days. You know. Anyway, this morning I got a good handle on how to get pserver working for CVS to such a degree that I think I'm ready for the tests tomorrow with one of the developers. This is a good thing as another developer emailed me today to ask for access to it. It hasn't even been built, let alone publicised yet. My new motherboard arrived for my home PC. I've put it to one side to take home this evening. I burnt final ISOs for RedHat Enterprise Linux 4 (at a blistering x8 speed. I swear there's something wrong with my CD writer) and started an install. Looked fairly similar to RHEL3 to be honest. On first boot the differences were obvious but I could still get it connected to RedHat Networks and get it up to date (already way behind). The 2.6.x kernel shows its advances over the 2.4.x kernel straight away. Unfortunately this is where I started to show my lack of knowledge as I tried to recompile it to get extra modules (specifically v4l) installed. After some posts to the Taroon and Nahant lists someone mentioned that I might like to RTFM. A couple of minutes later I was eating humble pie and passing on my perls of garnered wisdom to those people on the list who'd asked me what I'd learnt. I think now it's time to go home.

14/02/2005 [16:40] Today has been all about getting things done. This morning the guy who was set to do the garage roof texted me to say he could come today. This meant that I got to hang around at home until he turned up (on time). Once he had power from an extension in the garage I cycled to work. While at work I have been able to reinstall someone's machine with RHEL3 u4 (fully patched) rather than RH9 (unpatched), give the user Firefox and also navigate through the twisty maze of RPM dependencies to get Workrave installed for her. This is such a good thing that I'm going to put an article on it in the work wiki so that others may follow in my footsteps. I'm that nice. For most of the afternoon I've been burning RHEL4 beta CDs to try out tomorrow as well as getting firewall rules and host lists updated such that the final build of my new mail servers. We now have one machine that has a route through all the firewalls for SMTP in from everyone and out to everyone, and one which only allows SMTP in from the cluster we're beginning to hub through for virus and spam filtering and SMTP out to everyone. I've asked for the MX records to be updated to point to the cluster which is hubbing for us and, as soon as that change propagates everywhere, we'll be able to shut off SMTP to the other machine and tighten up security a bit. Once all of that is done I can shut down the listener on the older installation, clear the queue and bring it down for about forty minutes to do a fresh build. I must remember to upgrade the firmware too. Oh, and uncomment the line in Big Brother's monitoring configuration. For now though I'm going to go home and look at my new garage roof.

11/02/2005 [08:50] Looks like my motherboard is toast. I'm in the market for a new one now. Hopefully supporting 333MHZ FSB. I'm after an ASUS A7V8X (not the A7V8X-X) but can't seem to find one at the moment. I came in to work early in case I had to leave to be at home for the people coming to do the garage roof, but they're not coming today owing to it raining yesterday. [12:30] I've gone and ordered a ASUS A7V880 from eBuyer. Hopefully it'll arrive next Wednesday. [16:45] I've managed to get the DVI card working under linux now. If you want details then email me. Suffice it to say it's not hard. However now I have DVI working the quality appears to be worse than under analogue (which is pretty good), so I've remained on analogue although now I'm on the right driver which also allows 3D hardware acceleration. I think my work is done here for today, so I'm off home to paint a radiator and get the computer room back into a state where I can work on a PC without breaking bits off it.

10/02/2005 [11/02/2005 - 08:20] Again the entry is a day late. Never mind. Most of the morning was spent fine-tuning the build script to work with the final bits of the puzzle. I'm pretty sure it all works as expected now. Our flatscreens arrived in the morning and, excited, I powered down my machine and installed the DVI add-in card. The machine came up, the BIOS displayed, grub displayed (running RedHat Enterprise 3 u4) and I watched the kernel boot messages scroll by. However, as soon as X started the screen went blank. There were no error messages from the screen to say something was out of range. I removed the DVI card and booted with analogue input and everything worked perfectly. This is annoying. Currently I'm running on analogue rather than DVI input and hoping someone can tell me what the solution is. During this little episode I had to take time out to go to my old workplace and show the replacement guy how everything worked. I think he was a little overwhelmed by everything I had in place. I'm fairly sure he'll get a handle on things eventually. After two hours (an hour over the length of time I'd wanted to be there) I made my way over to another of my current job's sites to help a coworker unrack and rerack a large Sun. After that it was back to work, all in the pouring rain. As a result of being behind on things I was late home but eager to quickly get my new Athlon 3000+ XP installed in my A7V333 motherboard. I had a look at the manual and the motherboard supports 333MHz RAM (which) I have and a 100/133 FSB. The CPU I've bought says it has a 333 FSB. I now don't know if this is compatible or not. Of course this is all academic as in the process of taking off the old heatsink I think I managed to chip off two tiny, tiny little surface-mounted components next to the CPU socket, so now the motherboard can't find any RAM. I left it alone and went to spend time with Elaine and guests.

09/02/2005 [15:40] I'm currently in the middle of a kickstart rebuild on my server at the moment. This could be the final interation (scripts willing). I've added in a whole load of new MySQL functionality, Big Brother monitoring, ssh keys and authorized_keys files, updated exim configurations (kindly provided by the Mail Group) and a whole host of other little things. Now I just have to see if it builds. If I don't write anything else today we're off to a curry tonight, after climbing.

08/02/2005 [10:25] Turns out I can now boot a PowerEdge 2850 completely from serial console now, including from kickstart. Which is nice. What's also nice is that apparently my LCD monitor has arrived so I can get rid of this piddling little 17" goldfish bowl. What isn't so good is that while I'm also rejoicing that my final server (for the moment) has come in, it's another 2850, rather than the 2650 I was hoping for. I know that Dell don't make the 2650 any more (damn them), but the other two servers in my cluster are 2650s and I'd dearly love to use the same configuration file for all three. Now, there used to be a third server which was a 2650, but we gave it to another group before I got here because the project was taking so long to get off the ground. I have to plead with that group to take my brand, spanking new 2850 and swap it for their (my) 2650 which has been running Windows for I don't know how long. Here's hoping. [17:40] It's been a bit busy again today. I've been working on getting my other servers switched over to serial rather than VGA/keyboard/mouse. This has necessitated a good bit of fettling in the machine room to get things in place. It turns out that while I've been able to get on a bit further with my build scripts I won't be able to do any more until the people I'm doing it for come up with their complete list. In the meantime I can get Big Brother on and working with help from my colleague and keep on at the people who have the 2650 I want. I dunno, maybe if I polish it and put a bow on it that might help. Other than that I've been slogging through paperwork for buying more RedHat channel subscriptions.

07/02/2005 [12:35] Of course when I came in this morning I solved the partitioning thing in about thirty seconds (the docs don't match the implementation). I now have a happily booting/installing/post-install configuring PowerEdge2850 running on the latest megaraid2 driver for RHEL3u4 and currently everything seems fine. The weekend was pretty good. We had a great Saturday and worked out a whole load of things we need to and can do to the house. At some point we're going to lose the box/computer room and make the landing really spacious. This will necessitate a lot of building work, but it almost all needs doing anyway. That will give me the excuse to recarpet everywhere upstairs. In the meantime Elaine went out and got some point and we started painting the room. And what a difference it made. Sunday hit Elaine with a headache so she dealt with it and made some bread while I finished painting the coving and walls in the computer room. I also did some research on the whole partition thing and discovered why things weren't working. I can't remember the last time I was eager for the weekend to be over so I could get to work ang get things sorted. This morning (after solving the partitioning problem) I started off in a BigIP presentation meeting but left at the halfway point as I felt I could be more useful elsewhere. I've already prodded the people I'm doing the first project kickstart builds for to see if they've got a complete list of the things they want added/changed and for them to talk to Central about mailhubbing. With luck they'll get their arses in gear and we can get this put to bed. I need to learn more about CVS administration this afternoon.

04/02/2005 [07/02/2005 - 12:25] I really should get this done more on the day, rather than the next day (or after the weekend). Friday was all about getting the new PowerEdge 2850 with a PERC 4e/Di controller to boot from kickstart. It worked fine from CD, but when I started doing it from floppy to a HTTP-based install the installer couldn't find the RAID controller, and hence the disks. After some pointless hacking about with options to initrd to do with "expert noprobe" and similar I settled for adding one line to the 'pcitable' file on the driver floppy to list the RAID controller. This proved to be the best, easiest and simplest solution. If you want to know what I did, email me. Of course then I couldn't get the install to run because it seemed unable to remove the old partitions. I left work wondering what was up with 'clearpart'.

03/02/2005 [16:00] What a day. I thought I'd take a look at eximon on the server I recently handed over to the development team. Colour me extremely surprised when the machine appeared to have mail queuing on it. To cut a long and rather involved story short, some things had happened more quickly than people had thought and certain changes had been made. Suffice it to say that we got the listener shut down and opened up a hole in the right firewall to let the mail back out again. That was fun. I've got some extra stuff to do to the build (mainly getting mail hubbed incoming from elsewhere which will involve an MX record change at some point) but I'm basically waiting on other people for the moment. This means that I can go back to my other project of getting the new CVS server up and running. I should have had a call from RedHat Network today about purchasing, but that doesn't seem to have happened yet. I think I'll call them after finishing this entry. Other than that I've worked out a bit more about how things work here, probably annoyed a few people and confused a few others. I'm extremely tired now and have only just had lunch. I'm looking forward to going home at 17:00 and having hot chocolate with Elaine.

02/02/2005 [03/02/2005 - 10:40] The days are just so busy at the moment. A far better morning than yesterday. I spent most of the time getting the documentation of my build process down on the wiki we have here for future generation to gaze at in awe and wonder before racking up the new Dell 2850 we have and trying to get its DRAC card set up. Once installed (I love Dell rack kits) I set the RAID to initialising the two RAID 1 mirrors the box is going to have and went back to working on a new version of exim for the build I've just done. Documenting as I went I now have a very simple way to update exim and incorporate it into new builds of the servers. It'll also be simplicity itself to drop it onto existing boxes (built with my build script) to replace the current older version. It then being late in the day I went home as the road to the climbing wall was blocked by a few large accidents.

01/02/2005 [09:20] I'm tired and stuff today. Sometimes it all just gets on top of you a bit and you wonder just how worth it is doing what you're doing. Currently I don't know whether I'm going to go climbing in London tonight or go home. I may try and meet Elaine for lunch to say hi and stuff, try and shake this feeling. I've got the second half of the perl course today, just like last week. I had real trouble staying awake through yesterday's, however there's a chance I may do better today, again, just like last week. I just wish I wasn't so down at the moment. [19:00] Well, the day brightened considerably after the sad start. First of all I had another useful set of conversations with those nice people at RedHat Network Customer Support. I think I know why everything is as it is, including why we seem to be down a couple of channel subscriptions, and why we have Management entitlements in the way that we do. In need of some company from the delicious Elaine I'd already send off an email asking to meet her for lunch and so, at around 12:45 we met by the pond in town and had a wonderful time feeding the ducks and talking about the last few months. Once we'd had a good walk around I headed off to the last of my perl seminars and Elaine went back to work. In the break I popped off to see Bob and retrieved my Firewire drive and borrowed his USB2 drive to get my MP3s to work (no Firewire at work). Once the course was over I headed home rather than going climbing to keep Elaine company for the evening. We're having something nice to eat and I'd rather have it freshly cooked than reheated when I got back from London.