BOFHJournal - October 2008

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

September's Journal
November's Journal

17/10/2008 [15:55] Well, the time has come. This is my last entry until early November when I'm back in the saddle and home from China. It has been something of a fraught week work-wise. I haven't mentioned a lot of it because frankly it would bore the pants off almost all of you. Basically we've been working our little legs off trying to get our flagship system to actually work under high load. We haven't quite managed it but not through want of trying. I was in quite early this morning rebuilding a RHEL5 box with two RHEL4 Xen guests on it into a flat RHEL4 box so we could have another application layer server which didn't "suffer" from the effects of being virtualised. I call repeated bullshit on that particular complained leveled at it and us by those who claim it was a contributing factor in the slowdown and resource hogging we were seeing. In fact as I think I said yesterday those people saying the above actually admitted something else was to blame, and still requested we do what I did this morning. Le sigh. The rest of the day has been used up writing and rewriting the documentation to allow someone else to set up, build and configure a Red hat server without me here. I hate the idea of someone else doing it, but this is the job of a sysadmin sometimes; to make sure someone else can do your job if you're not there. I'm sure I'll have to do things to it when I get back anyway. So, last night was a nice Thursday with a film and an episode or two of Bablyon 5 watched. I was completely drained by the end of the evening and was probably asleep before everyone was a mile from the house. This evening I intend to go to the gym and have a nice light session before an equally light dinner and some test packing of my rucksack for the trip. Tomorrow morning I'll be racing in an VIII with a mixed crew of good to very experienced people. I'm actually quite interested to see if we generate a better time for the 2.6km over the same distance my novice crew did around about this time last year. Saturday afternoon will be packing for real and potentially some last minute Skype chat with Kris before I'm down to the boathouse to sanitize a boat to take to the swimming pool where a number of us will be doing capsize tests. Over a year of rowing and we only now get around to it. Amusing to say the least. Dinner will be steak, and chips. Sunday morning I'll be on a coach to the uncertain hell of Heathrow Terminal 5 and then I vanish from this timezone for quite some time. I'm sure you'll all get along swimmingly without me. Have fun. I certainly intend to.

16/10/2008 [16:50] For a long time now the developers here have been complaining that Xen is to blame for our application servers being jammed up over the last few weeks. This has been exceedingly annoying as I and my team are damned sure that wasn't the case. Today, despite being told to tear down a Xen box and rebuild it as a standard server we were proven right. It's actually the shonky code they have which every time a user pushes a button does some massive amounts of views on a database, pulls in a crapload of information, creates an RTF and then converts that to a PDF to spit back to the user. This, obviously, completely flattens the apps servers when they're asked to do it as much of the code is Java. Have more than one person try it and you get the kinds of problems we've been seeing recently. As for the box I have to rebuild, they got around to demanding I do it so late in the day I've only had time to copy off the DomUs and the configurations this afternoon. The (re)build is going to have to wait until tomorrow, my last day here for two and-a-tiny-bit weeks. Necessarily I've therefore spent the entire morning writing updated documentation for building a RHEL server from the ground up and having it ready to turn over to the Applications and DBA teams in case someone else needs to do another box while I'm away. I'm dreading coming back to find that someone has created a RHEL box or two and I wasn't around to configure it as I would have done. Still, this is Kris and China we're talking about here. Exercise day off today. Especially after I went to the gym last night when I was feeling really quite crappy. The endorphins kicked in about three quarters of the way through - which is good - as I was very close to just throwing in the towel. Happily I didn't and was rewarded. Not only that but there was also icecream and Kris later on in the evening too, which made going to the gym feel a whole lot better as I could a) eat icecream and b) tell Kris I'd gone. And now, home. I will be in earlyish tomorrow to get this box built and prepped as quickly as I can so that I have the rest of the day to put my tasks in order for the duration of my time away... oh, and because the Applications team need the server back as soon as possible. Last Thursday for two weeks tonight too. I know of only one person who's confirmed they're coming. It doesn't matter, I have a sofa, things to watch and an over to put pizza in. I'm sorted. Well, apart from Sterling dropping against the Yuan.

15/10/2008 [15:55] I tried sleeping under a sheet last night in an attempt to work out what's happening with my body temperature at night. Ended up having to drag a bit of the duvet on to me some time in the early morning. It wasn't another hot night anyway. Rowing being cancelled last night meant that I could just go home and relax for the evening. Part of this relaxation involved me doing a nice four egg omelette and then managing to cut some off it off and onto the floor when I flipped it over. This was a first. An annoying first considering there was some rather nice pastrami sliced up in it. It didn't matter too much as there was plenty of omelette left to eat. I caught up on some television and went to bed. It was raining when I got up this morning, but not too hard. Still feeling a bit sleepy I took a little longer than usual to get ready for work and as a result left the house just as the rain got really heavy. I don't know if there was something in the rain but I had to stop a number of times to wipe the water from my eyes, which were smarting. As I write this my shoes and socks are still in one of the boiler rooms here, drying out. As I'm going to the gym this evening I luckily have a spare pair of trainers and some socks that I'm wearing in the meantime. Aside from firefighting the overloaded systems we're dealing with at the moment I've also managed to cobble together the ultimate anal-attentive two page document of where Kris and I will be staying and what transport we'll be taking over the next two weeks. In conjunction with the other documents, maps, photocopies and printouts I have I think I'm almost ready to go! Just need to pack and remember that all my toiletries need to fit into a transparent zip-lock bag. I'm fairly certain my sleep patterns are beginning to have the first wisps of an effect on my mood and energy levels, which is annoying. I guess I'm just very lucky I have a lovely girl I'm taking two weeks off from all that structured exercise and stress to go and see soon.

14/10/2008 [17:15] After last night's 10057m coached ergo session I was extremely happy to go home and have a nice big side of cow with vegetables and a Crunchie for dessert. I was also very pleased to go to be early with what I thought would be a thinner duvet. Now, some combination of the duvet being thicker, me not having digested my food properly, a hot night or perhaps me coming down with something caused me to have an unpleasant night which resulted in a worse than usual wake up in the small hours. Still, I did get back to sleep and I wasn't in too bad shape when I got up this morning. In the spirit of trying to save a little bit of cash here and there I opted to make lunch today rather than have a burrito. Unfortunately this meant that at 11:55 when my body was used to getting up and filling itself with Mexican produce it instead had to wait another hour before I was ready to put food inside it. Consequently the pastrami, brie, rocket/spinach/watercress and mango chutney sandwich I had (along with cherry tomatoes, a yoghurt and a banana) vanished in probably less than three minutes. And I'm hungry again already. The day has been filled with my team running around trying to make application and web server layers of our flagship system work correctly without knowing that it was going to be being hammered by upwards of a few thousand people for the next few days. Bad performance was blamed on us and our hardware rather than the fact that load testing had not been carried out to high enough peaks and in fact if we'd known what was coming we'd probably have laid on more resources to shoulder the burden. As it is we're now playing catchup and hoping that we can grab enough kit (both physical and virtual) to deal with things. I also don't know if I'm rowing this evening or not. Part of me wants both the outing to be cancelled and to not go to the gym instead. I feel tired and I think I should be resting some more at the moment. I could be stretching myself a little too thinly (although I was doing all this and more only three or four months ago and feeling fine) so perhaps it's time to wind down to be ready for the race on Saturday and then two weeks of activities (but not so much structured high-intensity exercise) in China with Kris. The text has just come in confirming that the outing is off. I think I'm going to go home (eventually, when I'm sure that I'm not needed here any more) and not go to the gym. Tomorrow I can erg at lunch time and gym in the evening (or climb), but right now I really don't feel like doing anything more than cooking something nice and watching a film.

13/10/2008 [16:15] Busy weekend. After Friday's gym session with five minute full power ergo run at the end I was ready to collapse when I got home. I think I did, actually. I went to bed pretty early anyway. Saturday started with a mammoth double row in an VIII. The first session was a power outing which left us feeling rather wobbly. Which didn't really help when the second outing was a technical one. Happily the weather was brilliant again so I didn't mind hanging around to help the new starters get their row on. During the course of that I managed to put a 1.5cm cut in the top of my head by banging against a scull in one of the boathouse racks. Although it bled a little it wasn't a serious cut and didn't actually hurt at all after the event. Keith and I headed to B&Q straight from rowing where I managed to make the steal of the century on eight rolls of 5.0 R-value recycled plastic loft insulation. Getting it home on Keith's tricycle was relatively simple with the purchase of a small tub of bungie cords. Finally home at 15:40 after leaving for the boat house before 08:00 I had some lunch and watched about an hour of television before Dunk turned up. He helped me clear some cruft from the loft and get the new insulation up there. All I need now is some gloves (for the two rolls of normal fibreglass insulation I have) and I can get on and lay this stuff and finish a job which I should have done almost four years ago. Slow and steady wins the race... After Dunk had helped me with the loft stuff we headed over to Cat's party at her and Keith's house and spent a lovely evening enjoying food and drink and chatter even though I was starting to fall asleep by about 21:00. I headed home about 23:30 and went out like a light. Dunk apparently arrived some time between 12:00 and 01:59 ("It had a one at the left hand side"). Although my first outing for Sunday had been cancelled I was still up for 08:00 and ready to leave for my 10:00 outing when, at 09:00 I got an SMS saying that outing was off too. This meant that I was in when Andy arrived. This was something of a surprise as Dunk and I had been expecting Sam. She duly turned up half an hour later at which point we discovered that she'd planned for Andy to be here when I returned from rowing. We all had a damned good chat before Dunk headed home at about 11:45. That was when Sam and Andy went for a walk and I went for my Sunday run. I was happy to make it around the course in a good 57min 30sec and not feel too dead at the end of it. Kris and I had a brilliant and wonderful two way video chat between 13:00 and 15:45 when Andy and Sam arrived back (not having found a Sunday roast). They got to appear on Skype too and say hi before hunger drove them to suggest going to the local pub for food and then the cinema. Unfortunately we were a few minutes too late for food there so we went to the leisure park where the cinema was and had something vaguely Mexican. Following that we enjoyed the Bourne-like stylings of Liam Neeson in Taken, which is really rather good. By the time the film was over we were all quite tired, I wanted to go home and collapse and Andy needed to get back to London. Sam drove me home and stayed for a cup of tea before leaving me to unwind. This morning I again woke at some silly hour and got back to sleep again, went to work visited the surgery to have a practice nurse tell me that yes indeed, my cut was superficial, booked my coach tickets too and from Terminal 5 and confirmed the final hotel bookings and scheduling for my time with Kris in China. It's going to be amazing. I just wish it was now, not six days from now. What's now is going to Tesco, then a 40 minute coached ergo session which might possibly kill me to death.

10/10/2008 [16:40] A better day than yesterday on Planet Ben. Last night was really quite odd. In the end only one or two people were going to come over and I was feeling pretty tired so I just cancelled the evening and had some time to myself to grump a little, recouperate from the day and watch horrifically bad films. I guess I was in the mood for punishment because these sucked astonishing amounts of ass. I'm not even going to tell you what they were I'm that embarrassed to have seen them. Having been tired out so much from the day I was in no state to do anything other than cook some food and settle down on the sofa to see if I could sit through what I'd chosen to watch. I could, but it sent me to bed in a bad mood. Don't ask. As a result of emails and the stresses of yesterday I woke in a foul, but much less so than the previous night, mood. Reading emails from Kris brightened things tremendously and I came into work feeling really rather good. The morning proceeded apace with me getting the Apex stuff I've been working on fixed and working with one simple bit of effort. Lunch was a hog roast barbeque between us and the other IT group in the institution and then the afternoon was spent adding SSL-controlled access to the Apex stuff just because I could. I'm off to the gym now to work off the pork and apple sandwiches and doughnuts I had for lunch and then eat a single slice of bread for dinner. The weekend is rowing and DIY-packed with a party and some running in here ahd there as I can fit them in. See you Monday.

09/10/2008 [17:45] I slept really badly again last night. Must have woken at some horrible hour like 03:30 to judge from the blurry number on the clock when I rolled over and sighed. Still, that probably made getting up at 06:30 a bit easier. Or maybe it didn't, I don't know. Either way I still don't feel too bad after an hour of so of being up. Truth be told though I was on a real downer when I got up this morning and was prepared to be feeling pretty bad all day. Happily though I'd thought to cue up some good music on my iPod to cycle into town with and the weather turned out to be truly fabulous when I opened the door. Clear sky, almost but not quite chilly and no wind at all. I loved the cycle into work and was grinning by the time I got to the place I was going to be doing network switch replacements. After doing that (which was a doddle owing to the fact that the cabinet we were working in was nice a neatly laid out) we hung around in town eating bacon sandwiches and drinking hot chocolate until Nationwide opened and I was able to get my ISA book updated. It was a distinct surprise to find out exactly how much I have saved in that tax-free account. I'm pretty sure by this time next year I'll not only have a new kitchen but still have some savings as well. In work productivity continued with a slew of barcode producing Perl modules needing installed (don't ask me about Barcode::Code128) and a deeper understanding of Oracle's Apex web server being gained. Not that I really wanted the latter but it's important for getting the applications it's going to be serving available through our load balancing solution. I have to admit that I'm glad today wasn't an exercise day. I'm just not feeling the urge right now. Having looked at my ergo scores for the last few months (back to May) I really don't seem to have improved at all. In fact I appear to be less powerful to the tune of up to 200m over 30 minutes. I have no idea why. Anyway, Kris popped up on Skype just as I was thinking about her and after a brief bit of decompression from the rather large amount of stress she's continually under there I managed to coax a brilliant and shiny smile from her. I like that I always seem to be able to do so given time and a chance to think of the right things to say or remind her that are the case. Like the fact I'll be there in ten days, or that as a teacher she can put anything she likes on the final exam for her students... No-one's turning up for this evening's Thursday thingy so I'm going to take the opportunity to have an evening to myself, cook some proper food and watch something no-one else would be interested in.

08/10/2008 [12:10] Hurrah for rowing at night! Although last night's outing was some of the best and worst rowing this new crew I'm in has ever done it was a wonderful way to spend the evening. A perfectly mild night, no wind to speak of (except when we were rowing against what there was of it) and barely anyone else on the river. Rowing in almost pitch blackness is very hard but really does teach you to feel for the rhythm of the boat, listen to the sounds of the other blades and keep the balance by feel rather than by sight. Not a particularly tough outing physically but mentally it was draining. By the time I got home I was definitely ready for the Tesco Finest* Thai Chicken Soup and a handful of carrots and baby sweetcorn. It's really rather tasty and I look forward to buying more of it in the near future. Today though I enjoyed not getting up at the sparrowfart of dawn and having a nice morning of installing more Perl modules, replying to emails and getting people's machines to write DVDs without crashing. Now I'm off to do my lunch time ergo and try not to be as crap as I was on Monday. Hopefully I'll improve on the distance I did then without killing myself. Gym this evening. [16:25] Nothing else has happened today, other than a nice long chat to Kris after I got back from erging at lunch time. The weather has been rather nice and I'm feeling quite lively despite how much energy I expended at the gym. Apparently I'm doing another early morning tomorrow in the centre of town, but it's not something I'll be taking responsibility for. More being an extra pair of hands. I should get on and complete some stuff and then get out of here. They's gym to be visited and then a rather lovely pasta sauce with pancetta in it to pour over a big bowl of pasta as I watch something fun on the box.

07/10/2008 [08/10/2008 - 12:00] Ooops, sorry. Forgot to write this yesterday. Probably due to the fact that Tuesday was a very long day. I was up and out of the house before 06:30 again so I could be in town to try doing a forced upgrade of the system BIOS in that damned x4600 M2 server again. I had everything in place and was well on the way to getting it done (once I'd stolen a network connection on a VLAN capably of passing TFTP traffic) when the iLOM/SP refused to accept the new BIOS, claiming that the BIOS was unknown/inaccessible. I'd have sworn and become angry but frankly I was too tired and too sanguin about it to get worked up. I came into work where I ended up having a morning of achieving lots of other small but useful things instead. That made me feel a bit better. Lunch was a burrito (as usual) and a damned fine one at that. In the spirit of not overeating I actually turned down the remaining third of Cormac's (madness). The afternoon was quiet and I got some things done. I was actually shooed out of the office come 17:00 so I had plenty of time to cycle home.

06/10/2008 [17:10] Curious weekend filled with extreme apathy and dogged determination. After I found out at the last minute on Friday that there would be no rowing on Saturday morning due to a chronic cox shortage I ended up staying in bed until the unheard of hour of 08:00. Unable to stay there any longer I got up and pottered about the house, ending up in the loft to ascertain how much loft insulation was going to be required to do the remaining half of the house. Yes I know I should have done it years ago but sometimes when your life changes (people leave, etc.) you tend to stick to the things you want to do rather than those you don't. Yes, I know it would have saved me money on my heating in the winters since then but hey, there you go. Anyway, I went up twice (the second time replacing the bulb which died the first time I turned it on after about two years of not being used) and managed to shift all the rubbish that I'd never got around to getting out of there previously to right next to the hatch so that with some help from someone else I can get it down and out of the way. There might even be some bits I can use (like a bannister rail which is nicer than the one I currently have in place). I reckon I need about eight rolls of insulation so I'm hoping that me, my B&Q vouchers and someone with a car (or a large load-carrying trike) can head over there (perhaps next Saturday afternoon) and grab it (always assuming it'll be in stock). But back to this weekend. After all that work (and not going to help teach the new starters at rowing) I had a nice long chat with Kris which was its usual excellent and lovely experience and then had Sam over for tea and chatter. Unfortunately I had to shoo her out at about 18:00 so I could get down to the boat house for chilli prepration and an evening of entertaining people. It was supposed to be some kind of television-watching thing but with twenty people it became more of a social evening with plenty of food and drink. Lots of drink in fact. Although Keith turned up with some carrots from his allotment too. I left at about midnight while a good few people were still drinking. Sunday was all about rain. Lots of rain. Really very much rain. Rain in which I was rowing for two hours. Luckily the air temperature was much higher than the previous day so it wasn't totally unpleasant. Once I got home (soaked to the skin) I decided I really wasn't going to do any more exercise. Total apathy took over except that I did go out to Tesco for food after a second weekend chat with Kris. Unfortunately she was tired and I was in a rush so we ended the conversation on a stressful note. Happily it only took an SMS and an email and we were all sorted again. Yay for us. I didn't want to cook by the end of the day's 'excitement' so took the ultimate slacker option of a Dominos pizza and television before bed. Naturally, not having been for a run I figured I may as well use my broken sleep to get up early this morning and do my run then. My body conspired against me and contrived to keep me asleep until 07:30 (through my alarm). Therefore I'm going to go as soon as I go home (if the weather's still good) to build on the rather-more-tiring-than-I-thought-it-would-be 30 minute ergo session I did at lunch time. Bit disappointed about that. 7688m is not good considering the distances I've been achieving lately. Maybe the blood donation on Friday is still affecting me. Work today has been all about preparing another laptop to do TFTP duty tomorrow morning (07:00 in the server room) and reading up on how to bootstrap a bricked x4600 M2 just in case that's what happens when I do a force SP/BIOS firmware upgrade...

03/10/2008 [16:20] I actual laid in bed this morning until the distinctly luxurious time of 07:45. Getting up, doing my morning exercises and making lunch etc. were a joy to do. Cycling in was less enjoyable with a brittle headwind and the beginnings of rain. Replying to Kris' emails gave me the idea (a little late in the day, as it were) of applying for a Nationwide Flex Account which would give me a debit card I could use overseas (i.e. China) without incurring any additional charges other than the exchange rate. Going through the application on the phone I discovered that if I went into my local branch at lunch time with a certain reference code there might be a chance I'd get the account opened and the debit card through the post before I leave for China. This did mean that I didn't get to ergo at lunch time, but given I gave blood yesterday I figured it was best to give myself as much time as possible to heal before I exerted myself. I'll gym this evening and do an ergo session then instead. The weekend is going to be rammed as near as I can tell. Saturday is rowing, teaching the newbies, lunch, conversation with Kris, a cuppa with Sam and then a rowing club social event in the evening. Sunday is rowing, running, Tesco, housework, possibly another chat to Kris and then, finally, some time to myself. Shortly after that it'll be Monday, again.

02/10/2008 [15:45] Gah, stupid Sun x4600 M2. I was in a server room at 06:30 this morning powering down a server to put more RAM into it. I also upgraded the LSI RAID firmware and tried to update the iLOM and system BIOS too. The former of the final pair worked but no matter what I tried the latter refused to take. There were some worrying moments when I thought I might have bricked the box but as the end of my downtime window approached I was able to get it up and running again. Sun tell me it's in a slightly worrying place right now so I've booked some downtime for next Tuesday to try a TFTP-based force-update of the BIOS to see if that works. As it was updating the iLOM didn't get rid of the KCS IPMI-related minor errors we get 101 of every ten minutes. When or if the BIOS update works on Tuesday I'm going to reboot the box into an older kernel to see if that fixes it. Not that I can leave the box in that kernel for production use but at least I'll be able to see if that's the cause of the problem. Otherwise today has been a day of being too hot in the office, ordering the cheapest "LaCie Mobile Disk 2.5inch 320GB USB 2.0 (Black)" I can find (damn eXpansys for being out of stock and chatting with Kris before she went to bed. I should head off and give blood now before going home and doing lots of washing up before people come over (or don't) this evening.

01/10/2008 [15:50] October! This is great! It means there's less than nineteen days until I go to China to see Kris (and China, of course). I've just managed to get under a late-closing deadline for getting my BMC membership renewed at 50% off and purchased travel insurance for the duration of my time away. Expensive, but not as expensive as it could have been, which is nice. Last night was a rowing outing. Given that we didn't get out on the water until it was starting to get dark we actually ended up rowing back in the full dark of night. An interesting experience I look forward to repeating, especially in a boat as well-balanced as ours is becoming. Quiet evening once I got home, although I did take the opportunity to shave. I may have a slightly lackadaisical approach to facial grooming, but when people ask if you've had the week off from work because you're looking pretty scruffy it's probably time you neatened up a bit. Someone standing next to me did mention to the questioner that I worked in IT which seemed to satisfy him alway, which was fairly amusing. Work today has been mostly about meetings to do with meetings about (Oracle's) Apex, Nessus scanning and setting up our new SSL VPN solution. I've been to do my Wednesday ergo session via my house to drop off the laptop for tomorrow morning's very early work on the server I was fiddling with last Tuesday morning and about now I'm just working through some small fiddling bits of stuff to make tomorrow morning go as smoothly as possible. I'm off to the gym this evening for a non-legs-based session and then I've a mass of fun downloaded television to watch. I should get a reasonably early night though to make sure getting up tomorrow isn't going to be total poo.