BOFHJournal - September 2011

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

August's Journal
October's Journal

28/09/2011 [16:15] Well, it hasn't been the quietest day I could have hoped for. The server-and-storage move this morning went perfectly well apart from an apparently failed battery on one of the controller cards (which Oracle say is just a false-positive caused by the firmware revision it's on (what is it with RAID controllers, batteries and firmware and me at the moment?)). Having logged it with Oracle I'm hopefully that my co-worker will deal with it tomorrow or Friday while I'm on vacation. Otherwise I've been doing a few bits and bobs here and there, popping home for lunch to see Kris(!) and looking out of the window and wishing I was outside. Going over to the new server room to help rescue someone's lost swipe card was a nice way to use up half an hour and be outside, though. I'm aiming to get out of here in about ten minutes and see what kind of quiet mischief Kris and I can get up to before she falls asleep in my arms because of jetlag. See you on Monday when I get to tell my line manager about everything that's gone on in the last three weeks.

27/09/2011 [17:15] Today I have mostly done everything. Every single thing. That's right. Among the things I did today were: get up and into work for 06:30, firmware patch two machines, watch as one machine bricked the BIOS, opened a P1 call with Oracle, had some instructions to reset the CMOS on the BIOS, augmented these instructions with repeated power cycled and a cold reboot, eventually got the server working, worked with Oracle to (not, as yet) get some resolution on why fully half of our firmware updates are failing in some way which gives me a raised blood pressure, went for a 6K interval run, did a crapload of load balancer configurations including some pretty hairy response ruleage, sorted out some firewall issues and even managed to send a few emails to Kris (hopefully before she starts traveling). All I have to do now is make to the gym and home alive and today will have been something of a Big Day. Of course, there's also the small shopping trip I need to do to make sure there are nice things for Kris to have for lunch tmorrow, but otherwise I think I'll then be done! Phew. Not that tomorrow is going to be much lighter in terms of things to do. It just has a happy end.

26/09/2011 [17:25] It got to this time without me noticing. Which is both good (as it means I was getting stuff done and time ticks on to when Kris comes back) and bad as I've had a devil of a few days, work-wise. If we rewind to Friday night I went to the gym, and then headed out to watch a film over at a friend's house where there was bacon and pancakes and maple syrup. It was quite a long film so I didn't get home until some time after ten. I jumped into bed so that I could be awake in time for parkrun. Saturday's parkrun was pretty good. I wasn't sure I'd done enough to get a personal best, but apparently I managed it, by six seconds. Which was nice. After that it was just a case of going to Tesco and then going home to snack and chat to Kris. I know I did something else on Saturday, but for the life of me I can't remember what it was... Oh no, of course I know what I did next. I went into work to firmware update two servers. One worked perfectly. The other went so badly wrong that the ILOM part of the server completely bricked itself. Which was nice. In the end I left at around 18:30 having opened a call with Oracle to get the thing replaced on Monday. Sunday morning I got up and decided to go for a nice long run. So I did. About 16K of such. It would have been longer but I ran out of energy and had to detour by the boat house to get some water before jogging home. Next time I'll do better. I slobbed for a bit of the middle of the day and then headed out to the cinema with Cormac before grabbing a burrito (chicken, I'm trying to be a little healthy) and then bed. Today I was trying to fix the issue I had over the weekend, unsuccessfully, and worrying that the wrong part had been sent out by Oracle (it hadn't). The engineer turned up on time, we got the part replaced (the whole motherboard) and the server back up and running just within the two hour downtime period I kind of hadn't quite asked for. Still, all appears well now. This afternoon I have mainly been checking all the scripts and files I'll need for tomorrow morning and adding in little tweaks to the shutdown scripts for the new UPS client software. I'm pretty certain something is going to go wrong tomorrow no matter how hard I plan, but I've covered every eventuality and have notified everyone that I can of what can and could (and should) go wrong (and right). All I can do now is do my job tomorrow morning just before 07:00 and hopefully finish before 09:00. I'll probably try and update this before the end of the day tomorrow so you have some idea of how my morning went. Of course, if I don't that probably means it either went really badly, or I just forgot.

23/09/2011 [16:40] One of those mornings when you have to concentrate continuously for three hours at a time to make sure you don't miss anything. That's what I had today. I was doing DNS, firewall and load balancer stuff all at the same time as well as generating new SSL certificates and writing notes on everything so I could send an email about everything I'd done, just in case my team leader decided to take over everything I'd done when he got back. It was hard, but very rewarding eventually to know that I'd got everything done (as far as I can tell) in one fell swoop and now it's just a case of someone else bringing up the servers underneath the neat and tidy service infrastructure I'd built. Here's hoping I didn't miss anything or misconfigured anything due to a lack of information as to how everything is actually supposed to hang together. After yesterday's complete washout in terms of exercise and this morning's bout of high concentration it was a real pleasure to get out in the frankly lovely weather and push myself for 25 minutes or so. It did wonders for clearing my head and body of mental and physical tension. I've also just had an email from my mother about her second eye operation. I think I'll just go with "not as easy as the first one" and leave it at that. This weekend I will actually be in work for a few hours (hopefully no more than two) doing firmware updates on an application server and its database server. Given it's the development stream I shouldn't be affecting anyone. Other than that there's parkrun, Tesco, chatting to Kris and maybe some other exercise here and there if I can be bothered.

22/09/2011 [17:15] Most of today I have spent in the other server room with engineers trying to get the maglocks to work. This was dull. It meant that I missed going for my run and that I didn't get to have lunch until gone 14:30. I now have to decide if I'm going to go home and go for a run. Or go home and then to the pub and see people. Or just go home and slob. I'm really feeling like I'm somewhere between the first and third options. Spending half an hour trying to figure out where the timeout was I needed to increase on the load balancer to allow a set of long reports to run without connections to the back end web node timing out wasn't the way I wanted to end the day either. Still, last night's double episode downloaded television viewing with Kris worked pretty well, which was nice. Although I'd have much preferred to have her sitting next to me and not watching anything than the way it was. She's wonderful, and even though I told her not to she's getting me dried fruit. Now that's love.

21/09/2011 [17:00] Urgh, what a day. A mild telling off by my line manager for advocating using "> " rather than different text colours for quoting emails in replies, then lots of confusion over what to do about a new set up of Oracle Discoverer, then another RAID cache battery apparently failed and I had to spend lunchtime doing a firmware upgrade of the ILOM/BIOS which, oddly fixed the issue. This now means we should be doing the same thing to about two dozen other servers (currently two others are showing the same issue). The chances of anything other than three of the servers I listed in my notification email getting done before Christmas are pretty much zero though, which is frustrating. Not to mention the fact that this update would be the perfect time to do the RHEL5.6 to 5.7 update which has been pending for months, too. That'd limit things to one downtime instance, too. There was a confusion with meeting the UPS engineer this morning (it wasn't me who was doing the meeting) so that's going to have to be rescheduled and I'm meeting other engineers at the new server room tomorrow morning for 08:00, which means a slightly earlier rise time. I may skip morning exercises just so I can be there in plenty of time. Gym tonight though, then some Skype/TV time with Kris!

20/09/2011 [17:10] Mostly today I have been trying to figure out why things weren't working. It was lots of things and you don't really care enough to have me explain what they were. Suffice it to say pretty much all of them were working by the end of the day (i.e. now). I ran at lunch time. A nice set of intervals (100 paces on (fast/unsustainable, 100 off (jogging), repeat) for 6K. I was definitely feeling it by the end, which was good in combination with the slightly heavier weights session I did last night. To be honest I think that's it for now. Not much else happening right at this moment. Also not sure what I'm going to have for dinner tonight. I have this feeling that something is supposed to be going on this evening but for the life of me I can't remember what. And there's nothing on the calendar either.

19/09/2011 [17:10] Lots went on this weekend. Top of the list was Kris deciding that she just couldn't wait another fifty days for us to be together for a bit (and I mean, who can blame here, look at me) and booking a flight back to England for a few days from the middle of next week. Of course, the day she arrives I can't get off work as I'm the only person in the office as well as doing some important server moves, but otherwise for the rest of the week I'm gone and spending time with my wife! Saturday I was out doing parkrun again. Although the humidity was well down on the previous week and I actually felt fitter and fresher I wasn't as fast or faster than the week before, which was a bit of a shame. After that I decided I was just going to go to Tesco and then relax for the rest of the day although I did go over to Cormac and Steph's to socialise and watch a bit of TV (and eat) after helping Keith move the raised beds Kris sold him over to his house. Sunday was pretty much completely filled up with rowing an eight 27K back from a city downstream of us. The people in the boat with me weren't the most skillful of rowers, but we got the boat back in one piece and even took it through a lock (with us in it). We stopped at two pubs (it'd've been one, but that one had had a power cut so there wasn't any food to be had, only crisps and drinks), carried the boat over another lock and got home in time for me to have a shower, snack and watch a recorded TV show with Kris over Skype before bed. Today I've converted a RHEL5 server to be a Xen host, installed and totally configured three Xen guests, resucitated a desktop box which wasn't booting, helped someone get mixed resolution dual-heading working and sorted out a SQL*NET connection thingy through the firewall and load balancer after being given red herring information. Now all I have to do is get home and to the gym before it absolutely pours down. I don't rate my chances very highly.

16/09/2011 [15:00] A much nicer last 24 hours. Delivery pizza last night, slept pretty well, came in to work and didn't have much to do, went for a very gentle 10K run, had lunch, now going home via the gym. Early departure due to an excess of hours worked this week in the name of service protection. This weekend I'll be doing parkrun, Tesco and rowing an VIII about 25K. Also, some drinks in at least one pub, a pub lunch and meeting my landlady. Oh, and very definitely Skyping with Kris at least twice. Because I love her, innit?

15/09/2011 [16:40] Even more relaxing today in terms of work. There was some fallout from yesterday's complete hash of an upgrade, but everything seems to be OK today. The big Java upgrade to the finance system resulted in just one person contacting the helpdesk to mention that they couldn't get in. This was down to the fact that they had no on-site sysadmin to upgrade the Java on their workstation. Hopefully someone will get to them soon. I got in this morning and straight away headed into town to collect the last four-but-two servers from the server room there. A quick taxi back to the other server room and they were racked up and running again within two hours. Since then I've installed the last of the new RHEL6 KVM guests I was tasked with doing and installed all the ancillary software they're supposed to require. I imagine that won't be all of it, but it's everything I can do for now until someone tells me differently. I've also learned a good bit more about KVM since beginning this project, not to mention the new version of Dell chassis I was handed. Last night, rather than being at the gym I was out with the Vets for a quick 10K row to the lock and back. I have to say, my hands (or one of them at least) weren't hugely happy to be gripping a blade again. I worry a little that this weekend's (gentle) 25K row of a boat back from a city just down the river could prove to be painful. Then again, I'm not above wearing a pair of gloves to give me some protection. 14/09/2011 [17:30] A much better day today. Came in, after a bit of frustration with the ZXTM box managed to get it rebooted. Since then everything seems to be working properly, which is good. Got some nice emails from other sysadmins to say thank you for pointing out out of date Java installations. After a bit of faff managed to get the downtime web page working for the service I was told it would be really good if I was in work for 07:30 to help with upgrading. Yes, I was in at 07:30 (07:10 actually) this morning. Mostly due to a combination of an absolutely awful night's sleep and dyed-in-the-wool desire to do a good job. Since all of that I've been watching the upgrade go slowly through, helping set up multipathd on a few servers and going for a 10K run at lunch time with a co-worker who isn't as fast as me thus making it a bit of a leisure run/jog, which was nice. The weather was pleasant so it was a good way to decompress from the previous 18 hours or so of stress and strife. Last night's dinner with Kate was nice (although the food was probably part of the reason my sleep was so disturbed) but cycling from work to home and then out to the Bedouin restaurant didn't help in me maintaining an un-harried appearance. Still, no matter. It's late again today, again. Only not quite so late as I have to get home and down to the boat house in time to be on the water at 18:15 with the vets. After that it's shower and home, dinner and chatting to Kris before I watch something fun on download and see if I can get a better night's sleep. 13/09/2011 [18:50] A frustrating (long!) but ultimately satisfactory day. I've been here too long but that's mostly down to me not being able to let something go once I've got my teeth into it. And despite getting help from someone else in the end (bless the Internet) I think I did at least 70% of the donkeywork myself. The task was writing a script which would pull out what version of Java people were using to access a major web-based application we have here. I'd come up with a set of commandline things to do to get the information but I wanted to script and parameterise it to make it more useful in the future. That's what took the brain power. It's done now and is throwing up some very interesting information which I hope to be able to use over the next few days. Next up was sending out a good few dozen similar emails to IT people to tell them that they had computers/users who were accessing the web application with out of date Java. What frustrates me is that since sending out the emails (and everyone having been exposed to a number of notices) about the version of Java they should be upgrading to I'm seeing lots of lines which indicate that they just threw on any old newer version of Java, rather than reaching the sub-version we explicitly mention. Unhelpful. And we're the ones who're going to get it in the neck in a few days when the server side of the upgrade goes out and suddenly people can't access the system. Chris and I laid four more fibre cables under the floor of the server room here this afternoon to connect up a few more database servers. It was good to get out of my chair a bit but it did take me away from the script mentioned above which has lead to me still being here at 18:50 in the evening! At about 16:00 I got an email from a developer asking if I could do a few very important things for his service tomorrow morning at 07:30. I have no issue with getting in that early, it's just that a bit more notice would have been nice, you know? As it it, before I go I'm going to have to remind myself where the things he wants done are located in the load balancer's web application. It's a good job my dinner date isn't until 19:30 or I'd be in real trouble now. Unfortunately the load balancers seem to be having some kind of (currently!) non-terminal strop every so often. I just hope they don't fall over completely before my team leader gets back and helps us work out where the problem is. 12/09/2011 [17:20] I'm late. I should have left for the gym 20 minutes ago but I've been busy writing up more KVM stuff and finding out new things about it. Anyway, yes, to summerise: achieved new 5K PB at parkrun on Saturday (19:38, hurrah!), went to Tesco, didn't go dragonboat racing (and really should have), talked to Kris twice, felt a bit lonely on Saturday evening and slept like a log on Saturday night. Was up early on Sunday to help move boats down to the part of the river the regatta was on. Set everything up (including dropping buoys in to mark the lanes. Rescued a few scullers, had some lunch, talked to Kris again, went back to do more rescuing, helped clear everything away again, went to the pub with people and then went home and to bed. Today I have mainly been reinstalling the machine I installed on Friday because I didn't check what the disk layout was beforehand (RAID0 rather than the RAID1 I'd planned on). Still, all that went pretty smoothly and now I have a KVM installation I'm going to try virt-cloning tomorrow to see if it all works out. I don't have much confidence though, so we'll see how that goes. 09/09/2011 [16:45] Busy day! I should be on my bike at this point heading to the gym so that I can be home in time to talk to Kris on Skype but I've had things to do which mean that I've only just got around to doing this journal entry. The very first thing I had to do when I got in was work out why the new RHEL6 KVM guests (well, just one of them, really) was spewing masses of errors and had meant that the server itself had been mostly inaccessible since midnight. I got into a bit of trouble for not having written any documentation about dealing with KVM instances, but I soon had everything up and running again. I hope so anyway. I think the bug was something to do with the kernel version I had everything running on. We've moved past that version now so hopefully everything should be OK now. I guess we'll see over the weekend. After that little bit of excitement there was a machine to move from this server room to the new one. Then I had to do some configuration stuff which... well, you don't care about that. Lunchtime and a run next, followed by getting the sister server for the one mentioned above up and running. Lots of notes to follow meant that it all happened a lot more quickly than the first one. However, there were some problems with the kickstart, specifically the booting of a more recent kernel than the one installed as part of the installation, as it were. I've decided I'm going to work on that before I leave for the day. Hopefully it won't take too long. Over the weekend there's parkrun, helping out with the rowing regatta and some Tesco shopping. Also a bit of television here and there. 08/09/2011 [17:00] Moved a few more servers and got them nicely squared away at the new server room. Also discovered two mounting kits for the 2540 storage arrays which I'd been trying to get from Oracle for the past few weeks. Hurrah for me looking through old boxes of mismatched cage nuts and similar. Now all we need is some downtime so we can pull out the array, stick in the shelves and put it back in again. Again, otherwise there's not a lot happening. I have a server to move to the new server room tomorrow from here (as opposed to the server room in town) and then an identical server to put in it's space to build with RHEL6, two KVM guests and then configure those with all manner of PHP modules and Ruby gems and things. Once that's done, other than the udual daily maintenance and things I'm not sure what'll be next on my schedule. I'm sure something'll come along eventually. I think I'll be out this evening for a bit helping put a landing stage in the river for this weekend's rowing regatta thing. That might make me feel like I'm entitled to get a pizza delivered for being so selfless. Not to mention the fact that I sold (probably too cheaply) my Axis 2100 network camera last night as I just wasn't using it for anything worthwhile. And paring down what I own feels pretty good. We do accumulate a lot of stuff in our lives, don't we? 07/09/2011 [16:50] I bit the bullet today and did a round of UPS client installations on all of the unix servers at the new server room. This amounts to about 34 servers so far and leaves very few slots left if we stop at the "recommended" fifty servers per event limit we've had mumbled to us repeatedly by the manufacturers. Still, not to worry. I'm also not worrying about the fact that despite being told by the web server page that you can put in hostnames (and there's a DNS server setting too) if you try to use hostnames (which are far easier to recognise than IP addresses) it doesn't work... ... Although I've just double-checked and it seem it does now. Which is really odd, but very welcome. I think my last job today will be changing all IP addresses for hostnames. I will then go to the gym. 06/09/2011 [17:25] Not much happening today. After yesterday's run and evening gym I decided that today should be a rest day. I didn't even do morning exercises, which was a bit odd, but rather good considering I didn't get to bed until midnight for various reasons. I've pulled the other Dell server I should be setting up soon out of storage ready to sort-of rack it up tomorrow and get an OS and some KVM guests on it. This will probably be the one which ends up going over to the other server room, but not until I've done something with it. Hopefully I'll be able to take it over in a car rather than rattling it there on a trolley. It's now just a touch under 71 days until I see Kris in person again and I'm really not enjoying the feelings engendered by being apart for so long. I know this is the way it needs to be, and in many ways it'll mean that we appreciate being together for significant periods of time so much more in the future (research trips notwithstanding)... but, I don't like it. I don't like it at all. Neither does Kris. It just doesn't feel right and it's so very much the exact opposite of what we hoped for for our first year as wife and husband. I'm just so glad that I managed to find someone who'll still want to be with me as much as I know Kris will do after the amount of time we've been apart. 05/09/2011 [16:20] Mostly this weekend I have been running. That's not strictly true, I did a parkrun on Saturday morning and a 12.8K run on Sunday morning and otherwise wasn't running at all. I did clean the house, watch some television and go to a party where I saw Andrea (freshly back from her gold and silver medals at the World Masters Athletics Championships in Sacremento, California) too. But otherwise it was running. And my first session at the gym I haven't been a member of for four years. I think my shoulders still ache, which just proves how long it has been since I did anything with those particular muscles. Today I have been finding out more about the UPS client software I have to work with, discovering via the magic of Skype that something broke a window in Kris' apartment and going for a nice 6K run at lunch time. I'm off to the gym as soon as I leave work this evening in the hope that that'll work the kinks out of my shoulders. Then it's weekend catchup time with Kris and dinner. This week should be moderately interesting, I hope, in terms of work and things. It'd be nice to get the UPS stuff squared away on the Solaris and linux boxes anyway. 02/09/2011 [15:05] Did stuff today. Not much. Gym tonight. Should be good. A weekend of more gym, Tesco, house cleaning and a parkrun (not in the order) to come. Now: switch installations and then home early! 01/09/2011 [16:25] Dude, seriously? September? What the hell? I swear that by the time I'm in my 50s years are going to pass by in a month or less. How can we be nine months into 2011 already? Oh well, I guess time seeming to pass more quickly does mean that I get to be with Kris again sooner. Anyway, today I was intending to do a lot of UPS client installs but it just didn't seem to happen. I'm not quite sure why. To be honest I don't really know where today has gone. I know I went for a 10K run at lunch which seemed to go pretty well other than a bit of calf muscle (not tendon) tightness, but otherwise... no, the day just seems to have slipped by. I am waiting on someone to come and see my Axis 2100 network camera with a view to buying it (I've even got it all set up and displaying an image) but they haven't turned up yet, which is annoying. There's a rowing AGM this evening which I feel like I should sort of go to, except that I'm slowly and quietly ducking out of rowing stuff these days as I rejoin the gym for the next six months (heading over there this evening to pay my membership fee). I will help with the regatta though (in a few weekends from now) as it's a moderately fun day out, potentially in the sun, and I might get to do some stuff in a motor launch. I think that's it for today. I'm not sure what I'll have for dinner this evening. Last night's thin crust pizza from Domino's was mostly OK (although I think the base was overcooked) but I really can't do that two days in a row given the vegetables I have in the fridge that need eating. Kris has her second day of teaching today. Her first in two classes. I know she's going to do really well, especially as one of the classes is pretty much standing room only. I'm looking forward to hearing how they went either tonight or tomorrow.