BOFHJournal - October 2011

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

September's Journal
November's Journal

31/10/2011 [13:25] And now suddenly it's the end of October. I can't say I'm disappointed as there's all kinds of good things lined up to happen in November. However before they can happen I've got to get through a full week of 06:45 starts at work patching machines. This morning's patching went almost perfectly except for discovering that the web team hadn't been moving with the times with regard to control Apache on RHEL6. That took me a little while to solve, but it felt good when I eventually worked out what was happening. The other patching went well, which gives me hope for the rest of the week and the two or three days over the following weeks when I'll be doing the same thing to many many other boxes. Tomorrow's going to be the worst day though, unless everything goes right. I am updating two servers' ILOM/BIOS firmware and thus far fully 50% of the machines I've attempted this one have bricked in one way or another. One had to have its motherboard replaced. So tomorrow - glutton for punishment that I am - I'm attempting two more updates. On our live application/DB server pair, no less. The other server updates should (should) be a straightforward update, reconcile configuration files and reboot. We'll see. Over the weekend I did parkrun (3 seconds off my PB, dammit) and went for a long run on Sunday morning which slightly upset one of my knees. As a result I didn't run at lunchtime today. Also this should help me put up a better performance this evening at the gym with Tom (an AR friend) who it turns out goes to the same gym. Admittedly he's taken some time off after having surgery for a collapsed lung (a week or two ago) but he's also taller and I think stronger than me, so we'll see. The good thing about getting to work at 06:40 every day is that I get to go home early. I'll be doing that today so I can pop to the doctor before I head to the gym. Nothing to worry about, I just need some information on immunisations. But speaking of which, I also saw Contagion at the cinema with Cormac and Steph after renewing my rail card on Saturday. Never have I been so paranoid about not touching public surfaces as I was after seeing that. I think that's everything apart from mentioning a slew of good Skype chats with Kris. She's awesome. I may have mentioned that previously. [15:55] It's the most rare of days; one where I do two updates. The reason is that my Very Important Letter has finally made it to its destination in the US. So I'm both happy and relieved. Now all I have to do is wait, again. Anyway, time for me to head off!

28/10/2011 [16:30] I've got more machines scheduled for updates next week. Every day I think I'm doing at least six and as many as fifteen. The only positive aspects to being in before 07:00 every day is that I get to go home early and all my servers will be fully patched. Something that hasn't been the case in quite some time. I should make sure I get to bed early every day, too. I guess unless I want to get up really early it's going to be a week without morning exercises, too. Speaking of exercise, I was all set to do a fast 6K at lunch time as I did on Monday and Wednesday when I met one of my fellow work runners just as she was about to head out. So we ended up doing a 10K instead. I have to admit it felt pretty much like a jog from my point of view, but I put down a bit of speed on the final up hill as we approached the end. Hopefully the combination of a tiny bit of speed work and a nice long run will see me in good stead for tomorrow when, for the first time, I'll be turning up actively hoping to get a personal best. We'll see. Otherwise the weekend has the usual shape of parkrun, Tesco, home, stuff I haven't worked out yet, bed, some kind of long distance exercise, breakfast, lunch, stuff I haven't worked out yet, hopefully Skype with Kris who should be back from Kansas City and then bed. Then it's the week of early mornings! Work? Yes, I did some work. Also had to go over to the other server room to let an electrician rewire all of the meters over there so they actually showed what we were using, rather than what was coming in. Or something like that. Switching wires around anyway. Oh, and last night's pizza was probably a bad idea. Not only did it take over an hour to arrive but it also was probably a bit too greasy/cheesy/meat-loaded and as a result today I've been feeling a little less than optimal. Hence going for the longer run and stuff. Also, gym tonight. Then I will try to eat a few more vegetables than usual, and drink more water. Not that I've drunk any today, and I should have.

27/10/2011 [17:30] From not being able to get any machines scheduled for updates I now appear to have pretty much 80% of them scheduled to do next week. This means for an entire week (I'm guessing Friday will also be used eventually) I'll be getting into work before 07:00 to patch and reboot servers. The only day I'm actively worrying about is the Tuesday as that's the day I'm also doing two firmware updates and I can practically guarantee that at least one if not both of them is going to brick when I do it. I'll be taking every precaution I can including shutting down each box and cold booting it before I do the actual firmware update. I don't think this'll help, but it will make me feel better or that I tried everything. Then I have to get four other servers patched and rebooted. Or maybe I can do them while I'm fretting over the boxes I have to firmware update. Probably. It's not until the 17th that I have to do ten servers in one go (including a live DB box). By that time I imagine it'll all be going perfectly... yeah. So yes, 31 Oct to 3 Nov is going to be pretty 'exciting' in the mornings. In other 'exciting' news the post that I set to the US appears to be going very much the long way to its destination. Here's hoping it's just the way post is routed, rather than an actual mistake. I had some other worries about other bits of bureaucracy but Kris has sorted it all out. Because she's amazing, that's how. Workwise I've been able to do some updates today, which has been good as it cuts down on the mass I mentioned above. Not much time for anything else other than working with DBA/developers to get new versions of Oracle products intergrated into the systems we have here. Sometimes I forget that not everyone knows about boot sequences and rc-chains and the like. I was going to go to the pub tonight but frankly I'm just a bit too tired to consider cycling out and then cycling back. I may even be too tired to cook (pizza!), but I'm not sure yet. I did just get paid again (the fools!) so maybe I can treat myself.

26/10/2011 [15:50] Started the day off at 06:45 getting to the other server room to wait for a UPS engineer who apparently had called me at that time to say he'd arrived (only he was in the wrong place). I didn't check my phone until 07:10 at which point all I had was an answerphone message of a loud sigh. Once I'd called him back I got to talk him down/round to the right entrance to the site and watch him do his maintenance on the UPS. After he hadn't left a message on my phone he called one of my colleagues so he ended up turning up too, unnecessarily. Once that was all sorted out I came into work and helped deal with the fingerfumble self-same colleague had which wiped out some SAN flash copies that were mounted on one of our database servers. This kernel panic'd the server so we then spent the morning watching it fsck some rather large volumes. The lunchtime run felt leaden. I'm not sure why. The new showerhead someone bought turns out to be too heavy for the already-damaged angleable showerhead mount, so showering required you to hold the showerhead with one hand while rinsing off. Apparently someone's temporarily fixed this with elastic bands while we wait for someone to get a new mount. Mostly this afternoon I have been reading about the new version of the load balacing software we us. No major updates to the version I just installed yesterday other than a complete rebranding and (most importantly) a fix to make the web GUI much more responsive when you have a lot of virtual servers, traffic IPs and pools. We may have to wait until 8th December to upgrade again, though. Having been in so early I'm going to leave at 16:00 today, do a quick gym session and then shower, chat to Kris and do some washing up. Almost certainly in that order. Then dinner.

25/10/2011 [16:15] Lots of learnings about Puppet today. I may have a go at some of the examples tomorrow as well as rereading the entire manual/examples/help again to get it a bit more bedded in. Whether or not I/we start using it here is beside the point. I think it's looking more and more like something that's worth knowing. If nothing else it exercises my learning muscles. Something I haven't had the opportunity to do in a while. I also took a quick look at Pulp, but it doesn't look mature enough for us, nor do we have a server spare (with the disk space required) to run with it. Puppet though could be something I could replace a lot of the post install Kickstart setup with and have it make things all clever when it came to different project/service's different tiers of servers. Especially with the slight differences 32 Vs 64 bit makes to some things. As I was in early this morning upgrading the load balancers I'm heading off nowish. I'll be going straight to the boat house to do a tiny bit of work for Keith and then heading home to make an attempt at plum chicken tempura (there wasn't any lemon sauce at the Tesco I went to) with egg-fried rice with salad onions. I'll let you know how it turns out tomorrow. I'm pretty glad I'm not exercising today (and didn't do morning exercises either) as the run and last night's gym pretty much wiped me out by the end of the day.

24/10/2011 [17:00] So over the weekend I had a pretty good time. I appear to be going backwards at parkrun, which is a bit annoying. Slower again than last week. I'm hoping to do better next time by pushing myself back to the training I was doing in the weeks leading up to my run of two back to back personal bests. Either way I'll be fitter, if not also faster. After parkrun it was the usual Tesco trip, then after a quick bite of lunch the train to Reading via London for Dunk's rehousewarming. I don't know what it is at the moment, but I don't seem to be able to get into reading regularly at home. Which is probably why I had such a great time on the train(s) on Saturday with over two hours of uninterrupted reading. The party was brilliant with a good few people I haven't seen in years and years turning up. Staying overnight wasn't too bad either as I had a whole room and an air mattress to myself. Breakfast while watching the second half of the rugby and then trains home again (more reading!) were also great. It was definitely a good idea to get out of the house (and town) for the weekend, as well as not doing any real exercise on a Sunday for once. That plus the massive amounts of pizza and soft drinks I had on Saturday evening was definitely a break from the usual routine of exercise and food I've been trying to stick to. The remainder of Sunday was spent on Skype with Kris as well as a tiny bit of house work, dinner and a moderately early bed. Today has been mainly catching up on email and stuff, reading about Augeas (for configuration management, not any other definition), a hard (pushed myself somewhat) 6K run and then an afternoon of mostly more of the same. Tomorrow morning I'm in early to upgrade the Zeus load balancers to the most recent release of ZXTM (or whatever it's called now) and then ponder the movement of a piece of mail through varied postal systems. Gym shortly, then home to collapse and catch up on things at home.

21/10/2011 [16:25] I didn't really know what to do with my run today. Shuold I do a 10K at a gentle pace, 8 or 6K or what? In the end I did a 6K, but pushed moderately hard the whole way around, did a fake push and finish at the 5K mark and then ran hard for the final K back to the office. Hard, but not 'completely wasted for the rest of the day and parkrun' hard. Work has mainly been about learning lots about dstat and the RHEL5 kernel when it comes to I/O wait. That's about it, really. After last night's gym session which left me not wanting to go back out to the pub once I'd showered and even glanced at the sofa I'm not even going home this evening but heading straight out to the pub to meet people. I'm taking a book and will ensconce myself in a snug corner and wait for people to arrive. Tomorrow is parkrun, Tesco and then in a change to the usual schedule I'm off to Reading for a friend's house rewarming and won't be back until Sunday. As a byproduct of this there'll be no exercise on Sunday, which is probably for the best. Once I do get home though there'll be a long old Skype session with Kris. Because some things are important.

20/10/2011 [17:00] Mostly today I have been doing things with stuff. Not particularly interesting, but then it's better than not doing anything at all. Last night's dinner with Kate was pretty good. An Orange Wednesday code coupled with a free bottle of prosecco meant that a three course meal for two with alcohol came in at just over a tenner each, with just over thirty-three pounds of 'savings'. And we were in the good room of the restaurant, too. Oh, I also went into town at lunch time and posted something very important. Track and Trace and everything. Now I just have to wait. Kris posted something too. I guess we're both waiting, now. Gym tonight, then probably the pub for a little while. Although people are going tomorrow too and I'm not sure I can afford two trips in a week this close to the wrong side of pay day.

19/10/2011 [17:05] Today I have been mostly reading a lot about Puppet. Not doing any Puppet stuff, just reading about it. It's quite interesting, really. I may get around to doing something with it before the end of the week, who knows? Today's run was good. Pushed just a little harder than normal, I think. Coupled with last night's slightly harder than normal gym session I think I've peaked in the middle of the week quite well. No gym this evening as I'm out celebrating Kate's birthday early (it's next week) with Pizza Express and free wine. After that I'm home to look through a packet of documents for the last time which I'll be posting tomorrow. Tomorrow's work day should be a little more interesting than today, I hope. If it isn't I'll try to find something new to do which makes it so. That's just the kind of guy I am.

18/10/2011 [16:45] Didn't go to the gym last night. I figured that running at lunch time and generally feeling quite tired was a good enough reason not to. That and having discovered x11vnc for the first time and installing it on Shaun's new work station and generally getting getting it ready for today was a better use of my time. As a result I'm off to the gym this evening instead. Plus the weather looks a bit nicer for being out later today. Not much happening at my desk today work-wise. I was hoping to get some more Puppet learning in, but that didn't happen due to helping Shaun with his new computer and then some other minor niggly stuff to do with version 3 firmware ILOMs and recent web browsers. Suffice it to say it was a good idea to keep the Firefox 3.6.x installation that comes with RHEL5/6 otherwise there'd be no way for me to access any version 3 ILOMs which need updating or their information viewing. Booked tickets and hotel for Paris last night, which was great. Now all we have to do is wait. Also drew Kris a new picture (as requested in a note she left me a few days ago). She seemed to like it. Which was good.

17/10/2011 [16:40] It was an OK weekend, really. The weather was fabulous, which meant that even after running both mornings I felt I should get out in the afternoons rather than let the days waste away. So, after getting one second away from my PB at parkrun and then going to Tesco and stuff I ended up getting Cormac to come out and cycle 12 miles or so, which was nice. However, on Sunday, after running about 14K I just couldn't be bothered to go out again and wasted the day away looking for missing Euros and watching television a bit too much. I still got my exercise though, so that's OK. Spoke to Kris a lot too given that she was online both days doing yoga teacher training which started at 03:30 her time. That can't have been pleasant, even for someone who routinely gets up at 05:00 to start writing. Before I met her I thought waking up at 06:00 and then getting up at 06:30 to exercise was an early morning. Today I have been mostly trying to discover why logging into a Solaris box from a RHEL5 workstation with a certain Intel graphics chipset and attempting to run Oracle's "runInstaller" caused X to crash. I seem to have narrowed it down to something to do with the chipset in Dell Optiplex 620 chassis and the "intel" driver. As such I'm right now preparing a new workstation for someone (and upgrading to RHEL6 for them too) so they can actually do their job. Note that there are no problems running the same installer when SSHing to a linux server. Gym tonight, then purchasing Eurostar tickets with Kris on Skype.

14/10/2011 [16:10] Had a lovely run at lunchtime in the sun. Pretty slow though, given that I was pacing for someone else quite a bit slower than I. The morning and afternoon worky things have been mainly about trying to get a data import to work cleanly by updating tables and then propagating those changes to the right views and this afternoon, well, not much. Kris is having stuff delivered today, so I'm watching it on Skype before I go and vote. No gym tonight as I went last night and then had an brilliant pizza which was far cheaper than I thought it was going to be. This weekend I'll be parkrunning, probably rowing with the Vets again and doing the usual Tesco thing. I may even enter a ten mile race (enter, not run this weekend) if I can track down some transport there and back.

13/10/2011 [17:00] Another connection made which could yield positive things in Q1 next year, the first faltering steps on installing and configuring puppet and a whole lot of tiny boring stuff which I really could have done without. That was my day. I'm going to the gym now because I was just too tired and unenthused to go yesterday. Then there will be takeaway!

12/10/2011 [16:30] Stuff, things, more stuff, a few more things. That's about it really. Oh, and two servers that I'm going to be updating the ILOM/BIOS firmwares of in a few weeks have apparently pre-crashed their ILOMs (GUI access anyway) such that even a reset of them won't give me buttons and things so as to be able to perform a firmware upgrade. Brilliant! Honestly, other than a run at lunchtime (still with the 6K of intervals) I have nothing else to report other than having to go to the gym via the other server room to let someone from Estates in to read some meters, which is why I'm leaving now.

11/10/2011 [17:35] Arrive at work early, remove network cables from one server after cleanly shutting it down... remove power from server above. Nicely done, sir. Luckily the server I brought down accidently was the web server for the application server I'd shut down to replace the RAID cache battery on. No harm done. That was the highlight of my day. Otherwise it has just been a succession of small jobs and things. In other news Kris and I have completely booked our 1 year anniversary trip now, and have designs on a short trip to Paris too. Good to know that there's an up side to living a long way apart and therefore barely spending a penny on anything outside of food and utilities.

10/10/2011 [16:50] An exercisey weekend beginning with another personal best for 5K at the park run (19:23). Given I woke up feeling horrible, achey and generally not in the mood for running, and attending the run feeling like I was just going to pootle round and get a slow time, just to show willing by being there at all it was a very pleasant surprise. After that it was a cycle to Tesco and home, some lunch and a chat with Kris. I have to admit that I didn't actually do much exercise for the rest of the day (unless washing up counts). It rained on and off so it was nice to stay in the house and catch up on things. Sunday morning I was out rowing with the Vets. Having had a terrible night's sleep due to back muscle discomfort (still, but mostly gone now) I wasn't looking forward to rowing. However, it was all fine, the rain stopped just as I got to the boat house and in the end I didn't need any of the extra layers I'd brought along once we were moving. It was a pretty good outing for the most part too. The vet boat is almost always very well balanced, which is nice. I hung around with people afterwards for coffee and cake before heading home for some more television catchup and then two and a bit hours on Skype with Kris where we pretty much sorted out all of our Christmas travel plans (train tickets included) as well as my parents' Christmas presents. We also decided on where we'd go for our first year wedding anniversary. Today I have mostly been preparing for tomorrow's early morning RAID card work (just replacing a battery this time, rather than bricking a box with a firmware update) and getting terribly frustrated with a developer who seems to be able to discern incredibly tiny differences in his screen fonts and complains about them even though there doesn't seem to be much I can do about getting them 'fixed'. Hopefully I'll be able to do something about it once he's had another registry edit added. Who knows. Gym tonight, then a quick Skype with Kris again about car hiring and a trip to Paris at some point.

07/10/2011 [17:10] The most important thing I solved at work today was getting QuickTime movies working in my browser again. In the end it came down to the newer version of Firefox I was using checking an mplayerplug-in.conf file that the older (RPM-packaged) one apparently did not. This was why the plugin (mplayerplug-in-qt.so) worked when anyone else (root) fired up the same browser as they didn't have an mplayerplug-in.conf file to check, so everything was default enabled. In mine the plugin I was having trouble with was marked as disabled (enable-qt=0). This lead to it being dumped into the "[INVALID]" section of Firefoxe's pluginreg.dat. In the end it was using trace to see what files Firefox was reading that solved it. Definitely something to remember if something like this happens again. Hopefully there are enough keywords in here to make anyone else Googling for a solution's day. Ran at lunchtime. It was an astonishingly liberating experience today for some reason. I can't explain why. I set out intending to do a gentle 6K and ended up blowing through 10K and not feeling too bad at the end of it. Either I'm getting fitter or I didn't push too hard. Either way it felt wonderful. Just a feeling of freedom and happiness at being able to run for forty-five minutes or so without needing to stop, slow down or generally like I was going to die. Being fit and able to be active is a privilege, and one I intend to embrace, utilise and improve upon until I can't any more. And now, the weekend, beginning with the gym, then parkrun, Tesco, some lazing around, rowing on Sunday and maybe reading my two new books bought with an Amazon voucher converted from Nectar points.

06/10/2011 [17:25] The majority of the day today was spent installing updated UPS software on all the Solaris and linux servers we have in this building. That and fine-tuning the installation wrapper script to deal with all the slightly different configurations we have for networking. I'm reasonably happy I've got it working for pretty much 19 out of 20 setups. This afternoon mostly I have been doing forms. And now I am going home. I may go to the pub and see people, but I'm not sure. I could go and do circuits but I'm not sure I can be bothered this week. Maybe next week, and then the pub. Yes. No pub, no circuits this week. No pub means not spending money, which would be a good idea.

05/10/2011 [18:10] Is it wrong that I've stayed at work an extra hour just to write UPS documentation? Yes, I guess it is. Went for a run today (6K), did some good work, talked to Kris (she's awesome) and tried not to eat cake (not terribly successfully).

04/10/2011 [17:25] Ended up not going to the gym last night. I just couldn't face the idea, and my lower back was still aching a little from what I guess was the heavier weights session I did last Tuesday, plus the heavy lifting I did the next morning. I'll be going on the way home from work tonight just as soon as I finish this journal entry. For some reason the graphical menu thing on bofhcam.org no longer seems to work properly. Considering it was old and busted (figuratively) five years ago let alone now I've slimmed it down to something a little more streamlined. I don't imagine you'll mind (or notice). All the same links are there, just not as oddly displayed. Kris made it back to America OK. Which, other than her not being here, is good news. She's been moving the apartment around in terms of what room is for what. I think it's a good thing, but I'm waiting for the photos to be sure. Lastly, it looks like one of my coworkers has slightly (read: totally) broken the (fos)wiki installation we have which has meant I haven't been able to do the documentation for the new UPS hard/software and he's still here after he would have normally gone home. And now I'm going home and leaving him to it.

03/10/2011 [17:50] It has been really very wonderful few days. Kris arrived on Wednesday and since then we've been to London to see the Tower, the Aquarium, a Wagamama (yeah, OK), outdoor activities shops, traveled a fair bit on our bikes to see places, spent some time relaxing at home, seen people for curries and pub lunches (more like pub afternoon meals) and watched a few shows that we've always liked to see together but haven't been able to. All wonderful stuff really. She left this morning in a taxi at 02:30 and somehow I managed to get back to sleep. Since then it's been back to work as normal for the next few weeks until she's here again (for a little longer that time). Went for a good run at lunch time today. Have decided to not go to the gym this evening to see if I can give my back another 24 hours to not be quite so stiff when I stretch in certain directions... Or maybe I will go, but just not for very long. I don't have anyone to be home for right now.