[16:15] Well, it hasn't been the quietest day I could have hoped for. The
server-and-storage move this morning went perfectly well apart from an
apparently failed battery on one of the controller cards (which Oracle say is
just a false-positive caused by the firmware revision it's on (what is it with
RAID controllers, batteries and firmware and me at the moment?)). Having
logged it with Oracle I'm hopefully that my co-worker will deal with it
tomorrow or Friday while I'm on vacation. Otherwise I've been doing a few
bits and bobs here and there, popping home for lunch to see Kris(!) and
looking out of the window and wishing I was outside. Going over to the new
server room to help rescue someone's lost swipe card was a nice way to use up
half an hour and be outside, though.
I'm aiming to get out of here in about ten minutes and see what kind of quiet
mischief Kris and I can get up to before she falls asleep in my arms because
of jetlag. See you on Monday when I get to tell my line manager about
everything that's gone on in the last three weeks.
[17:15] Today I have mostly done everything. Every single thing. That's
right. Among the things I did today were: get up and into work for 06:30,
firmware patch two machines, watch as one machine bricked the BIOS, opened a
P1 call with Oracle, had some instructions to reset the CMOS on the BIOS,
augmented these instructions with repeated power cycled and a cold reboot,
eventually got the server working, worked with Oracle to (not, as yet) get
some resolution on why fully half of our firmware updates are failing in some
way which gives me a raised blood pressure, went for a 6K interval run, did a
crapload of load balancer configurations including some pretty hairy response
ruleage, sorted out some firewall issues and even managed to send a few emails
to Kris (hopefully before she starts traveling).
All I have to do now is make to the gym and home alive and today will have
been something of a Big Day. Of course, there's also the small shopping trip
I need to do to make sure there are nice things for Kris to have for lunch
tmorrow, but otherwise I think I'll then be done! Phew.
Not that tomorrow is going to be much lighter in terms of things to do. It
just has a happy end.
[17:25] It got to this time without me noticing. Which is both good (as it
means I was getting stuff done and time ticks on to when Kris comes back) and
bad as I've had a devil of a few days, work-wise. If we rewind to Friday
night I went to the gym, and then headed out to watch a film over at a
friend's house where there was bacon and pancakes and maple syrup. It was
quite a long film so I didn't get home until some time after ten. I jumped
into bed so that I could be awake in time for parkrun.
Saturday's parkrun was pretty good. I wasn't sure I'd done enough to get a
personal best, but apparently I managed it, by six seconds. Which was nice.
After that it was just a case of going to Tesco and then going home to snack
and chat to Kris. I know I did something else on Saturday, but for the life
of me I can't remember what it was... Oh no, of course I know what I did
next. I went into work to firmware update two servers. One worked perfectly.
The other went so badly wrong that the ILOM part of the server completely
bricked itself. Which was nice. In the end I left at around 18:30 having
opened a call with Oracle to get the thing replaced on Monday.
Sunday morning I got up and decided to go for a nice long run. So I did.
About 16K of such. It would have been longer but I ran out of energy and had
to detour by the boat house to get some water before jogging home. Next time
I'll do better. I slobbed for a bit of the middle of the day and then headed
out to the cinema with Cormac before grabbing a burrito (chicken, I'm trying
to be a little healthy) and then bed.
Today I was trying to fix the issue I had over the weekend, unsuccessfully,
and worrying that the wrong part had been sent out by Oracle (it hadn't). The
engineer turned up on time, we got the part replaced (the whole motherboard)
and the server back up and running just within the two hour downtime period I
kind of hadn't quite asked for. Still, all appears well now. This afternoon
I have mainly been checking all the scripts and files I'll need for tomorrow
morning and adding in little tweaks to the shutdown scripts for the new UPS
I'm pretty certain something is going to go wrong tomorrow no matter how hard
I plan, but I've covered every eventuality and have notified everyone that I
can of what can and could (and should) go wrong (and right). All I can do now
is do my job tomorrow morning just before 07:00 and hopefully finish before
I'll probably try and update this before the end of the day tomorrow so you
have some idea of how my morning went. Of course, if I don't that probably
means it either went really badly, or I just forgot.
[16:40] One of those mornings when you have to concentrate continuously for
three hours at a time to make sure you don't miss anything. That's what I had
today. I was doing DNS, firewall and load balancer stuff all at the same time
as well as generating new SSL certificates and writing notes on everything so
I could send an email about everything I'd done, just in case my team leader
decided to take over everything I'd done when he got back. It was hard, but
very rewarding eventually to know that I'd got everything done (as far as I
can tell) in one fell swoop and now it's just a case of someone else bringing
up the servers underneath the neat and tidy service infrastructure I'd built.
Here's hoping I didn't miss anything or misconfigured anything due to a lack
of information as to how everything is actually supposed to hang together.
After yesterday's complete washout in terms of exercise and this morning's
bout of high concentration it was a real pleasure to get out in the frankly
lovely weather and push myself for 25 minutes or so. It did wonders for
clearing my head and body of mental and physical tension.
I've also just had an email from my mother about her second eye operation. I
think I'll just go with "not as easy as the first one" and leave it at that.
This weekend I will actually be in work for a few hours (hopefully no more
than two) doing firmware updates on an application server and its database
server. Given it's the development stream I shouldn't be affecting anyone.
Other than that there's parkrun, Tesco, chatting to Kris and maybe some other
exercise here and there if I can be bothered.
[17:15] Most of today I have spent in the other server room with engineers
trying to get the maglocks to work. This was dull. It meant that I missed
going for my run and that I didn't get to have lunch until gone 14:30. I now
have to decide if I'm going to go home and go for a run. Or go home and then
to the pub and see people. Or just go home and slob. I'm really feeling like
I'm somewhere between the first and third options.
Spending half an hour trying to figure out where the timeout was I needed to
increase on the load balancer to allow a set of long reports to run without
connections to the back end web node timing out wasn't the way I wanted to end
the day either.
Still, last night's double episode downloaded television viewing with Kris
worked pretty well, which was nice. Although I'd have much preferred to have
her sitting next to me and not watching anything than the way it was. She's
wonderful, and even though I told her not to she's getting me dried fruit.
Now that's love.
[17:00] Urgh, what a day. A mild telling off by my line manager for
advocating using "> " rather than different text colours for quoting emails
in replies, then lots of confusion over what to do about a new set up of
Oracle Discoverer, then another RAID cache battery apparently failed and I had
to spend lunchtime doing a firmware upgrade of the ILOM/BIOS which, oddly
fixed the issue. This now means we should be doing the same thing to about
two dozen other servers (currently two others are showing the same issue).
The chances of anything other than three of the servers I listed in my
notification email getting done before Christmas are pretty much zero though,
which is frustrating. Not to mention the fact that this update would be the
perfect time to do the RHEL5.6 to 5.7 update which has been pending for
months, too. That'd limit things to one downtime instance, too.
There was a confusion with meeting the UPS engineer this morning (it wasn't me
who was doing the meeting) so that's going to have to be rescheduled and I'm
meeting other engineers at the new server room tomorrow morning for 08:00,
which means a slightly earlier rise time. I may skip morning exercises just
so I can be there in plenty of time. Gym tonight though, then some Skype/TV
time with Kris!
[17:10] Mostly today I have been trying to figure out why things weren't
working. It was lots of things and you don't really care enough to have me
explain what they were. Suffice it to say pretty much all of them were
working by the end of the day (i.e. now). I ran at lunch time. A nice set of
intervals (100 paces on (fast/unsustainable, 100 off (jogging), repeat) for
6K. I was definitely feeling it by the end, which was good in combination
with the slightly heavier weights session I did last night.
To be honest I think that's it for now. Not much else happening right at this
moment. Also not sure what I'm going to have for dinner tonight. I have this
feeling that something is supposed to be going on this evening but for the
life of me I can't remember what. And there's nothing on the calendar either.
[17:10] Lots went on this weekend. Top of the list was Kris deciding that she
just couldn't wait another fifty days for us to be together for a bit (and I
mean, who can blame here, look at me) and booking a flight back to England for
a few days from the middle of next week. Of course, the day she arrives I
can't get off work as I'm the only person in the office as well as doing some
important server moves, but otherwise for the rest of the week I'm
gone and spending time with my wife!
Saturday I was out doing parkrun again. Although the humidity was well down
on the previous week and I actually felt fitter and fresher I wasn't as fast
or faster than the week before, which was a bit of a shame. After that I
decided I was just going to go to Tesco and then relax for the rest of the day
although I did go over to Cormac and Steph's to socialise and watch a bit of
TV (and eat) after helping Keith move the raised beds Kris sold him over to
Sunday was pretty much completely filled up with rowing an eight 27K back from
a city downstream of us. The people in the boat with me weren't the most
skillful of rowers, but we got the boat back in one piece and even took it
through a lock (with us in it). We stopped at two pubs (it'd've been one, but
that one had had a power cut so there wasn't any food to be had, only crisps
and drinks), carried the boat over another lock and got home in time for me to
have a shower, snack and watch a recorded TV show with Kris over Skype before
Today I've converted a RHEL5 server to be a Xen host, installed and totally
configured three Xen guests, resucitated a desktop box which wasn't booting,
helped someone get mixed resolution dual-heading working and sorted out a
SQL*NET connection thingy through the firewall and load balancer after being
given red herring information. Now all I have to do is get home and to the
gym before it absolutely pours down.
I don't rate my chances very highly.
[15:00] A much nicer last 24 hours. Delivery pizza last night, slept pretty
well, came in to work and didn't have much to do, went for a very gentle 10K
run, had lunch, now going home via the gym. Early departure due to an excess
of hours worked this week in the name of service protection.
This weekend I'll be doing parkrun, Tesco and rowing an VIII about 25K. Also,
some drinks in at least one pub, a pub lunch and meeting my landlady. Oh, and
very definitely Skyping with Kris at least twice. Because I love her, innit?
[16:40] Even more relaxing today in terms of work. There was some fallout
from yesterday's complete hash of an upgrade, but everything seems to be OK
today. The big Java upgrade to the finance system resulted in just one person
contacting the helpdesk to mention that they couldn't get in. This was down
to the fact that they had no on-site sysadmin to upgrade the Java on their
workstation. Hopefully someone will get to them soon.
I got in this morning and straight away headed into town to collect the last
four-but-two servers from the server room there. A quick taxi back to the
other server room and they were racked up and running again within two hours.
Since then I've installed the last of the new RHEL6 KVM guests I was tasked
with doing and installed all the ancillary software they're supposed to
require. I imagine that won't be all of it, but it's everything I can do for
now until someone tells me differently. I've also learned a good bit more
about KVM since beginning this project, not to mention the new version of Dell
chassis I was handed.
Last night, rather than being at the gym I was out with the Vets for a quick
10K row to the lock and back. I have to say, my hands (or one of them at
least) weren't hugely happy to be gripping a blade again. I worry a little
that this weekend's (gentle) 25K row of a boat back from a city just down the
river could prove to be painful. Then again, I'm not above wearing a pair of
gloves to give me some protection.
[17:30] A much better day today. Came in, after a bit of frustration with the
ZXTM box managed to get it rebooted. Since then everything seems to be
working properly, which is good. Got some nice emails from other sysadmins to
say thank you for pointing out out of date Java installations. After a bit of
faff managed to get the downtime web page working for the service I was told
it would be really good if I was in work for 07:30 to help with upgrading.
Yes, I was in at 07:30 (07:10 actually) this morning. Mostly due to a
combination of an absolutely awful night's sleep and dyed-in-the-wool desire
to do a good job. Since all of that I've been watching the upgrade go slowly
through, helping set up multipathd on a few servers and going for a 10K run at
lunch time with a co-worker who isn't as fast as me thus making it a bit of a
leisure run/jog, which was nice. The weather was pleasant so it was a good
way to decompress from the previous 18 hours or so of stress and strife.
Last night's dinner with Kate was nice (although the food was probably part of
the reason my sleep was so disturbed) but cycling from work to home and then
out to the Bedouin restaurant didn't help in me maintaining an un-harried
appearance. Still, no matter.
It's late again today, again. Only not quite so late as I have to get home
and down to the boat house in time to be on the water at 18:15 with the vets.
After that it's shower and home, dinner and chatting to Kris before I watch
something fun on download and see if I can get a better night's sleep.
[18:50] A frustrating (long!) but ultimately satisfactory day. I've been here
too long but that's mostly down to me not being able to let something go once
I've got my teeth into it. And despite getting help from someone else in the
end (bless the Internet) I think I did at least 70% of the donkeywork myself.
The task was writing a script which would pull out what version of Java people
were using to access a major web-based application we have here. I'd come up
with a set of commandline things to do to get the information but I wanted to
script and parameterise it to make it more useful in the future. That's what
took the brain power. It's done now and is throwing up some very interesting
information which I hope to be able to use over the next few days.
Next up was sending out a good few dozen similar emails to IT people to tell
them that they had computers/users who were accessing the web application with
out of date Java. What frustrates me is that since sending out the emails
(and everyone having been exposed to a number of notices) about the version of
Java they should be upgrading to I'm seeing lots of lines which indicate that
they just threw on any old newer version of Java, rather than reaching the
sub-version we explicitly mention. Unhelpful. And we're the ones who're
going to get it in the neck in a few days when the server side of the upgrade
goes out and suddenly people can't access the system.
Chris and I laid four more fibre cables under the floor of the server room
here this afternoon to connect up a few more database servers. It was good to
get out of my chair a bit but it did take me away from the script mentioned
above which has lead to me still being here at 18:50 in the evening!
At about 16:00 I got an email from a developer asking if I could do a few very
important things for his service tomorrow morning at 07:30. I have no issue
with getting in that early, it's just that a bit more notice would have been
nice, you know? As it it, before I go I'm going to have to remind myself
where the things he wants done are located in the load balancer's web
It's a good job my dinner date isn't until 19:30 or I'd be in real trouble
now. Unfortunately the load balancers seem to be having some kind of
(currently!) non-terminal strop every so often. I just hope they don't fall
over completely before my team leader gets back and helps us work out where
the problem is.
[17:20] I'm late. I should have left for the gym 20 minutes ago but I've been
busy writing up more KVM stuff and finding out new things about it. Anyway,
yes, to summerise: achieved new 5K PB at parkrun on Saturday (19:38, hurrah!),
went to Tesco, didn't go dragonboat racing (and really should have), talked to
Kris twice, felt a bit lonely on Saturday evening and slept like a log on
Saturday night. Was up early on Sunday to help move boats down to the part of
the river the regatta was on. Set everything up (including dropping buoys in
to mark the lanes. Rescued a few scullers, had some lunch, talked to Kris
again, went back to do more rescuing, helped clear everything away again, went
to the pub with people and then went home and to bed.
Today I have mainly been reinstalling the machine I installed on Friday
because I didn't check what the disk layout was beforehand (RAID0 rather than
the RAID1 I'd planned on). Still, all that went pretty smoothly and now I
have a KVM installation I'm going to try virt-cloning tomorrow to see if it
all works out. I don't have much confidence though, so we'll see how that
[16:45] Busy day! I should be on my bike at this point heading to the gym so
that I can be home in time to talk to Kris on Skype but I've had things to do
which mean that I've only just got around to doing this journal entry. The
very first thing I had to do when I got in was work out why the new RHEL6 KVM
guests (well, just one of them, really) was spewing masses of errors and had
meant that the server itself had been mostly inaccessible since midnight. I
got into a bit of trouble for not having written any documentation about
dealing with KVM instances, but I soon had everything up and running again. I
hope so anyway. I think the bug was something to do with the kernel version I
had everything running on. We've moved past that version now so hopefully
everything should be OK now. I guess we'll see over the weekend.
After that little bit of excitement there was a machine to move from this
server room to the new one. Then I had to do some configuration stuff
which... well, you don't care about that. Lunchtime and a run next, followed
by getting the sister server for the one mentioned above up and running. Lots
of notes to follow meant that it all happened a lot more quickly than the
first one. However, there were some problems with the kickstart, specifically
the booting of a more recent kernel than the one installed as part of the
installation, as it were. I've decided I'm going to work on that before I
leave for the day. Hopefully it won't take too long.
Over the weekend there's parkrun, helping out with the rowing regatta and some
Tesco shopping. Also a bit of television here and there.
[17:00] Moved a few more servers and got them nicely squared away at the new
server room. Also discovered two mounting kits for the 2540 storage arrays
which I'd been trying to get from Oracle for the past few weeks. Hurrah for
me looking through old boxes of mismatched cage nuts and similar. Now all we
need is some downtime so we can pull out the array, stick in the shelves and
put it back in again.
Again, otherwise there's not a lot happening. I have a server to move to the
new server room tomorrow from here (as opposed to the server room in town) and
then an identical server to put in it's space to build with RHEL6, two KVM
guests and then configure those with all manner of PHP modules and Ruby gems
and things. Once that's done, other than the udual daily maintenance and
things I'm not sure what'll be next on my schedule. I'm sure something'll
come along eventually.
I think I'll be out this evening for a bit helping put a landing stage in the
river for this weekend's rowing regatta thing. That might make me feel like
I'm entitled to get a pizza delivered for being so selfless. Not to mention
the fact that I sold (probably too cheaply) my Axis 2100 network camera last
night as I just wasn't using it for anything worthwhile. And paring down
what I own feels pretty good. We do accumulate a lot of stuff in our lives,
[16:50] I bit the bullet today and did a round of UPS client installations on
all of the unix servers at the new server room. This amounts to about 34
servers so far and leaves very few slots left if we stop at the "recommended"
fifty servers per event limit we've had mumbled to us repeatedly by the
manufacturers. Still, not to worry. I'm also not worrying about the fact
that despite being told by the web server page that you can put in hostnames
(and there's a DNS server setting too) if you try to use hostnames (which are
far easier to recognise than IP addresses) it doesn't work...
... Although I've just double-checked and it seem it does now. Which
is really odd, but very welcome. I think my last job today will be changing
all IP addresses for hostnames. I will then go to the gym.
[17:25] Not much happening today. After yesterday's run and evening gym I
decided that today should be a rest day. I didn't even do morning exercises,
which was a bit odd, but rather good considering I didn't get to bed until
midnight for various reasons.
I've pulled the other Dell server I should be setting up soon out of storage
ready to sort-of rack it up tomorrow and get an OS and some KVM guests on it.
This will probably be the one which ends up going over to the other server
room, but not until I've done something with it. Hopefully I'll be able to
take it over in a car rather than rattling it there on a trolley.
It's now just a touch under 71 days until I see Kris in person again and I'm
really not enjoying the feelings engendered by being apart for so long. I
know this is the way it needs to be, and in many ways it'll mean that we
appreciate being together for significant periods of time so much more in the
future (research trips notwithstanding)... but, I don't like it. I don't like
it at all. Neither does Kris. It just doesn't feel right and it's so very
much the exact opposite of what we hoped for for our first year as wife and
husband. I'm just so glad that I managed to find someone who'll still want to
be with me as much as I know Kris will do after the amount of time we've been
[16:20] Mostly this weekend I have been running. That's not strictly true, I
did a parkrun on Saturday morning and a 12.8K run on Sunday morning and
otherwise wasn't running at all. I did clean the house, watch some television
and go to a party where I saw Andrea (freshly back from her gold and silver
medals at the World Masters Athletics Championships in Sacremento, California)
too. But otherwise it was running. And my first session at the gym I haven't
been a member of for four years. I think my shoulders still ache, which just
proves how long it has been since I did anything with those particular
Today I have been finding out more about the UPS client software I have to
work with, discovering via the magic of Skype that something broke a
window in Kris' apartment and going for a nice 6K run at lunch time. I'm off
to the gym as soon as I leave work this evening in the hope that that'll work
the kinks out of my shoulders. Then it's weekend catchup time with Kris and
This week should be moderately interesting, I hope, in terms of work and
things. It'd be nice to get the UPS stuff squared away on the Solaris and
linux boxes anyway.
[15:05] Did stuff today. Not much. Gym tonight. Should be good. A weekend
of more gym, Tesco, house cleaning and a parkrun (not in the order) to come.
Now: switch installations and then home early!
[16:25] Dude, seriously? September? What the hell? I swear that by the time
I'm in my 50s years are going to pass by in a month or less. How can we be
nine months into 2011 already? Oh well, I guess time seeming to pass more
quickly does mean that I get to be with Kris again sooner.
Anyway, today I was intending to do a lot of UPS client installs but it just
didn't seem to happen. I'm not quite sure why. To be honest I don't really
know where today has gone. I know I went for a 10K run at lunch which seemed
to go pretty well other than a bit of calf muscle (not tendon) tightness, but
otherwise... no, the day just seems to have slipped by. I am waiting on
someone to come and see my Axis 2100 network camera with a view to buying it
(I've even got it all set up and displaying an image) but they haven't turned
up yet, which is annoying.
There's a rowing AGM this evening which I feel like I should sort of go to,
except that I'm slowly and quietly ducking out of rowing stuff these days as I
rejoin the gym for the next six months (heading over there this evening to pay
my membership fee). I will help with the regatta though (in a few weekends
from now) as it's a moderately fun day out, potentially in the sun, and I
might get to do some stuff in a motor launch.
I think that's it for today. I'm not sure what I'll have for dinner this
evening. Last night's thin crust pizza from Domino's was mostly OK (although
I think the base was overcooked) but I really can't do that two days in a row
given the vegetables I have in the fridge that need eating. Kris has her
second day of teaching today. Her first in two classes. I know she's going
to do really well, especially as one of the classes is pretty much standing
room only. I'm looking forward to hearing how they went either tonight or