[16:45] Extremely tired this morning. It's not as if we stayed up particularly
late last night either. Watching Supersize Me on the Channel 4 was an
extremely entertaining and thought-provoking experience though. I don't know
how many diets/food plans/changes of direction/exercise regimes have been
started as a result of watching it, but I can testify to the fact that I'm
going to make further efforts to get back into the kind of shape I was in
around three years ago based on having watched it.
Today I've been dealing with very little, to tell you the truth. The guy from
MGE who was dealing with our replacement network card has been on voicemail all
day (yeah right). There's this issue where the date on the card can be NTP
synchronised. This is working. It lists the date in ISO format too, which is
even nicer (YYYY/MM/DD, yeah I know it should be -, but never mind), however
when you go to the email notification settings page there's a dropdown for
which day of the month you want notifications to be sent out. This dropdown
runs from 1 to 31. Obviously days of the month, right? That's what I thought
too. So I decide that as it's the 29th today, I'll select 30. When I do this
a popup box appears saying (remember, it's 2005/04/29 according to the time
page) that "30/5/29 is not a valid date." Damned right it's not, but as I
can't specify more than a day of the month this is your fault, not
mine. Gah, helpful eh? All I want is a firmware that works, is that too much
We went for lunch at a great little pub a few minutes drive from here. Mexican
food. I was pleasantly surprised to find that a chicken salad (heavy on the
salad) filled me up nicely. Luckily there was a decent amount of chicken in
there too, but it was good quality and with no skin, etc. That, coupled with
the exercise I did this morning (every morning now) means I don't feel too bad
about the delicious Lindor chocolates I've been accepting all day from my
co-worker who has to give them to someone else as he's on a diet
I think that's it for today. Off to Mel's "I'm not going to be 40 tomorrow!"
party and drink for a little while. I have no idea what to do this weekend or
on Monday (UK bank holiday, no journal), but I'm sure it'll involve being
outside and happy.
[17:05] Not quite as good a day as yesterday unfortunately. While the morning
went fairly well with my taxi-transporting of two servers to another machine
room going to plan and after a bit of worry, getting them configured to use the
terminal server there, I had the horror of finding that some essential Oracle
installation scripts had vanished into the ether in the disk lost episode on
Monday. Luckily I'd had the foresight to squirrel a copy away on another test
server and was able to recover all the files bar one (the response file for the
Oracle installer). I'm going to build that again before I go home.
The fun thing of today was discovering the new network card for the MGE UPS.
It looks like they've sent us a Network Management Card rather than an XML-Web
Card so there's no SSL/HTTPS any more. However all the clients were still
trying to talk to it (once I'd configured it with the same IP as the old card)
via HTTPS so they managed to lock up the card in exactly the same way as the
old one. Cue me ssh'ing to all thirty-six machines to disable SSL
certification and restart the clients. Stupid thing. Now we're not using SSL
I imagine the old card would work too, but the menuing system on the new card
is much nicer so I think we'll stick with this. The firewall will keep out
people wanting to shut down our servers anyway.
My only woe now is about the email reminders thing. Don't get me started on
endianness of dates. I think all in all I just want to go home. I forgot to
have lunch today so I'm tired, hungry and missing Elaine.
[14:55] Today has been pretty good thus far. Setting aside the fact I had to
cycle with it in a rucksack on my back, I took the eighteen kilograms
of 1p, 2p, 5p, 10p and 20p coins to the only bank that opened at 09:00 (HSBC)
and was gratified to learn that they were willing to take it all. After Elaine
and I counted and recounted it last night we were only fifty pence off what we
thought we had. In the end Phil had managed to amass £248.60 in his loft
over who knows how may years. This should go very well towards providing a
good meal for the six people who were in the room at the time the idea was
Once I'd relieved myself of that load I headed to a vendor-initiated seminar on
Sun's new ZFS filesystem, Veritas NetBackup 6.0, Opteron servers and some other
NDA things. All things considered it wasn't quite as good as the Esteem one
I went to a few weeks ago, but there was the bonus that I won a brand new iPod
(20GB, like the one I have, but a more recent generation) in a prize draw.
This means that Elaine can have my old one if she wants it and she doesn't have
to buy one herself, which is always good.
I'm in work now and while the replacement XML-Web card for the UPS hasn't
arrived everything else seems to be working fine at the moment. If they
continue to for another two hours I think I'll be able to go home and climbing
having had a very good day. Must remember to leave in time to get home for
that and to email Phil and Co. with the loose change total.
[17:00] Urgh! I need more sleep. After dashing home in time for Cormac to
pick us both up, we got everything we thought we'd need for the evening
together and were only a few seconds behind when he turned up. After going to
Phil's house to collect him and Helen we then left them behind while Cormac
dropped the two of us off at the place we were to fly kites and eat the picnic.
Of course, the weather began to close in ominously as we lay on the grass at
the side of the car park and chatted about the day. Eventually Cormac returned
with Phil and Helen and we set off up the hill to put out blankets, try flying
the kites in the dead air and eat some food/drink wine.
The weather deteriorated as more people arrived and we ended up wearing
waterproofs while playing with an Areobie. As the light faded to a point where
it was impossible to see the (spinning, fast-moving) flying disk until it was
inches from your face we made a move towards the bowling alley where we were
told that there wasn't any chance of a lane (let alone two) until well past ten
o'clock. With it only being nine, we left rapidly for Phil's house where Elaine
unpacked her cheesecake, covered it in raspberry coulis and served it to the
extremely impressed assembled masses.
As the night wore on we drank a lot more, found out some extremely weird things
about our host (like the fact he has bags of pocket change in his loft because
it's "too hard to spend", and two antlers which he'd really like to get rid of
because they're both lefthand side ones) and played human Buckaroo with Helen
once she'd fallen asleep. This is just like the game with the little plastic
mule, only you use a sleeping human and cover them in anything to hand. We
managed to get skiing goggles on her and lit candles on her shoulders without
her waking up, not to mention plates, glasses and other lounge ornaments. In
the end it just got too late for everyone, especially the person who'd have to
drive home after driving us home so we left in good humour early in the
morning. I ended up taking about 30kg of loose change away with us to count,
take to the bank and then (on the orders of Phil) take everyone out for a
meal with the proceeds.
Getting up a few hours later was a real trial, but luckily all my work is
still in other peoples' courts at the moment. I went to the other two machine
rooms over lunch to make sure there was enough power for the machine I'm
moving soon, prodded the people in Web Services to get their bit done and
wandered round all the linux boxen with SSH and up2date getting kernel updates
in place for their next scheduled reboots.
The backups have been vaguely shafted because of the problems we had over the
weekend and on Monday. Developers and DBAs with the root password on the
staging server are causing minor problems which means my colleague is tearing
her hair out. Hopefully words in the right ears have stopped that for the
Personally I have a monster headache which has come out of nowhere so I'm
going to go home.
[13:05] Already today we've had an LTO2 drive (Sun L25) jam, a disk in a
concatenated stripset go down taking lots with it and further disk failures in
the SAN. Sun are in and out of here like regular employees at the moment. I've
been in initial consultation with MGE about our XML-Web card and have been
doing some kernel upgrades on our RHEL3 boxes.
Rather than wait for the phone to ring this afternoon with progress on the MGE
issue I'm going to a lecture/seminar/calming-down session on the pay and grading
exercise that is going to be taking place here in the next few months. I have a
few questions I want to ask.
Cormac's birthday today, hopefully if the weather stays fine we'll be off
kiting this evening. Elaine made some wonderful cake and cheesecake over the
weekend so I'm looking forward to eating them too.
On Saturday we were at a picnic with Dunk and Alyzande and assorted partners,
children and food. A great time was had by all, frisbees were thrown, balls
lobbed and caught and birds watched. All in all, a great time. Must head off
and find this presentation thingy.
[15:25] I've run out of useful things to do today. I have people prodding our
UPS vendor to deal with the network connectivity issues we're having, I have
multiple developers working on testing suites for the new machine's
applications, other machine builds in other people's courts, Scrubs (Season 1)
ordered on DVD, run out of things to eat and am generally at a loose end. I
have a picnic planned this weekend in the hopes that it won't rain. We'll have
to see how that turns out. Otherwise we may go climbing in London again on
Sunday (the Castle this time I think).
[17:40] OK. So creating a set of statically NATted addresses, pointing them at
the internal IPs and then creating a whole new set of firewall rules
has meant that most - if not all - of my projects on there can now no-longer
talk to the Oracle databases they need to. I have a large number of firewall
rule change requests stacked up to give to the extremely helpful network admin
people tomorrow morning.
On the plus side I did go into town today to pick up my brand spanking new SSL
certificate and ended up having an extended pub lunch with Bob, Mel, Charlotte,
Jo and Paul. That was most enjoyable and a wonderful chance to catch up with
what everyone else has been doing. I may see if I can make it there every
Thursday to see other people and get away from the keyboard during the day.
Speaking of which, I should head off and do all the things I need to do this
evening that meant I cancelled the regular Thursday get-together I host.
[17:20] Unless I'm very much mistaken I'm just a hair's breadth, or an SSH
tunnel test, away from being able to say that all the applications (bar one
that's being re-written) are up and running on the new server. I spent the
entire morning working my way through some extremely nasty crontab stuff that
was visible from one part of the file system but not from another inside a
chroot. This is fairly obvious until you factor in the fact that the reverse
was true for other crontabs. Suffice it to say that with some help from the
original sysadmin I've been able to sort it out.
Lunch with Shaun and Jenny was a chance to talk about the Lib Dem's local
income tax with regard to firemen and nurses (see online) and the cost of
sending your children to University these days. By the time I have children
I hope things are a little easier for parents.
During the afternoon I was tied up with talks with my line manager to get IPs
allocted to the server in the static NATting range, emails to the developers to
come up with testing strategies for the new services and a sudden announcement
from one of the developers that about now was a good time to completely rewrite
one of the applications to use a centralised authentication mechanism. This is
great as it means masses of unmaintainable perl get thrown out of the window...
I just wish he'd make a start on it some time soon. Finally today I've done my
first real SSL certification signing request. This needs to be
fulfilled before the middle of next week so I'm hoping it'll be processed soon.
In the meantime I've created a thirty day self-signed certificate so the httpd
will actually run in HTTPS mode. We're planning on setting a deadline for
getting the new server online soon. Frankly I can't wait because the OS level
and general disrepair of the current live server gives me the screaming willies
Climbing again this evening. Hopefully I'll remember my rope this time and
will be able to do some leading again.
[17:20] Wahey! A productiveish day. For most of the morning I had a go at
categorising all the stuff in the live instance of the web application I'm still
fighting with. This went fairly well until I started hitting bits that made
no sense whatsoever. Happily at that moment the Sun engineer turned
up and we spent about half an hour replacing disks in an E250R and using lots
of meta* commands.
Once he'd gone I handed some RHEL4 CDs to someone and settled down to see what
I could do with the remainder of the files in the live instance. Luckily, one
of the two people I was hoping to talk to walked by the office and I was able
to grab them, sit them down in front of the computer and get rid of a lot of
crap from the setup. This has made my job a whole lot easier.
What helped even more was one of the actual developers of the application
happening by to do something else. Quick as a flash he was snared and forced
to work through the entire directory structure to point out what was necessary
and what was cruft. This helped enormously. I honestly think I have a chance
at cracking this thing tomorrow. As for now, I'm going home before my run of
good luck fails.
[17:15] We made it to Westway on Sunday. The weather was excellent, which was
helpful as it took us a little time to find the place. In fact we got off
the tube a stop early initially and wandered around bits of London we'd never
been to before for about half an hour until a call to someone with a web
connection got us sorted out. The wall was interesting. A huge selection of
routes (mostly to lead, rather than top-rope). The grading system confused us
and a few other new climbers for a while until we realised it was French
grading and consequently the 6A climbs I'd been leading weren't actually that
Owing to one thing and another we didn't last more than about two hours on the
wall before we needed to go and get some food inside of us. A quick couple of
tube rides took us to Tokyo Diner in Leicester Square and some much needed
calories. Next time you're there try the Teriyaki Chicken, it's gorgeous.
Saturday was a half-day on account of not actually getting up until gone 12:00.
We didn't do much other than relax and get the house in order.
Most of today has been spent exploring the largest project on that server I'm
trying to consign to hell. Well, that and the UPS interface card which refuses
to respond even once you've actually taken it out. I'm worried that a power
failure here is going to result in some really unpredictable happenings.
I've been thinking about how I'm going to vote in the upcoming election and
find that I have very little feeling about the whole thing. No matter how
energised I try to get about it, all I can think of is the sense that the
Liberal Democrats seem to make on most things and how for me it's just an open
and shut case. Replacing the council tax with a local income tax, their
policies on immigration and their scrapping of University tuition fees just
seem to make a lot of sense to me. I can't honestly find anything to discuss
with people unless they're rabidly opposed to those ideas. Does this make me
apathetic? I hope not, but I don't care (!) if they do. I'll be voting, I
just don't really feel any need to get into politics any more than that. If
you think there's something else I should be doing let me know. I'll happily
chat, discuss or otherwise argue with you if you really want, but besides that
I really can't think of anything to do other than vote. I'm not politically
active, and don't really want to be. I don't think that leaves much else.
Something I am thinking of doing again soon is writing stories. It's been a
while since I did, and I think it's time I had another go.
[17:00] I should have gone home half an hour ago but I was determined to get
something working before I did. After a meeting I thought was going to be on
Monday I had another meeting with my line manager to discuss the moving and
test period of the services on the old server to the new server. This turned
into an extended discussion of IPs, DNS, static NATting, the number of network
ports and power sockets at the target site and when things could be done by.
After that I was pleased to be able to sit down and get some scripts working.
At least I thought they were working. Turns out I knew less about RedHat's
rc chain than I thought and while scripts beginning S.. work perfectly well in
places like /etc/rc.d/rc3.d/, you can't expect K.. scripts to work in the same
way unless you've got a matching file in /var/locl/subsys/ for it to check
against before running your script with a 'stop' argument. This took me a fair
while to work out. Still, now that I have chkconfig doing my bidding with
some well chosen additions to my script I think we're looking good. Or at
In a further attempt to be healthy and have some fun we will be going
to London this weekend to climb, probably at
Westway rather than
the Castle, to see what it's like. Other than that I plan to sleep late, kick
back and try and relax.
[15:50] I've managed to move another installation from one machine to another.
While I will have no problem creating a CSR for the new site I'm working on,
there's one site I'm waiting for someone else to deal with (Monday) and one
that is just so damned complicated that I don't really have any idea how to
get it sorted and tidied at the same time.
In fact I'm exremely tired right now. I could do with going home and getting a
bit of sleep. I think we both could. Thursday tonight, so just some DVDs to
watch and some friends around.
[16:30] Still ill today, and keeping Elaine awake during the night isn't
helping her either. I spent the morning at a meeting trying to get people to
agree simply to split one project off from another and not digress into four or
five other topics of interest. About an hour and a half later I was pretty
sure that I had my answer and asked people to go ahead and do what they needed
to do, as I was doing what I needed to.
We 'lost' the root password for one of our servers so a colleague and I worked
through breaking the mirror of the root filesystem disk, resetting the password
and recreating the metadevice information. Certainly very interesting and a
worthwhile procedure to remember for the future the next time I have a Solaris
misshap like that.
Other than that I've been feeling terrible all day because I'm not really up to
being in work. I really only came in for the meeting but thought I could get
some other stuff done. I think I did. Don't think we're climbing this evening
as the usual climbing wall is closed. We may go to London, but I think
everyone could do with saving money for once.
[13/04/2005 - 09:25] Ill today. Well not actually 'ill', more an extremely
sore throat, coughing and generally feeling a bit tired. I guess that's ill in
one sense of the word. Anyway, cue lots of relaxing, eating too much food and
trying not to speak. With no-one else being around this was fairly easy. I
did go climbing at someone's home wall last night though. That was interesting.
Some very hard-core climbers there, far better than me. I may go back next
week, I think it'll be good for me. I'm also intending to join the local
climbing club in a month or so. I've held off for a while for various reasons
but now I think it's time I do what I enjoy.
[14:10] It was my birthday on Sunday. Saturday was pretty good. We got up late
and didn't do much (this is good). Elaine went out to get some shopping while
I spent some serious time relaxing from the week. Shortly after she came home
people started turning up and after a while we wandered down to the curry house
via Keith's house to pick him up. The food was good, the company good and
afterwards the conversation on the way back interesting. We drank wine and
watched films until the small hours when everyone else went home.
Sunday was good until Elaine's father (still on Australian time) rang us
indecently early and wished me happy birthday. Energised, Elaine dashed off
and returned with a bucketload of presents and cards which I opened in bed. I
don't think I've had a better birthday morning in the last ten years. I was so
pleased with how things were going I didn't even mind cooking my own birthday
breakfast. We had previously decided to go to London to climb after breakfast
but the opportunity to be outside in the sun rather than on a train and then
indoors meant that we grabbed out bikes and headed out to cycle downriver a
few miles. As the sun shone we looked at the wildlife, laughed and joked and
watched dogs jumping in the water. We found an excellent pub (with
non-excellent drinks prices) to stop at for a while before going far enough to
watch topper/laser racing. Heading back Elaine cut the grass while I started
tidying the house. A couple of houseguests later and we were feeling quite
tired so I ordered some Chinese. I have to admit that we watched Air Force One
before falling asleep on the sofa.
This morning I seem to have a really sore throat. I'm hoping it's not whatever
some other people had last week. I'm going to try and climb this evening at
someone's personal climbing wall.
[16:20] Whee. One and half months after I left my old work place, about one
and a half months before they got around to getting someone permanent to
replace me, it looks like they were hacked. What with one thing and another
they're going to have to clear the machines up substantially, reset everyone's
passwords and generally cause hassle for a number of people through no fault of
[16:35] And on cue the Dynarod guys appear to have made the smell of urine
permeate the entire corridor, and my office. I'm going home before I begin to
smell of it myself.
[14:45] Right. I don't think there's anything I can reasonably do today. I've
been twiddling my thumbs for most of the day, emailing developers to ask them
to look at the live areas of the projects they're responsible for. I've got a
few things done - two projects moved across for a start - but the other three
are going to need some care an attention to ensure I don't miss anything
It so happens that it's my birthday this weekend. I intend to have a good time
with those people who're able to turn up on the Saturday (curry at some point)
and hopefully go climbing in London at some point. I'm trying to ignore the
fact that this will mean I'm entering my thirtieth year on the planet. Still,
I'm not sure what else I can usefully do today. I may head off at 15:00.
[17:35] I've made some stonking progress with the new live server, and also
fixed some silently failing holes on the development box that we (the developer
in question and I) didn't think to check previously. As a matter of fact I
I've moved two of the five project areas across already. The others are going
to be a real bitch, but I think it's going to happen. Once that's done all I
have to do is come up with some kind of rsync thing to deal with the live data
and we'll be sorted. Yeah, right. Nice to dream.
I left for work this morning in a really strange mood. Wasn't sure about all
kinds of things. Happily, as the day went on, I was pleased to find that my
mood actually perked up on its own. How about that? Not even
listening to the non-stop disturbations that are Chris Morris' Bluejam season 1
recordings managed to make a dent in what has turned out to be a fairly
positively-mooded day. I really have to say that this job is so much better
than my last one for making me think, work and generally improve myself.
[16:50] I ran in this morning. The headwind aside I think I did less well
than normal. I think it was something to do with lack of sleep again. Still,
I'll be doing it again so I don't doubt I'll have some kind of improvement over
the coming weeks.
I sent out an email about my birthday this year and next year. Hopefully I and
a few other people will be going to the south coast of Spain at Easter next
year. I had planned on Canada but I think that was a bit rich for everyone's
blood at the moment. I know I'll go one day, just not in the next two years.
I've been doing reinstalls of what will be the new live server and have got the
basic setup complete. Now comes the hard bit of getting the extra stuff on and
configured in such a way that it looks like the live service, only tidied up
behind the scenes. I think for the moment I'm just going to write some scripts
and see what happens. I won't do another install until tomorrow afternoon at
this rate. But that should be fine. Climbing tonight, so I better head off.
[17:15] This morning was spent joyously updating the BIOS, RAID, backplane and
PSU-handling firmware on the server I've just taken out of commission. Of
course the RAID scrub is now taking up most of the rest of the day. I've also
taken the opportunity to open the case and have a look at what I'm working with
inside. I've removed one of the SCSI cards (the other two being on the
motherboard) and turned the other one off. The box now boots in something less
than half the time it used to.
I was actually able to get into the chassis by dint of going via a server room
of ours on the other side of town on the way into work this morning. Luckily
the sister machine did indeed have keys dangling from it and I have
appropriated one for this one.
The remainder of my time has been spent fighting firewall rules and messy
cabling in an effort to get the box connected to the serial terminal console.
This one doesn't have BIOS serial redirection so you can't see what it's doing
remotely until the grub bootloader comes up. But that should be OK.
I didn't go climbing last night on account of things needing dealing with at
home. Not that we got everything sorted, but some of it. Tonight we'll have
to make a decision on one thing, or it'll happen without us. I've also thought
about where to go for my 30th birthday next year and decided that Canada was
just a bit too far to expect people to come with me so I've begun to plan on
going to the south coast of Spain instead.
[17:30] I've developed a mild hatred for RedHat in the last few hours. I got
in this morning and twiddled my thumbs waiting for any one of a number of
developers to get back to me with new requests for the builds I've been doing.
When one of them did it turned out that they needed a whole load of perl modules
installed. Weeding out the ones they actually didn't was easy enough, but
getting one installed was a complete pain in the arse.
Let me tell you a little story about Apache::libapreq. This little module
wants a version of mod_perl less than version 1.99 but greater than 1. This
was annoying as my box was running 1.99 according to the output of rpm -q. No
worries, a bit of ferreting around found me a version of libapreq2 which
(while development-level) would probably work. After fettling for a bit I got
to compile it by hand. Lo and behold it required a version of mod_perl greater
than 1.9915. Naturally I turn out to be running 1.9909. Thanks RedHat. In
desperation I decided to upgrade the version of mod_perl. On attempting this
I'm informed that mod_perl2 requires Apache 2.0.47. Of course, I'm running
2.0.46. RedHat really need to catch up a bit with their packages. I don't
believe that there are (m)any functional changes between 2.0.46 and 2.0.53, let
alone 2.0.47. And as for mod_perl, 1.9909 is extremely alpha as far as I can
Happily in the end I was able to convince the developer in question that CGI.pm
was a simple solution to this "for want of a nail" problem and went away happy.
In fact doubly happy as I took a detour by the last developer on the old
development box and he confirmed that pretty much everything on the new box was
working as expected and he'd be emailing his 'official' sign-off on the old box
shortly. As a result I've emailed all the developers involved with a deadline
of 17:00 today to get anything they forgot about off the old box before I wipe
The weekend was pretty good. After an extremely pleasant morning, Elaine and I
drove with Cormac and Steph to High Wycombe for a party. After walking from
the house we were sleeping at to the party house we were fairly amazed at what
we saw. Long story short; it's a house that belongs to a gambler, it's full of
cool toys. We drank, played, danced and had a great time before wandering home
at around three in the morning.
Sunday, once awake, we went climbing in Amersham and I for one had an excellent
time. I think we were all a bit tired by the end of the session so drive home.
I'm tempted to go climbing this evening. But probably won't as Elaine and I
need to discuss weekends away.
[11:15] I don't know about you, but I don't really have time for the April Fool
web pages today. I've got far too much on at the moment. On the plus side if
I can get everything I want to do done today I'll have an absolutely great
weekend of relaxation to look forward to with something enjoyable on Monday. I
have managed to get an Oracle 10g client (for linux) installation wrapper
script completed which I think will work perfectly (fingers crossed). I've just
started what I hope will be the last rebuild of the first of the new web servers
for one set of developers (including a nifty fiddle with a perl module to get
around our firewalling issues here and a silent installation of the
UPS client software as well), been told I can start moving the last
developer from one of the ageing machines to the new development server
(freeing up the ageing server for a rebuild as the new live box) and hopefully
will be having a nice lof-fat lunch as well. It can't last. For the moment
though I'm going to ignore the bad night's sleep I had last night and just go
with the flow.
[17:55] Well damn. I've only managed to get the final and hairiest project
moved over from the old development server to the new one. Not only that but
I've managed to decipher some really odd logging variables to make the logs go
to somewhere sensible too! Fear me! Or don't, I don't really mind. Either
way, that coupled with the pretty good Oracle installation wrapper script I did
this morning and the fact that I get to go home and be with Elaine in a few
minutes knowing there wasn't anything else I could have done today means I'm
feeling pretty damned good. This weekend we should be going down south to a
party, we'll have to see how that goes.