[17:20] I really should have gone home by now. However Elaine's going home by
bus so I have a little while before she turns up, I think. One of the
developers I'm building machines for gave me one back today to incorporate all
the changes he's made into the build. This has taken slightly longer than
expected due some oddnesses concerning Java, the UPS module I've added to the
build and the firewall here which makes adding certain Perl modules which
require connectivity a little harder to deal with. Still, I think I've got it
all sorted now. I was going to re-kickstart it today but RedHat Networks was
down and what with one thing and another I just haven't had time. Still,
tomorrow should be good as I've got this machine to do in the morning and then
hopefully another developer coming to see me to get the last of his stuff of
that other server I'm going to be nuking.
Tonight is Thursday, with all that entails, so I should head off and be home
before Elaine is.
[13:00] Although we went to bed fairly early last night we were both shattered
this morning. I decided not to run to work as I needed to get in in good time
and be able to deal with any issues which had arisen since I left on Thursday
evening. Luckily nothing seems to have gone wrong and I think the UPS thing has
sorted itself out, at least for the moment.
Most of the morning has been spent sorting through email, doing NTP server
updates and scheduling up2date on all the RHEL boxes I administer. I may get
around to doing some actual planned work this afternoon.
[16:10] Progress. I've put in motion the procedures which should cause the
Windows administrators to call me when they're ready to install the new Java
clients for talking to the MGE UPS. Hopefully they'll do this in the next day
or so. I won't be able to get the client on to the numerous Sun boxes we have
here until I've talked with the Solaris administrators here who know more about
the specific Java versions we're required to run here.
[30/03/2005 - 12:30] After one of the coldest nights in a while we stripped
most of the insides of the tent out and packed them into the car. After heading to
breakfast at the Longlands Cafe again (we go there every morning when in the
area) and making some further purchases we drove over Winnt's Pass to the
Chestnut Centre and spent an enjoyable few hours looking at owls, otters and
other such creatures. Not quite the weekend we were after but enjoyable
nevertheless. We're definitely going to go back (and elsewhere) with that tent
as it's perfect for four people. Being able to stand up while camping and
under cover is an experience I've not had since I was in niger tents in the
Unfortunately we had one casualty of the trip. Elaine woke on the Sunday
morning and managed to wrench her neck badly because of the cold and sleeping
in an odd position. After a trip to A&E in Sheffield that day we
ascertained that there wasn't anything serious wrong but it was going to be
painful for a while. Cue me forgetting, pointing things out and Elaine turning
to see them and then shouting in pain.
Back at the camp site after the Chestnut Centre we took down the rest of the
tent, wiped all the mud off the bottom and drove home. I have to say that we
were all extremely disappointed not to be able to climb at all. The walks
we took at places like Mam Tor and Hope were extremely enjoyable and allowed us
to stretch our legs, but weren't what we'd gone there for. I have a feeling
that the next couple of months are going to see a lot better weather so we'll
have a fair number of opportunities to introduce Elaine to the perils of proper
[30/03/2005 - 12:00] Cormac and Steph got caught on sandbanks returning from
releasing seals on Saturday. Consequently we didn't actually leave until well
gone 17:00. As a result we arrived at Laneside campsite in the pitch black.
Setting up a large tent for the first time in the dark when it's spitting and
the ground is extremely muddy isn't a huge amount of fun, but we managed it and
did it well. Unfortunately, the entire weekend consisted of rain most of the
time so the rocks were unclimbable except by the terminally brave. We opted
for some long walks and retail therapy in 'Outside' in Hathersage.
[30/03/2005 - 11:40] For most of the morning I stayed in bed and tried to
catch up on some sleep. After a while though I decided to get out of the house
so cycled into town to get Elaine a sleeping mat for the weekend's camping.
Other than that we did very little other than watch the weather reports.
[24/03/2005 - 14:50] The day was kind of in two halves. The morning was spent
trying to work out how to get the last developer and his mass of project files
over from one server to the other. This currently doesn't seem to have a happy
ending, but I'm hopeful. For the afternoon I was asked to return to my task of
getting the MGE UPS to talk to all the servers in the this server room. We
have thirty-nine servers in there (Solaris, Linux and Windows) and we need them
all to listen to the XML Web Card thingy in the UPS. This requires a) firewall
ports to be opened, b) software to be found on MGE's site, c) custom installers
to be made which use SSL rather than plain HTTP to authenticate and d) the UPS
not to throw a major wobbly when it realises that it's suddenly being asked to
furnish SSL rather than unencrypted traffic. This is what happened as I was
waiting for Elaine to turn up and help me home with the new tent which was
delivered to work during the day. Luckily, as I was leaving the web interface
for the thing seemed to starting to respond again.
When I got home, before Cormac came to pick us up for climbing, I emailed my
work colleagues, let them know the score and how to disable the new clients I'd
installed if there were any problems. Climbing was pretty good although we all
looked a little wiped out.
I have tomorrow until Tuesday evening off, and I intend to relax, tidy the
house and then go climbing in the Peaks between now and then. Photos to come
plus some kind of climbing report when I get back. Promise.
[16:35] Turns out that you can't give blood if you have a mouth ulcer. I only
found this out when I had waited for the hour break the nurses were taking, the
queue to move forward and for me to get my interview with a nurse. If I hadn't
mentioned it I probably would have a small hole in my arm right now, but it
seems that an ulcer could be virus related and viruses end up in the blood as
it were. Still, I got to see a huge mechanical arm thing tearing down a car
park next to the church I was attempting to donate at.
Oh, something I forgot to mention yesterday is that Elaine won a bottle of
whisky at the show on Saturday. I'm sure she/we will get around to drinking it
some time soon. In other news I made an absolutely wonderful risotto last
night. Pancetta and Borlotti beans with some shallots, garlic and a splash of
vermouth in with the rice to give it some more flavour. Very nice. I think
Elaine was only slightly more surprised than me at how it turned out. I'm
looking forward to doing the next experiment next week. Of course it required
a cycle to Tesco all the way from home, but that was part of the fun.
[12:15] We bought a tent. OK, we bought a tent between four people. This made
it very cheap and the chances are it'll only ever be used by the four of us at
the same time. The show itself was huge. Lots of interesting things. Elaine
bought some new rock boots. Come 15:30 of course Cormac was glued to the large
plasma screen in the Wales Tourism Board area (which for some unlikely reason
was right next to the Ireland stand) for the rugby. This seemed to draw
probably the largest crowd next to the British Bouldering Championships. On the
way back we got some shopping in and watched The Incredibles before Cormac
sloped off without most of his stuff.
Sunday was spent mainly getting housework done and playing FarCry (now that my
new DVD drive has arrived). We did go to see Constantine at around 16:00
though, which rounded off the afternoon nicely. When we got home Elaine was
interested in watching Calendar Girls, and against my better judgement I decided
to watch it too. It's a wonderful film.
As soon as I got in this morning I had to order a taxi to take me and a Sun
engineer to another site to install a new fan tray in an E250. One of the fans
had completely siezed it turns out. Bringing the databases and the operating
system down didn't take long and I think all told it was offline for about
fifteen minutes. Once back at the ranch I had a few problems getting the web
application working with it again but a few kill -9 commands saw it right again.
I may go to give blood this afternoon, depends whether I go to try a new
climbing wall this evening or not. I really could do with getting a bit more in
[15:15] I still haven't decided whether to go climbing tonight or go and give
blood instead. In the meantime I've been given a new job; working out which
machines are connected to the UPS via the network and a Java/XML client type
deal and which aren't. First off is giving the UPS a DNS name that isn't in
the zone we want to delete. I think I'll give blood and climb another time.
[16:55] Today ahs been, on the whole, a good day. For most of the morning I
was able to do some little bits of documentation which'll allow me to build the
new live server more sensibly. I also was also able to tick off my first
fully moved over project from the list as well as ticking another one off as
never having needed to be on there.
I went out for lunch with Shaun and a few others to say goodbye to one of the
developers before coming back to help some of my other migrating developers
shift across certain files before shutting down apache and samba daemons. The
old development box is getting quite sparse in what it's serving out now, but
it's still getting hammered. I have to wonder how it's survived as long as it
has with what it's been forced to endure. Now that a Microsoft Access job has
been moved completely from the development server to the live one (where it'll
remain) we're very close to having only one project left on the box and the
developer for that one is extremely willing to help with his move. I think I
may have it all done by the end of next week. At that point I get to trash the
crap install and do some major firmware upgrades as well as take out about four
SCSI cards which aren't doing anything other than making the box slower to
This weekend a few of us are off to Birmingham to The Outdoors Show. If I
don't come back with anything I will be Most Surprised.
[13:30] Aside from a few logfile location and database related misconfigurations
I've just managed to get my first signoff from a developer on moving stuff from
one server to another. I haven't disabled his logins on the old box yet, but I
may do soon. I also had to rename the new machine this morning when I realised
that I'd spelt the name of the person I'd named it after incorrectly. Bah.
Other than a bit more work on the new server (getting some more DNS A and CNAME
records added and removed here and there) that's been about it this morning. I
head off to the old workplace this afternoon to do what is hopefully the last
bit of handover to the new guy there. It's great, I can already think of it as
somewhere I know a lot about, but don't really have any connection with. I'm
not even going to get stressed if he's started dumping files all over the place
and making things untidy. It's Not My Problem any more.
With it being St Patrick's Day today and me knowing an actual >50% Irish
person, I'm going to the pub this afternoon/evening and pretend I don't have
[13:45] Happy birthday to my brother today. I'm sure he won't want to be
reminded of how old he is today, so I won't say anything. I had an excellent
meeting with another developer today about his part of the development server
I'm porting across. He also made me feel a lot happier - on being told his
project had two separate parts to it - when he told me he could probably talk
to someone else for me and get it moved off completely. He's also decided to
modify his current development environment on the current machine to mirror the
new one that I'm creating, all without my prompting, which is great.
I'm off for a technical seminar this afternoon and then home.
[17:55] For most of the day I've been in an Esteem/Sun presentation day thingy
at a nearby hotel. As I'm under the usual NDA all I can really say is that the
speakers were pretty interesting and the food was OK. I left before the last
talk so I could come back here and change a machine's DNS details. I took the
opportunity to reboot the machine after putting some symlinks to shutdown and
startup scripts in the rc chain. I was pleased to note that they worked
flawlessly and the services I wanted brought back up came up and the ones I
didn't (didn't exist yet) weren't tried.
In celebration I bought myself a new DVD-ROM from eBuyer which should
apparently arrive some time before Wednesday of next week. Oh, yes. While at
the seminar thing this morning I met up with Julian who waxed lyrical about his
almost completed new kitchen and I decided it was time to start planning for a
new kitchen of my own. Now that I have some money saved up I think I can start
making serious plans. Of course, I won't be paying quite as much as
Elaine's been out of town all today doing some much-needed clothes shopping.
I'd imagine she's on her way home by now so I should probably pop off too.
Tomorrow I have a meeting with another developer regarding another project on
the development box I'm building. Hopefully what I need to move over won't be
[18:00] What a Friday. Not only did I burn myself out totally doing all kinds
of mindbending things with the box I'm trying to configure, I then left at
lunch time and spent two and a half hours in an MRI machine doing extremely
intensive brain things. By the time I got out I was ready to drop off. That,
coupled with the hard day/week along with the fact that some dumb student
caused me to crash into them on the way to the MRI appointment and caused
bleeding from knee and finger and by the time I got home I just wanted to
collapse on the sofa and forget about the week. Luckily Elaine was on hand to
soothe my fevered brow for a while. Of course I couldn't forget about the
stuff I had to do until some time on Sunday afternoon.
The weekend was mainly about Elaine cooking cake and stuff. That and getting
my hair cut, finally. Cormac and Steph invited us out to see a Celtic folk
rock band type deal on Saturday night which I had to leave early from to run
home and check we hadn't been broken in to. Long story, don't worry about it.
Of course the house was fine.
Today has been pretty much as intensive as Friday, only for a complete working
day. I'm shattered. On the plus side I think I'm very close to getting the
first complete project moved from the old development server to the new one. I
had the two developers who've worked on the project sitting at my desk today
showing me where to change certain parts of the code to make it work with my
new directory structure. Aside from some Samba fiddling and a firewall issue
with getting CVS to work I think that's about done. I'm not in tomorrow (on a
Sun demo thingy) so I won't know if we're ready to go until Wednesday.
Other than that I can't think of much else that's happening at the moment. My
parents are both finally moving off Windows 98 to XP and getting machines
capable of running at a decent speed, my brother's out of hospital and seems
fine, I finished Homeworld2 and I don't know why usernames with hyphens in them
don't show up as anything other than a UID when you do a ps under linux.
[13:55] Short day, short entry. Have been fiddling with Samba for most of the
day as well as advising a colleague on Apache installations. I managed to get
the web application I'm installing to talk to the database, which was a major
fist-punching-the-air experience. I'm pretty sure I know what to do for all
the other ones now. I made sure I got it all written down too.
Off for an MRI experiement now. Then the weekend!
[14:15] Today has gone really quite well thus far. I've managed to get one
project theoretically up and running (setting aside things like incorrect SSL
certificates and no Oracle client installation). As a result I should really
have lunch some time soon.
I slept like the dead last night, only waking once because I was far too hot.
Elaine and I are in pretty good moods at the moment, I think it's the way things
are going at work and at home, coupled with more relaxation when we're around
I've had or arranged meetings with three of the four project people who'll be
using my replacement development box, managed to document most, if not all, of
what I've done to make the box work as far as it does and am really,
really hungry. I should have got my hair cut yesterday but missed my
appointment due to work. The hairdresser was very surprised to get a call from
me apologising. Now, food.
[10/03/2005 - 14:10] Lots and lots and lots and lots and lots of fun and games
with Apache2, Resin-2.1.16 (not 2.1.6 as I said yesterday) and configuration
files. I think I spent most of the day umbilically tied to my headphones
listening to MP3s in an effort not to go made and kill someone or something.
I ran in in the morning without breakfast, worked through lunch and when I'd
finally achieved a minor breakthrough, left straight for climbing. This meant
that although I climbed like a demon, by the time we got home I was running
very low on energy both in body and brain.
[18:20] What a day. It's been one that I would prefer to forget, only I can't
because everything I've learnt today I have to remember or I'll put myself back
another day trying to do the same things again. I feel I haven't actually
learnt anything today, more bounced from problem to problem, fixing them by
luck rather than skill, or judgement. It's been very frustrating.
For most of the day I've been getting a new development server ready to work in
the same fashion as the old box it's being groomed to replace. Getting the
directory structure in place was overly complex until I settled on doing one
project at a time. Hell, working out which projects were still live was an
issue in itself. I now know that there are probably four live projects on the
server, rather than the seventeen that seem to exist. Once I'd settled one
which project to try and port across first I had a go at getting some of the
back end infrastructure into place. The Java SDK went on easily enough, and
with a bit of fiddling so did Resin 2.1.6 (we may change to version 3.x in the
future if things work). I decided to bypass the sticky-looking samba setup in
favour of the apache setup (mod_ssl). This turned out to be an extremely
irritating exercise. There are a lot of fiddling little changes between 1.3.x
and 2.0.x. Suffice it to say that it's taken me in the region of two hours to
get a working SSL setup going on the new box using the cascading configuration
files that I have to use to deal with the multiple projects and their
requirements for individual web servers.
In the middle of all this I've been getting the machine a new IP and DNS
details as well as a slew of extra IPs that it's going to be listening on too.
I had one of the web developers come down to take a look at the SSL part of my
configuration file and he couldn't see the problem. The problem was that there
was IfDefine HAVE_SSL stuff in there, which really didn't help. I don't know
if it needs to be there or not. Probably not given that when I commented it
out everything began to work as it was supposed to. I'm going for the cleanest
configuration files I can to make it as maintainable as possible. The more
cruft I can clear out, the better.
Anyway, now that I have a server which servers HTTPS in the special way I need
it to I can start thinking about moving over the content (I have the developer
looking at what can go, what can be moved to the new CVS server I've already
set up, and what needs to be copied across), working on getting samba working
for the people who need it, and then moving on to the next project. The other
three projects are going to be much harder as far as I can tell. There's the
Resin stuff, the Oracle intergration, the Perl modules and on top of all of
that the java setup.
[13:40] Yes! Fixed a stupid problem with a Dell PowerEdge 1750 using the
megaraid rather than megaraid2 driver for its hardware RAID setup. I should
really thank two guy called Andrew and Eric for two halves of the solution
which involved PCI ID strings and creating a new netstg2.img cramfs image.
This morning the networks guy and I moved the other mail server to another site
to ensure that we have proper redundancy in the event of power failure. That
was easy and didn't take long. I think the machine was down for an hour and
nine minutes all told. Once that was done we took a trip to another site to
clear away some old network cables in a room with no lights. That was
Back at the ranch I took some time to work out why the server insisted on
building with the wrong driver and thanks to some emails I sent out over the
weekend and some Googling have now implemented the solution which is building
as I write this. Of course this is the easiest part of the whole project, but
it gives me a stable and monitorable base to work from.
The weekend was most excellent. Saturday started very well, and then remained
almost as good while we laid about the house doing not so very much at all. I
took the time to get some games reinstalled on my PC and finally killed
Homeworld: Cataclysm. Just like with Homeworld, once I'd take a break of a few
months it was easy. Cormac and Steph took us out for dinner as thanks for
letting them stay while the house was replumbed for heating. We went to a
noodle bar type deal where the service was almost too quick, the food wonderful
and the heating warm enough to chase away the chill outside. Back home we sat
and watched far too much Scrubs before heading to bed. Sunday was fairly good
with some cleaning, some tidying, some shopping and much too much game playing.
Most of the evening was spent playing Homeworld2 (or The Sims 2 in Elaine's
case). I don't think I stopped until some time around 12:30 this morning. We
didn't get to bed until 01:00.
[12:55] We've finally had the initial meeting of the 'what to do with the weird
and clever system that's undocumented and needs replacing' project. Already
two years behind there's not really a rush to get it done, more of a
pressing need which would be usefully furnished by getting on with things
right now. So, we have a first stage plan which is
basically 1) get a machine up and running, 2) get the right software on there,
3) start consolidating the multivarious configuration scripts on the old
development server into simple neat ones on the new one, 4) ask each developer
to come and look at the machine and ask in a whining voice why their bit of
stuff doesn't work correctly. The last part may take a while.
Once that's 'done'. I get to make the whole thing scriptable and add it to the
the build script, try it, fix it, try it again and then start on the live
I built a snowman this morning in the area behind work. Now that the snow's
all gone it looks pretty good out there by itself. A couple of people have
remarked on it, including one of the bosses who asked me to come back in.
Still, all done now.
[17:10] I think today was good. I know that decomissioning old machines is a
good thing. Especially when they're old Windows 2000 machines serving DNS. So
now I have a few more U's of space in the rack for my new servers. Other than
that I've been tidying the server room a bit, getting one of my servers booted
for the first time with an initial build script, working out why Big Brother
was going nuts, trying to figure out a way to get HTML::Mason installed along
with mod_perl for Apache2, PHP and mod_ssl, all without user intervention. I
don't think it's going to be easy.
I'm leaving some of the configuration to the people who want the servers so
they can do what they think they need to do and I'll turn it into scripted
actions at a later date. I hope. The machine I added to the backup actually
backed up last night, which is helpful.
I don't think wearing contacts and looking at this screen is one hundred percent
good for my eyes. I may go back to glasses and wear contacts on the weekend
only. Oh yes, we didn't go climbing last night because Cormac and Steph came
over to have showers (new boiler ignited today) and in the end we wouldn't have
climbed for very long once we'd arrived there. So, that's about it I think.
Time to go home.
[17:50] Kind of an exciting day today. While it started off slowly I did get
to provide some entertainment to Julian who's languishing at home with nothing
to watch. At lunch I went out with Shaun and my co-workers in Operations
Support to a pub where much food was eaten after a bit of a delay. I've just
realised that my packed lunch is sitting in my bag uneaten. I think I'll share
it with Elaine on the way to the climbing wall this evening.
Anyway, back from lunch (where we tried to set our working world to rights) I
started unboxing one of the five new servers I have to set up for the Web
Services team. Getting the IP addresses allocated and the rights ports open on
the firewall is going to take a bit longer. Either way the consistency checking
on the RAID array can happen while I'm away enjoying myself.
I found an abandoned PowerEdge 1400SC in a store room a few days ago. For a few
minutes I wandered around asking people who might know who it belongs to. I'd
like to use it for some test installs and perhaps as a dedicated RedHat
kickstart installation box (to take the load off one of the other machines).
Even if I don't use it for that, it'd make a useful Proof Of Concept platform.
Anyway, that's it for now, time to go home and prepare for climbing.
[12:10] Today we've been learning about rsync and how we probably should have
picked it up about a decade ago. The thing is that I've never had cause to use
it before now. Still, a new command assimilated and ready for production use.
I'm a big fan of --dry-run too.
I spent from about 17:00 to 00:30 last night getting my home machine up and
running with the new graphics card, a fresh OS installation and getting it back
to the same state as it was before. I think it was about time I moved from
that incarnation anyway, even Firefox downloads were starting to misbehave. I
doubt I would have been allowed to spend so much time (I also forgot to eat) on
it if Elaine had been around. I guess that's one of the few good things to come
out of her being away at the moment. I'm very much looking forward to her
return tonight. For the moment there's more modifications to be done to the
build script for the CVS server (and future PowerEdge 2850 machines) to allow
Big Brother to say hi to the RAID controller as well as getting the backup
regime in place.
[18:05] I've just spent the entire afternoon from 13:00 until about five minutes
ago working out and then documenting the bespoke XML-based backup system we
have here. I think someone was doing some bad crack when they came up with it.
Don't get me wrong, it's ingenious, clever and generally impressive and I doubt
I could come up with anything nearly as good, but it's so complex,
badly documented and basically impenetrable. It doesn't help that about six
other people have fiddled with it since it was implemented, and the person who
created it has left for somewhere else. I'm pretty sure I have a handle on
how it all works now and I can even add another machine to the regime and have
it actually get backed up when it's supposed to, but it's been a labour. As a
result I'm going to go home, make sure the house is tidy, and wait for Elaine
to come home. I may even go and meet her at the train station as a surprise.