[16:35] 161 Right, OK. This is the last journal entry for aproximately two
weeks as I'm off to Spain tomorrow morning. I'll be keeping a journal on
paper as I have done before now and transcribing it on my return. That could
take a while though so I imagine the next entry you read'll be April 17th's
and then I'll fill in the intervening period as I have time. For those of
you watching via LiveJournal those entries will be backdated, so you won't be
innundated with entries when I put them up, you'll have to go and seek them
out. Not that imagine you will, but the option's there.
Today has felt damned rushed, even though I haven't actually needed to be.
This morning I reinstalled Jenny's workstation for the temp who's starting on
Monday, got the Windows box updated and plumbed in with a KVM next to it and
tidied the whole work area. That section looks quite shiny and purposed now.
The remainder of the morning was spent plumbing in a UPS for my workstation
and watching the electricians throw big red toggle switches to take the
server room UPS onto bypass so they could install another set of batteries
and bring it up to 80kVA.
Over lunch I went into town and bought myself a decent pair of shorts for the
holiday as well as a few supplies. Must remember to take them home with me
I haven't really done much this afternoon except tidy my desk, remove myself
from some mailing lists and generally try to lock down as much as I can for
two weeks away from the helm. I hate leaving my machines for protracted
periods of time. No-one understands them like I do. Still, it's got to be
Amusing thing to leave you all on. I got an email from my potential new team
leader last night. Turns out he's an ASR denizen and an occasional BOFHcam
journal reader. How worrying is that? I'm really not sure what I'm going to
do about this particular 'small world syndrome' effect but I knew when I came
to this job that the amount of detail I put in about the place would have to
change and I don't think I've been anything other than honest about how
things go here. I guess we'll have to see what happens. Just thought you'd
appreciate the quirk.
So! Yes! Off home soon, get the packing done, settle down with a beer, have
some soup, crusty bread and ham, maybe have another beer, chill out for a bit
and then bed. In the morning I'll do one last email check and then power
down the house arrays, as it were. The train from Waterloo leaves at 13:41
and from there it's Paris and then South! Plenty of photos, lots of fun and
hopefully a chance to forget about everything that's on my mind, at least for
a week. Then it's my birthday and suddenly I'm 30. How weird is that?
[16:05] 162 Jenny (my outgoing team leader) left me and the Networks guy an
email last night to try and get load balancing working on another service
(just a testbed installation). I got in this morning to find that he'd
already started on it. I managed to give some good input before the end and
was able to provide my part of the operation in time to reach a successful
conclusion. Of course it didn't all work as I know next to nothing
about PeopleSoft (the application we were balancing) so it was up to Jenny to
show me the ropes in getting everything to behave as it should. I swear when
she goes we're going to be half way up shit creek without an means of sensible
Since then I've spent the time updating my kickstart installations with RHEL3
update 7 and RHEL4 update 3. With that and the kickstart USB stick of joy
sorted I'm ready to get Jenny's machine reinstalled tomorrow for the temp
who's coming in to keep her seat warm for a month or so. It's going to be
crap to have her go. She's been excellent in teaching me and generally being
the kind of laid-back sysadmin I hope to become one day. Her level of
knowledge is awesome and it's a terrible loss to the department. Still, I'm
on holiday for two week from tomorrow so I don't care if things go to pot for
Tonight is more eating things in the house so there's nothing to go moldy
while I'm away. I may also do some packing and washing too.
[16:30] An even fuller day of battling with the old version of Zeus'
ZXTM software to get it to do what I wanted with an HTTP/HTTPS site. That's
been a battle and a half. As per usual I know know a lot more about
the software, how it all hangs together and how calm I can remain when
confronted with a solution I'm not allowed to go with (software upgrade). We
can't do the upgrade for various reasons so the Networks guy and I had to
bodge something permanent and maintainable together instead. Not ideal but
we know it'll work and continue to work until we can do an upgrade-in-place
of the kit.
It doesn't really matter, we'll get around to doing it eventually. In other
news I was up and at one of the other server rooms for 07:45 this morning so
we could down a half dozen services and databases and their hosting servers,
move them physically across the floor and bring them back up again. That was
fun. Nothing out of the ordinary to be honest. Just earlier than I wanted to
be awake this morning.
I have to say that I'm getting more and more ready for my holiday next week
though. I so need a break. I imagine that after the trials of getting to the
villa (we're going by train all the way there) I may want to sleep in just a
little for the first day or so. I'm very much looking forward to it,
especially the second week when the remaining persons come out and it's my
birthday. Did I mention the holiday much? I probably haven't. Remind me
tomorrow and I'll give you some details.
Hmm, I'm tired. Climbing this evening is going to be fun. I may just do some
light technical stuff.
[16:35] A day full of uninstalling old UPS software and installing new UPS
software, working out which way to get it to the right machines through the
firewall and generally dealing with Java issues. Most of the afternoon has
been spent between that and trying to get load balancing working correctly
with an HTTPS/SSL domain. In the end I set up the newest version of the
software on a spare v20z and have been fiddling since. More on that tomorrow
I think. Gym time now, the pub. Early start tomorrow as the UPS that was
removed from here and take to another server room won't fit in the door until
the server I moved to allow it in is moved elsewhere so it really can
fit in the door.
[13:00] A weekend of much fun, oh yes. Saturday I decided to get an early
start and was up, out of the house and in London for about 10:00. This meant
that I got to start climbing at The Castle as soon as the sleepy staff had
opened the doors. With the entire set of bouldering walls to choose from I
was able to warm up on some 5a routes before ramping up gradually to 5c and
the occasional 6a within a few hours. I climbed for a few hours before I ran
into some people who offered to do some roped routes with me. We did that for
a while before I headed out at 13:00 to meet a friend in the local pub. We
chatted and got to know each other while I had some lunch and then headed
around the back of the climbing wall building to play with poi and staff and
flags (something I'd not seen before). I now have to put some serious effort
into learning the 'three beat weave' which currently allows me to swing one
poi with stunning accuracy into my groin.
This will change.
After the friend had headed off to find a shop selling wool for knitting of
funky things I jumped back on the wall and did a few more hours bouldering
before doing some more roped routes with Elaine and a friend of hers she
was climbing with, and then switching back to bouldering for the last few
hours after they left. Frankly by 18:00 there wasn't much more I could do
with my fingers than have a few abortive attempts at untying my laces and
working zips. As a result I decided not to head into central London to meet
friends, which was a shame but I'm not sure I would have been great company
given the effort I'd put in on the Wall.
Home; I slept very badly. So much so that in the end I got up and watched the
sun come up. It wasn't so bad as I had to be up for 0700 GMT (0800 BST which
we're now on here) to deal with the day's worth of power work in the server
room. Now that the new UPS is in and all the new commando-terminated feeds
are in under the raised floor we had to pull up all the tiles, power down all
the services and servers, swap over the feeds to the racks, rip up all the old
cabling (a crow bar used in some places) and generally try to balance the
phases with some modicum of effort. Suffice it to say by 16:00 we had almost
everything back up, the phases vaguely balanced at 35 36 27 and a huge
collection of scratches, cuts and bruises from moving and removing bits of
cabling, getting under the raised floor and generally lots of manual labour.
I headed home, thankful that the SAN hadn't thrown another fit.
Elaine was round fixing up her old bike to allow James to have some non-car
transport for around town, etc. I gave her a hand with some of the grunt work
while trying not to tell her about the effect the damp was having on her
hair. I do have some photos though, but she's going to have to vet them for
public release. After a while the efforts of Saturday and Sunday required me
to just fall down on the bed and relax for a bit. Exactly what I needed, and
very pleasant. Beds're good, m'kay?
The evening was spent watching new episodes of The West Wing and Family Guy.
The first time I'd turned on the television all weekend and it's for a DivX
feed. I'm glad I cut back my NTL packages. I'm not sure I can live with
slower broadband though...
Today I've been getting back up to speed, ticking things off my list of things
to do at a rate of knots and prevaricating about getting on with load
balancing the web servers. I really need to get that done, get my team leader
to brain dump her job to me (she leaves on Thursday) and get all my RHEL boxen
updated to the most recent errata levels. That and work out if I go climbing
this evening. Oh, and cleaning off tarball installations of things the web
developers have put on in place of maintainable RPM versions of same. It just
doesn't stop. But right now I really don't want it to. I'm loving being
busy, and learning things. I love the learning. Eventually I'll find someone
to witter about all of this to who doesn't scream. You're lucky, you can just
[22:35] Crivens, what a day. Last day of the course. For much of the day we
spent time talking about Flash installs of Solaris while waiting for things to
happen. Lunch was spent cycling around town picking up free toner for the
laser printer I have at home. The course finished early so I headed home and
tided for a while before heading to the gym for a damned good session. After
getting back I dashed out to the shop to get some milk for the weekend and ran
into my old administrative manager from my previous job. I've always thought
she was nice, and not working for her any more always a bit of a shame,
still we got to chat for a while about the atrocious state of the IT support
there now and how I might get to be on the appointments committee for the guy
to replace the guy who 'replaced' me. Apparently there are a lot of comments
about how things were when I used to work there. Nice.
I've had a lovely load of spaghetti bolognaise this evening with bits of bacon
in it and intend to have something else as soon as I can work out what is is.
In the meantime I need to work out some way of stopping my left sinus from
exploding. I think this is the tail of the cold I had last week giving me one
last prod for getting too little sleep these past few months. I've been eating
my greens, getting enough fruit and vitamins, I'm not sure what's happening.
Still, otherwise I'm feeling fit as a fiddle and I've lost a lot of the excess
I'd gained over the last few years or so. No, I'm probably not eating as well
as I could, but every week I'm coming up with new things to try. I guess it's
lucky I don't have anyone I'd be inflicting some of my more interesting
experiments on . I'm not dead, incapacitated or otherwise as yet though so it
can't be that bad.
Tomorrow I'm heading to London to go climbing for the day, meet up with a new
friend for the first time and then probably have a few cocktails at someone's
birthday bash before heading home. I can't stay late as a) the clocks go
forward and b) I need to be in work for 08:00 (really 07:00) to start powering
down servers for the second (and hopefully final) stage of the power work
which will affect servers in the machine room. With luck I'll be home by
early afternoon and be able to chill for a bit before Monday. Next week is
my line manager's last week, so I'm damned if I'm not going to get her to
brain-dump as much of her important stuff as possible before she goes.
[14:15] A monster session at the climbing wall last night with only a few silly
mistakes on my part. Hopefully they went unnoticed for the most part or
weren't that bad. We arrived really early and got do do a fair few routes
including one or two I'd not done before on account of just not getting around
to them. I'm intending to do a ton of bouldering on the weekend to get away
from muscle-and-pull climbs and into good technical routes for a bit. There's
so much more Wall at The Castle now that I'm sure I can lose myself in the
routes for a while or until someone I know wanders past. I've a few friends
I've made there who go on the weekends so I imagine at least one'll happen by
if I'm there long enough.
We reinstalled our machines on the course before going home early yesterday so
I had to reenable external connectivity before I could update my
journal and check my email this morning. The course continues apace and my
level of Solaris 10 knowledge gradually increases as I fill in the nuggets of
information not imparted by the '9 course I did a while ago. My coworker
beside me continues to play Patience and I have managed to rack up a
respectable score of 5160 on Solaris 10's Same GNOME game. Email from
work comes in to tell me that the web developers have been installing software
on their (my) systems and then telling me about it. I leave myself numerous
notes of the things I need to do when I get back next week and try not to grind
the enamel from my teeth as I sit here, powerless, mostly isolated from my
machines. Tonight is the regular Thursday thingy, hopefully a small quiet one.
[12:15] Immense stupidity last night, hopefully rescued most of my mistake and
haven't caused too much damage. The more I think about it, the more I realise
it was all my fault and something that could have easily been avoided with just
a little sensible thought. I'm sure none of you really want to know what I'm
going on about. Suffice it to say learning to walk before I try to sprint in
the Olympics 100m event would be a good thing.
Not sure I'm eating terribly well at the moment. This is especially silly
given I went to the gym last night for an excellent session. I'll be making
more of an effort in the future. I still much prefer cooking for more than
just myself though.
More course today. Slightly more interesting than yesterday; things are
gathering pace slowly but surely. Climbing this evening, nice to see some
people I've not had opportunity to cross paths with for a while. Also thinking
of more climbing this weekend to counteract the fact that I'm going to be in
work for most of Sunday. A roast dinner would go down a treat in Sunday
evening I think.
[12:50] Someone appears to have configured and brought up a virtual
interface on their Solaris 10 machine and gained access to the rest of the
world again. I won't be doing much with it, it's more of a technical exercise
than anything else... but it does allow me to update my journal, check for real
important work emails and have something to do during the really dull (as in I
know what's going on) bits of the course. Not being much of a Solaris admin as
yet it was actually a rather useful bit of experimentation given that I don't
think it's covered in either the Intermediate or Advanced courses I'm doing.
I am rather tired, I have to say. Last night was probably a bit too late to
stay up all things considered. Luckily this isn't the most demanding of
courses as yet (I've a very good idea which days will be) so it's not so bad.
I thought I'd have a lunch date today, but they cancelled. Hopefully that'll
happen later in the week instead. If I don't write anything more today I'm off
to the gym this evening and hoping to have a(nother) quiet night in.
[21/03/2006 - 00:50] I'm feeling very quiet and tired right now. It's been a
long day and I've had all kinds of things going through my tired and puzzled
head. I've spent much of the day alone, even if it's just been inside my head
while in the middle of a group of people and while I'm growing to enjoy the
feeling gained from solitude and alone-time there's still a sense of something
missing. I'm sure it'll come to me eventually.
I cleaned the kitchen from top to bottom on Sunday. Yea verily, even unto the
fronts of the cupboard units and behind the cooker. Sunday evening
was spent having interesting conversations online. The weekend was, on the
Today was mostly spent in the first day of the Solaris 10 course cruising
through the notes at the speed of an arthritic snail. Considering this is
supposed to be two one-week courses in a week I'm wondering just how things
are going to pan out.
The very definite highlight of the day was being in on a recording of a test of
Mitchell & Webb material by the men themselves (plus some other people including Olivia
Coleman). Met Andy at Goodge Street Tube station and spent some time talking
about the current states of our respective lives and just where we think we're
heading next. I swear every time I meet that man and get to talking we end up
discovering more about ourselves. Heaven help us if we get drunk together,
we'll probably end up solving all our problems in one evening. Suffice it to
say an absolutely epic evening was had and I even enjoyed the train
and cycle home which brought me to my door at gone midnight.
I'm now tired, confused, elated, wistful and thoughtful. I think I need some
sleep and to just let things wash over me and go away again before I risk
[16:30] More stuff done. Spent some of the morning getting a server over to
one of the other machine rooms and plumbing it in. This went fairly well
except for being told that the rack we were putting the machine into was
apparently slowly sinking through the floor. Good job it's being redone to
take the weight of a new UPS shortly.
Back at the ranch I've been raising virtual interfaces, modifying web server
configs and being stymied by a lack of team leader to decide how we're going
to modify the VLAN configurations and the load balancers such that we can do
what we need to do without exposing our virtual bottoms to the world. I'm
tired and busy and a wee bit lonely. I think I'll head to the gym this
evening and then work out what to do after that. I think I want a quiet
weekend this time around after last weekend's sudden exciting fullness. Maybe
if there are people around on Sunday I'll do something then, or cook a meal.
Oh, nearly forgot to mention that I went to a recording of the News Quiz
(Radio 4, UK) last night with Andy. Bloody excellent session and a Burger
King afterwards. Two reasons to thank Andy for a great night out. Shame I
had to leave early but them's the breaks when you need to get home before
midnight for once. Listen to it this evening or on Saturday.
I should probably mention that I'm going to be on a Solaris 10 course all of
next week so journal entries will be either short and to the point (i.e.
boring) or long and involved (still boring, but taking longer to read).
Hoping to go to London on Monday to a Mitchell and Webb recording (my last
trip to London for a while I imagine) and the usual climbing on Wednesday. A
week on Sunday I'll be in work moving things across from one UPS to another
(not that we're getting rid of the MGE one, just moving it elsewhere). If
anything else comes up I'm sure I'll get around to telling you.
I'm off shortly after having written a big long list of things I need to chase
up a week on Monday. Have an excellent weekend and stuff doing whatever it is
you do when you're not reading this, and I'll see you occasionally next week,
[14:35] It's my brother's birthday today. Happy birthday to him. I, on the
other hand seem to be getting over my cold (touch wood) and still managed to
have a reasonable climbing session last night. It's a shame my climbing
partner wasn't really feeling in the mood as it's always good to have someone
to bounce your enthusiasm off of when you managed to do a tough route.
This morning I got in to see that the powercut emails I'd received last night
were true (I'd been tempted to go in, but frankly there wasn't much I'd have
been able to do) and not false alarms. I've spent the morning trying to work
out why the UPS failed to send out shutdown orders again and engage
in some proactive firmware upgrades for it at the same time as doing a new
Solaris 10 install (01/06 release) on the V20z. This install looks a
whole lot more 'together' when it comes to opteron-based systems,
seems to behave a lot better with the stupid serial console stuff and
generally gives the impression that some thought was put into console-based
installation methods. Not only that but it doesn't require an ITU disk during
the setup either! This for me is a Big Plus. As soon as the network guy gets
back from whatever he's doing I'll get him to open the firewall to Sun on
HTTPS and I can start having a play with smpatch(1M).
More tickets arrived today, which is nice. Now all I have to do is make a
payment. Speaking of tickets I'm off to a recording of the News Quiz this
evening in London. Meeting the mighty Mr Disco for some fun and games (if he
turns up on time) before heading home. Not sure what I'll do this afternoon
if I don't have smpatch to play with... probably one of the other two dozen or
so things that need doing. I need a lovely holiday from the evil work.
[17:10] Ooops, busy day, no journal entry. Today's thought is: Sun V20z have
terrible serial console response while in the BIOS.
Dull, I know. Time to climb.
[11:45] After being teased mercilessly by at least three women about whether
or not I have the flu or just a cold I'm pleased to report that It's Just A
Cold, now stop prodding me. This is why bird flu is deadly serious, and bloke
flu is just a sniffle. Anyway, I'm in work and probably spreading the lurgy
to all and sundry. Still getting stuff done, which is good.
I went out to the pub for a little while last night. There was a group
playing jazz to what eventually became an empty bar (except for me). I got
them to play some favourites (and they were pretty good all in all) but then
decided I needed an early night for once. Not that I got one with needing to
blow my nose most of the night. I ended up getting in to work for 09:00 this
morning and have been doing small jobs and keeping away from things I could
break since then. Oh yes, I managed to identify the last of the birds from
Sunday's excursion to Welney. It was a White Cheeked Pintail. Apparently
very far from home!
I may not make it to the gym this evening, although I do have my stuff with me
in case I feel like really doing myself some harm. Either way I think a good
meal this evening and the finale of Battlestar Galactica series 2 before an
[11:10] A slightly more packed weekend than I'd been planning for after all.
As per usual I went to bed very late on Friday night after a rather good gym
session. The lateral crunches and the back hyperextensions I'm doing now
really seem to have got rid of the 'excess' I think I'd previously accumulated
around that part of my body. Anyway, I'd assumed that Saturday would be spent
at home relaxing and contemplating why I wasn't going out to take photos so I
was rather surprised when I found myself out, taking photos. I'm actually
quite pleased with some of the shots:
What was the first of a few pleasant surprises was that there was a spare
ticket to a Ball for Saturday night. I was informed of this about two hours
before it was due to start. I dashed about the house and managed to pull
together my DJ and verify that a) it still fitted and b) it was clean. Hurrah,
free fun at a Ball. The second surprise was that Dunk was bringing his new
girlfriend Eve (another spare ticket), so I had to remake the bed for them and
relegate myself to the single bed as Cormac and Steph were already on the
double futon. Not a trial, just more fun dashing about feeling useful.
We all toddled over to Keith's house and had some pre-Ball drinks before
taking a taxi over to the place. Cue two hours of standing around in below
zero temperatures taking pictures of the firepoi and firestaff demonstrations
and trying to keep the women in open-toed sandals from getting frost-bite.
The Ball itself was a usual noisy, bustling busy affair with lots on and tons
to do. I'm glad I've been getting lots of late nights otherwise I would have
been wiped out fairly early on. As it was most of us went until about 03:00
with the hardcore people probably lasting all the way through, I should ask.
Sunday morning was mostly about trying to figure out where my really sore
throat and rather unhappy stomach had come from. I'm pretty certain something
I ate did for my stomach and breathing in the shared air of people from all
over the country probably gave me something which appears to be developing
into a nice case of the flu. Still, bacon sandwiches with Cormac and Steph
was just what we needed before driving a ways to get to Wenley Wash WWT. This
was my first ever proper bird photography expedition and I was extremely
hopeful that I'd come back with some good shots. Naturally, I hadn't set the
ISO on the camera properly so many of the shots are quite grainy, especially
when cropped and zoomed. I'll do better next time. Of course, there were
people there with far better lenses than I (Canon EF 100-400mm L IS USM) but
they were more than happy to let me take a gander (no pun intended) through
the viewfinder. Of course, now I want one. That won't be happening though,
not for a long time considering my financial situation at the moment. Still,
the 70-300mm performed admirably considering the ineptitude of the user.
There's a collection of the best shots up now on my gallery and three of my
favourites here below:
I'm pleased, but not extatic. I think I'll see how I progress. If you know
the names of any of the ones I've marked as "????" please let me know.
[16:00] Busy busy busy. A morning of kickstarting and fighting with Perl
modules which unaccountably fail to install now having done so flawlessly on
previous systems. Some excitement with Procallator and lack of portfast on
some switch ports and generally getting small and annoying things done.
Testing and rolling out all the updates that make a machine running RHEL 4 a
RHEL 4 Update 3 installation. Eating my lunch without spilling things down
myself. All these things and more are that which have made up today. I
got in this morning after taking some (possibly crap) shots of the blackbirds
and chaffinches making a welcome return to the bird table in my garden. I
really hope that they start being regulars. I like collared doves (even seven
at once in a crowded mass of cooing and feeding) but the smaller birds are
especially welcome. Here's hoping.
Other than going to the gym this evening I don't really have anything much
planned for the weekend at all. I'm open to suggestions. Hopefully I may be
meeting someone in the pub on Saturday evening, if only for a few hours and
then heading home again....
... Although in the last few minutes it turns out I'll be providing crash
space to some friends on Saturday night and then hopefully going to Welney
Wash on Sunday to get some better opportunities to shoot some bird pictures
with the 70-300mm wallet-killer. Must remember my bird book.
I wonder what to have for dinner tonight. Have a lovely weekend whether
you're traveling across the country or staying right where you are.
[12:50] I think I should learn to dig up. Every time I think I've
got a handle on things I go and do something silly that just makes things
worse. This almost certainly isn't doing me any favours. I wish I was better
at stuff sometimes.
About the only thing going for me at the moment is that last night's climbing
was fun for me and hopefully for everyone who came. I did some leading for
the first time in a while and even got to use some quickdraws. Unfortunately
it seems like my rope - even though stored somewhere dry - felt a little damp
so we ended up using James' instead and I've laid mine out in another room at
home to see if I can dry it off. I enjoyed my climbs, got some stress out of
my system and got some good 'fingery' boulder routes done. I was impressed
with how good Elaine is now. It's been a while since I last got to see her
Today I've been tweaking Subversion for people for most of the morning,
setting up services to start on boot, trying to update documentation and
really, really failing to have the enthusiasm to do so and putting my foot in
It again. I really should stop doing that. I'm heading home in the bright
sunshine and rain at lunchtime to pick something up I should have put in my
bag this morning, then racking up and kickstarting a server this afternoon.
With luck I'll be meeting a friend I've not seen in years after work this
evening and then making it home for the regular Thursday thing, if anyone
wants to come along.
[I'm sure I was going to write something else here. Something of great wit,
importance and weight that would cause dozens of you to comment and argue for
days as to the fundamental rammifications of my postulate. However for the
life of me I can't think what it is because my Team Leader's phone went off
and everyone in the office started whistling the tune spontaneously
independently and reduced us all to fits of laughter. By the time we'd
stopped the person ringing had rung off. I should head home while the weather
[16:40] New server partially plumbed in (DRAC and serial console connections
will be snaffled from another box for the duration of the setup). I've set up
the kickstart scripts, got the USB key primed and ready and now just need
morning to come so I can as the Network admin to do the firewall rule changes.
In the meantime I'm waiting for my phone to ring so I can leave work and head
to the pub. Life a little better this afternoon. Hopefully I can hold things
together for a bit longer this time.
[16:00] Tired and a bit down today. Have been keeping brain occupied with
updating all my documentation on how the kickstarts work for my linux boxen
since I changed the monitoring to use Dell's OMSA suite and binned the kludgy
stuff we were using before that I threw together. Also it's an opportunity to
get a kickstart setup ready for our new web server which is going to be a twin
to the one we have and will be acting as a failover box for once the one in
service is behind the loadbalancer with it. Moving the SSL certificates and
getting all the IP/DNS stuff done is going to be 'fun' as I've not done it
before, but a lot of interesting new stuff to do. There's a plan to see how
much of this round of work can be pushed through before I head off to Spain
for the first two weeks of April for my birthday thing. This could mean some
hectic days in the next few weeks. I'm looking forward to it to be honest; I
do some of my best work and think less about other things when I'm in the
thick of things. This is one of the reasons why I go on and on about
climbing; there's nothing better than shutting down your higher-order
functions and concentrating on "this hand goes here at this angle. Can I get
another finger on there? Oh, what about that foot turning slighly?" and
forgetting about what was bothering you. Not that you forget about what's on
your mind, but as it's obvious that life's moved on and there's no point in
spending any more time on that which cannot be changed you are able to just
put it to one side and concentrate on not falling. I come off the Wall most
times feeling a sense of euphoria (which can probably be explained by
endorphins) which fends off other thoughts for the evening. Climbing with
other people who feel the same way, or are just good company is a bonus.
Now can you see why I hate missing climbing?
[17:00] Just had a very productive meeting with regard to web servers, machine
builds, load balancers and certificate shifting. It's left me with a ton of
work but it's all very much achievable and probably going to be mostly quite
Went climbing last night and, while I didn't do too many technical climbs
(I'll do those on Wednesday I hope), I did get some decent exercise and get to
show someone else how to climb correctly (i.e. belaying, knot-tying, correct
signals, etc.). I think he pushed himself fairly hard but the most important
thing I think I got over is that you don't need to rush at any point.
When I got in this morning both tape drives had thrown a fit and NetBackup
had downed them both. Luckily this seemed to be after the backups had
finished and it was only the offsite duplication which had failed to complete.
I tried getting the robot to remove the tape from the drive but it wasn't
having any of it. I powercycled the thing with no joy. Oddly, by the time
the engineer had come out (with a replacement drive en route) the tape had
been removed. Very odd indeed. We ran the drive through some test backups
and found no problems. However the engineer felt that at least three of the
tapes didn't feel right mechanism-wise so I've removed them from circulation
and put in some new ones. With both drives up and everything appearing to be
working correctly I've left it all to be picked up by my co-worker tomorrow.
I'm off to the gym this evening then making a phonecall in the hope I can save
myself some money. Noit sure what to have for dinner. It may be more
cockerel soup with bits of Parma ham in it.
[10:30] Everything seems to be working at work, which is good. I had a quiet
weekend. Went out to a nearby place to try out the new lens. No particularly
good shots (I was looking for birds) but one or two that made the cycle out on
Sunday morning worthwhile for more than just the pleasant trip.
I've been feeling a little low and worried about things recently. I've found
that the best thing is just to keep on going and not try to waste time on the
whys and wherefores or bother other people with it and to get on with the
things I can control. Obviously this isn't something I've been particularly
good at, especially in the last day or so, but life always picks up again in
the end and if you're positive there seems to be more chance of it happening,
don't ask me why.
I'm hoping to go climbing this evening. It's a nice cheap way to get some
fun into my life at the moment and it's something I can share my love of with
other people. I'd like to head down to London again soon though as the walls
there are much better.
Tried marinating chicken in mango and apricot with ginger chutney last night.
Not a complete success but it caramelised something wonderful and tasted
superb. Maybe next time I'll use a real marinate and see how that goes. I've
got a ton of shopping in the house anyway, so plenty of things to practice
with. It'd be nice to have someone to cook for too, but I'm not sure who'd
want to eat my experiments.
[18:05] (This entry written initially at 13:45) OK, so the SAN pretty much ate
itself last night. My co-worker stayed all night waiting for a Sun engineer
who turned up at 03:00. He worked the morning with my help (checking he
wasn't typing the wrong thing if nothing else) getting hosts up without fibre
fabric connectivity so we could have a go at restoring where possible. For
the moment we're onto our second Sun dude (the first one's gone home for some
sleep), my co-worker's headed home to get some shut-eye and I'm not so much
holding the fort as holding it together.
I've also been fiddling mightily with procallator (the linux bit of
Orcallator) in the hope I can get it to work with 2.6 kernels rather than the
current 2.4 kernels of RHEL 3 (RHEL 4 is 2.6-based)). I think I've
managed to pull out the stuff we need for a new installation build thingy but
there's still a warning/error thingy which pops up after a few minutes. Not a
major hassle, but it still shouldn't be there. It's kind of academic at the
moment as the Orcallator server is one of the ones affected by the SAN loss.
I tried to ask the Sun engineer what's happening with the SAN but he's Borged
himself into it so completely I couldn't get him to respond without damaging
the SAN itself. I've generated an interrupt in the hope he'll detach and tell
me what the status is.
I'm not joking.
Amusingly the LEDs on the back of the SAN (of which there are _many_) are all
green. He's even lit like he's inside a cube.
OK, it's 18:05 and we're fairly certain the SAN is pretty much fixed. Maybe.
I've learnt more about luxadm, cfgadm and fibre channel configuration in the
last seven hours than I've learnt in the last two years. I'm happy to say I
called what the root of the problem was, even though I didn't know why it was
the problem. I've missed the gym, again. I'll go tomorrow instead. Right
now I just want to go home and try to unwind for a bit. If you're at all
interested in what the hell has happened here in the last forty-eight hours
email me and I'll write a response I can copy to people. I won't put it here
as it's kind of unofficial and stuff. Anyway, looks like the fun's over for
now. "Time to go home," said Zebedee.
[14:20] There's a complete power down this evening for this building. This
means everything is getting turned off. The reason is a new power
feed such that the new UPS we're having installed is ready to be commissioned.
The fun starts at 17:30 and will probably go through until at least 21:00.
This means no Thursday evening people around. Luckily I still get my Dominoes
pizza fix as we're ordering into the building. The Networks guy and I are
going to take the opportunity to completely rewire two of the cabinets we have
here such that they're actually managable to add new boxes to without
suffering mental anguish.
I've spent most of the morning on the phone to the Council Tax office and my
bank trying to sort out what I thought was already sorted. I was told to
delete/cancel my old direct debit when I got the discount for single occupancy
(which necessitates a new account number). Turns out that this means the CT
office couldn't seamlessly move over the DD to the new account number. Maybe.
I've reactivated the old DD (did you know they could do that?) and am hoping
the billing tape the CT office send to the bank will do the reference number
update I'm hoping it will. If it doesn't there's a manual payment to be made
and a DD mandate I'm going to have to go in and fill out.
In other news my final Play and Amazon deliveries for the forseeable future
arrived this morning. Firefly and Serenity DVDs and some books (Collins
Bird Guide, The Time Traveler's Wife and The Kite
Runner). From now on (don't mention the lens) it's saving time. This'll
probably mean a lot less climbing in London, less eating (and ordering) out
and generally tightening my belt a little bit on fripperies so I can bank the
spare. All in good causes though; going to Spain and the upcoming Discworld
Convention and other things are going to eat my savings for dinner.
[17:20] So I leave the office for ten minutes to pick up a head torch from
Cormac just down the road and while I'm out some dozy bugger swings a chair
into my PC's case and knocks the power out enough that it reboots. This loses
me about an hour's work. I've a good mind to go through to his office and
insert a chair into him. Power down begins in ten minutes.
[21:00] Power back on at 20:30. Powerup is pretty much complete apart from a
few extended POSTs and some channel failover issues on the SAN. Worries with
the NAS have failed to be proven true. We've eaten an imperial buttload of
pizza, sat around in the dark and recabled some racks by headtorch-light.
Once the machines are all back up we can go home and I can get some sleep.
The worst thing was waiting for Sun to call back with a shutdown sequence with
the SAN, which is probably why we're having issues with the thing now it's
back up. Stupid Sun, stupid SAN, stupid power.
[22:45] Right, everything up that can be done now. Email sent to DBAs with
snags list. Home time. See you bright and early tomorrow. SAN's a bit
broken but co-worker's on a mission to get that fixed.
[09:30] What a beautiful day. Very cold but with a perfect blue sky. I
listened to David Cameron being badly interrupted by one of the Today
Programme presenters on Radio 4 and then toddled into work in the cold.
Lovely to cycle in such brilliant sunshine.
Last night was pretty good, I think. I got home and cleaned up the house, ran
the vacuum cleaner around and whipped up the first batch of batter (spraying
flour and half-mixed batter lumps everywhere). With some loud music on in the
kitchen I became a whirling dervish of organisation, stopping only to do as
much washing up as possible before people arrived. As they did we congregated
in the kitchen until there were too many to be sensible at which point we
spilled into the lounge where new arrivals pounced on the Geomag on the table
and it was theirs for the evening.
After a while of sitting around and talking someone suggested actually eating
some of the masses of batter and toppings we'd accumulated. Manfully taking
the pancake-making upon myself I set to and was quickly turning out pretty
bloody good pancakes with a utilitarian low-key flip half way through. The
batter level went down and the booze levels went up as people popped out for a
refill at the One Stop nearby. More chat, more relaxing and rather too much
sugar (chocolate spread, maple syrup and sugar, anyone?) meant that
by the time the last pancake was done I'm sure some people (including myself)
were suffering the beginnings of a sugar crash. Around 23:20 people began to
fade away and head home, leaving me with less clearing up than I'd imagined,
which was nice. I'd like to think it was a rather good evening; I certainly
enjoyed having the house full of people, cooking for others and generally
listening to everyone talking and expounding on their thoughts. Hopefully
everyone else is glad they came too.
[10:45] OK, today I intend to be out from 13:00ish changing a RAID battery in
town, then going to the Preferred Suppliers thingy in town after stopping by a
bookshop to pick up some useful books...
Heh! I've just been into machine room as I normally do and glanced around as
normal, paused, felt confused, realised that among all the LEDs that're
visible in the place there was one extra light on the front of the VMS box's
myriad-lit display that wasn't normally lit, and informed the VMS admin.
Turns out there was a fault that hadn't been flagged up. Odd how the brain
notices even slight variations amidst complexity.
[11:45] OK, for your delectation, amusement and general unimpressedness I've
now put up the shots I took on the weekend of the band
Self Inflicted that my
colleague and friend plays bass for, in the hope that one of them'll be good
enough for the insert for their upcoming CD. Also some selected shots of the
Beth Orton gig I went to