[16:55] Ooops forgot to do an entry for today. Well, nothing happened except
that I went for a run and nearly killed myself and put a lot of other things
right again after yesterday, which is excellent! Now to go home.
[17:40] What with one thing and another I've not achieved a whole lot today
work-wise. The morning passed in a haze of UPs communication errors (my fault)
and tnsnames.ora problems. The first of these issues was down to an out of
date install package with SSL security still enabled being installed by our
jumpstart process. That was probably my fault as I'd not told the person wot
does the installs where to get the new one. The second thing was brought to
light by one of the developers who was using the recently installed Oracle 10g
client I'd stuck on one of those machines I keep talking about. Seems he had a
Perl application which needed DBI::Oracle. Attempting to use it before it was
installed he found that not only could he not get to the machine, Oracle denied
all knowledge of it anyway. Turns out the tnsnames.ora file was way way out of
date. Fixing that, getting the firewall opened and actually installing the
correct Perl module took a little while but all seems fine now.
This afternoon I treated myself to an icecream from the van which comes round
because frankly I could do with the sugar buzz. It didn't work, being very
much of the Mr Whippy school of ingredients (mostly air). I hate email, that is
all. I can see things coming down around my ears unless technology improves.
I'm quite tired trying to keep things together, doing my increased exercise
stuff and holding down a job that's by turns completely dull and requires me
to write reams of documentation or is completely
insane-busy-all-hands-to-the-pump. I need a break, or at least some more fun
to counterbalance the frustration at work.
[11:00] Well, I did an unaided install of the new software this morning in
pretty much thirty-seven minutes. It went flawlessly. I'm still unsure as to
whether this is a good thing or not. I remain poised for the other shoe
dropping. My guess is that it'll be next year when we roll out and go live and
use a completely different version of the applications which are probably far
less tested. Not only that we'll be splitting the system into DB, application
and web front end, rather than having them all on the same machine. That'll
make things a bit more exciting. That and needing to use mod_jk and tomcat.
Thus far though everything has been pretty great and I'm pleased. The
consultant has been helpful and allowed us enough rope to explore for
ourselves, the documentation has been very complete and even mentions when we
may get warnings and/or errors and the software seems to be fairly
straightforward and doesn't have an interface which sucks utterly. I don't
know how I'm managing to stay sane.
A colleague's gone to power down and bring back an old E250 and D1000 which are
buried deep in one of our Dell racks. I'm not sure if he's going to be able to
get them out without damaging himself, the rack, the kit or the other stuff in
there. Still, he's rebooted a machine for me to which froze last night during
a regular update cycle. Oh, and I've sorted a backup issue on one of my
servers as well.
[17:25] Today seems to have been an extremely good day for all kinds of
reasons. There's been some fun and laughter online, a failing Samba update
thingy has been solved with some excellent firewall diagnosis, more progress on
my running (wasn't nearly so tired today afterwards, probably because it was
cold enough to see my breath (in July, in the middle of the day!)) and a couple
of bits of documentation if not knocked on the head, then severely brushed with
Of course we have a new Samba problem that I currently have no idea
how to fix. One of our linux boxes has been connecting to a pair of Windows
shares for years now. Recently I built up another box to replace it and got
around to mapping the shares as mount points. When you go to certain
directories and issue an 'ls' command you get nothing back, not even the '.' or
'..' files. Repeated execution of 'ls' may sometimes give an incomplete files
listing. When I look at dmesg it contains a load of these errors:
"smb_proc_readdir_long: name=, result=-2, rcls=1, err=123" which I'm pretty
sure is to do with long file names and lots of files in the same directory.
Only until someone tells me differently nothing's changed in those
directories in months. This happens with RH 9 and RHEL 3 with Samba
3.0.14a and 3.0.9 respectively. Bit stuck for ideas at the moment. Google
seems pretty unforthcoming.
Other than that little blot on the landscape things are going swimmingly and
I'm looking forward to things happening in the future which may be extremely
fun! No climbing this evening as our driver appears to have pulled a muscle or
trapped a nerve or something. Probably a good idea that I don't climb given
I'm running 10km every other day now and doing a reasonable weights session
[17:25] Things have been far better today. Aside from being very tired this
morning and only doing a shorter workout before showering (although as
recommended by Elaine I've added oblique crunches back into the routine to try
and get rid of my handles. Ouch) I got in in good time and was able to do all
the things I needed to before starting on watching the remainder of the testing
for the software we're test installing for the new HR/payroll system. What was
cool is that it all seemed to go off without a hitch after the DBA I'm working
with had corrected some accidental typos in the tnsnames.ora files. That went
on for most of the morning with some exhaustive testing of the client side
almost sending me to sleep at one point. Very much out of my zone of
We broke for lunch and wandered over to the place across the road for some
refueling and ended up talking about motorcycles for most of the time. I'm not
sure the consultant was really a petrolhead but he seemed to get into it. I'm
not either to be honest but the two DBAs who came along are very much so so it
The afternoon's been spent watching the DBA do an install of an entire instance
of the software with me riding shotgun (watching for typos/brainos) and the
consultant making comments if appropriate. Everything went without a hitch,
which in itself is a little strange. We ran through everything including
patching and upgrading a remote thick client. All seemed fine. I'm really
waiting for the other shoe to drop to be honest. The instructions are
complete, they list common problems, even show when you should expect to see
warnings and what they'll look like and and and and it all seems to
work. I'm officially freaked out, but am not going to say it's all
working until later in the year when we actually bin this version and jump to
a completely new version that doesn't use a thick client but just uses a web
browser for the client interface. At that point we're probably going to split
the DB server from the application server from the web server (currently all
on one (two (for testing different instances)) machine(s)). Hmm, hope you can
parse those brackets correctly.
Anyway, it's been a very interesting day as far as work is concerned, and
there's been some progress on other fronts too, which is always good. I'm
quite tired now so I'm going to go home and maybe upload some pictures of my
[17:40] What a day. I've been sitting (well standing) watching someone do an
install of the software I've got to install myself tomorrow. It wasn't so much
dire as I wanted to get in there, start doing things at twice the pace. In the
end I just let it go as I was too tired to bother getting worked up. I ran at
lunch time. That was hard too. Running alone and getting stitches isn't fun
when you're tired.
We completely failed to either get shorts or do gardening on the weekend. On
Saturday we ended up going for a long walk in the countryside and on Sunday it
rained so Elaine went out and got some wood for a shoe stand for the hall. She
made it too, and only made one small mistake which she's going to fix shortly.
I took some photos which came out reasonably well considering the subject. All
in all it was a pretty good weekend for relaxing. Of course Elaine managed to
end the weekend more tired than when she started due to some very very late
(early) nights (morning). But it's not anything she can't handle. I need to
go home now as I've been promised something and I don't want to miss out!
[17:20] Definite progress on all scores today I think. Not only with the
documentation but the networking I've been doing. The running was pretty good
too at lunch time. I missed someone's leaving do but by all accounts his
speech was a little odd and I didn't know him that well either. So, ce la vie.
I've got to deal with his application running unattended from Monday too. If
there are any problems I'm going to have to talk to someone else who only has a
vague understanding of the code. Still, best foot forward and all that.
This weekend I'll be gardening on one of the days and shopping for new shorts
and running gear on the other. I really need some new ones and Elaine is
beginning to tell me off for wearing ones with holes in. We'll probably be
online a bit too, talking to people and playing around. All hail broadband!
Depending on the weather this evening we're either going to a barbecue or get
vaguely tipsy at home. Have to see what happens. Have a good weekend.
[16:45] Still writing lots of documentation at the moment. It's not terribly
inspiring but it is essential in the even that someone forgets how to do
something with all the systems I've already set up here. It's actually useful
in that it shows me where I've left holes in the things I've done that no-one
else would think of, but I'd Just Do because it seemed obvious. Documentation
is a fine art and I have to say that that having a wiki to to put it all on in
some kind of semi-organised form is essential in a place this strangely
I took a break from it at lunch time to get my hair cut and head home for a
shower and shave. I feel a lot better with less hair on my head. I don't
think I'll get quite so hot tomorrow when we run again. Speaking of which, our
friend who came on Wednesday is cycling lots on the weekend so he's not coming
tomorrow which will be something of a relief as we may drop the pace a little
and not end up dying when we arrive back.
After I'd got cleaned up I headed into town to pay in some cheques and then
went to look for the place I'm supposed to take the UPS engineer on Monday to
park his car. Once I'd found that I had a quick (or what I thought would be a
quick) trip to make sure my swipe card would work in all the doors it'd need to
on Monday. Of course it didn't so I went to see the person I was supposed to.
Naturally he wasn't in. After about half an hour of waiting I decided to swipe
my card through a reader a few times and informed the receptionist of the time,
my name and what readers the card needed to work for. Hopefully he'll look at
the logs and do the neccessary.
Turns out though that on my return to work that the new project I'm lead
sysadmin for starts in earnest on Monday and I'm not going to be able to be
anywhere but at work. This means I've got to hand off the UPS stuff to someone
else. I don't mind, but the UPSes have kind of been my baby. It's good that
someone else will get something of a handle on them too, in case I'm not
around at some point when things need doing.
It's getting close to home time here again and I'm actually full of energy for
once. It's a great feeling and I hope to take advantage of it.
[17:20] Have been arranging a visit from the UPS engineer to our two main APC
UPSes for next Monday. The car parking around here is insane so it's going to
be kind of a mess getting from one server room to the other. Still, has to be
done. One of the UPSes has been erroring for months (perhaps years) but it's
only since I got my hands on it that the emailing of warning has been getting
to people and that's what's spurred them into action to book an engineer.
We went for a run today with a new running mate. He's far, far fitter than I
and finished the 6.4 mile loop we do easily two or three minutes ahead of me.
It wasn't embarassing as I know he's far better than me and has run at least
two marathons before. I think training with him will help me tremendously and
I look forward to keeping up with him in a few weeks or months of running.
Chatting online today has been fun and Elaine's been involved too. Had a whale
of a time. I have to admit that I'm using MSN, although in my defence I am
using a linux client over an SSH tunnel, so I'm not totally without technical
I'm going to attempt to climb this evening despite having completely worn
myself out at lunch time. I certainly feel a lot fitter than I did two months
ago, but I'm also a lot tireder in the evening. I think it helps me relax a
bit more which is good for Elaine as she's working very hard in her job at the
moment and needs the evenings and weekends to relax and recharge a bit.
Evenings are for talking about the day and saying hi to people.
[17:00] Not much today. More documentation. Managed to get the PC Support
(Windows admin) people to get the updated MGE UPS software on their machines
finally. It's only taken four months or so. Still, never mind.
[17:20] Elaine went out with work colleagues on Friday night so I stayed home,
chatted on line and drank my way through some Southern Comfort in comfort.
Elaine came home late on and we had a very enjoyable time chatting to people
while happily merry. I ended up needing to stay up in the toilet for a little
while whereas Elaine went straight to bed. This somehow meant that in the
morning I was fine and dandy and Elaine had to sleep for a lot longer and still
felt a bit under the weather. We have a lovely day spending it doing nothing
Sunday was better as we spent the morning and early afternoon relaxing even
more before Cormac and Steph came over to take us to their place to help clear
some of the jungle that is their back garden. Armed with shears, a scythe and
an axe, not to mention some very thick gloves we set forth into the tree-like
brambles and managed to clear a way to the garden proper. Within two hours
we'd made some serious progress and decided to stop and have a curry at the
place three doors down. That was wonderful after the heat of the day and the
work we'd been doing. Following that we just headed home and went to bed.
This morning we were woken at about 04:52 by a neighbour across the road who
decided to start up and drive away his diesel 4x4 while our window was open. I
don't think I slept properly again afterwards. Today I've been doing some
useful chatting to people and wondering how Elaine's part in the interview
process went (she was testing the applicants' ADS knowledge). Documentation
has been happening (as it always does at the moment) and I'm making some
progress with Oracle client installations on some of the web developers
servers. I don't think they're going to be finished any time soon. Especially
as I'm trying to find two packages that're allegedly in RHEL 3's RPM list yet
I can't find them anywhere. If anyone knows the whereabouts of mysql-jdbc and
mysql-jdbc-tomcat as referenced in
the release notes for U1 of RHEL 3
I'll owe them a pint or two. Hopefully tonight though I'll forget about my
searching, wailing and gnashing of teeth and Elaine and I can chat about
the day, say hi to some people online and relax some more too.
The run today was good. I got my running partner to make it all the way around
without stopping. All we need to do now is get the pace up a bit. I invited a
few more people out on Wednesday including someone who's probably far fitter
than I. I wonder how that's going to go.
[17:05] All in all it's not been too bad a week. In the last few days I've
made some real progress with something I've been working on with Elaine and am
very hopeful we'll move it further over the next few days. Today though has
been mainly about fighting with the current hairy dog that is the bespoke
backups system. I've been adding five machines to the setup and hating every
minute of it. If I hadn't been following the instructions which I
painstakingly consigned to the twiki many moons ago I'd have shuffled off this
mortal coil and gone to join the choir invisible. The sooner we move to
Veritas' NetBackup the better.
My running partner was off doing Cisco exams this morning so didn't make it
back in time for the lunchtime run. This meant I was free to go at my own
(slightly faster) pace and do the whole 6.5 mile circuit in one go. The heat
is unrelenting on the back part of the loop but I managed it, again. Go me.
I was intending to go to a party this weekend but owing to one thing and
another I can't, so I'll be at home, doing the things I need to do, playing
with my monitor and getting some reading done where possible. I'm looking
forward to not running for two days and not doing weights on Sunday morning,
but other than that this whole exercise thing is doing me a whole lot of good.
My defintion's coming back, my lung capacity is going up again and I feel a lot
better about myself and what I can do. All I have to do now is do the things
I want to do.
Everyone have a great weekend and stuff and I'll see you bright and early on
Monday morning. Well I would if any of you actually looked at the web cam
any more rather than just going to LiveJournal to read this. Come on, I'm not
that ugly. Load it up and notice that it looks at the other side of my face
in this job than in did in the last.
[17:45] Most of the day has been spent trying not to melt. It's hot here,
damned hot, real hot. My shorts you could cook things in them. OK, enough of
the 'Good Morning Vietnam' quotes. Today I've been doing more documentation,
discussing Orcallator and generally working very slowly towards arranging
something of benefit for a few people. It's early days yet but could work out.
Last night was great. Aside from getting home and having a great shower and
cool down with Elaine we watched Mutiplicity (a film I've not seen before) with
Cormac and drove home some chinese food from our current favourite place. Once
that'd finished we watched another Family Guy from season four before shooing
Cormac out of the door so we could get an early night for once. I was still
tired in the morning. Must be the heat. We didn't go climbing, as you may
have noticed. Turns out Cormac wasn't going anyway but the thing he was going
to do got cancelled too.
I don't think going for a Thai red curry for lunch was really the best idea.
It wasn't terribly hot but by the time I'd finished I was certainly a lot
warmer. Like I said, probably not the best idea. As things stand I should
probably go home and play with my monitor a bit more before people turn up. I
took some pictures this morning which're
on the gallery if you're interested.
[16:35] The monitor is absolutely beautiful when in its native resolution and
being fed by a good DVI signal. Elaine was suitably in awe and is probably
thinking of ways to bump me off so she can have it. We took my Iiyama CRT
around to Elaine's friend's house (although Elaine had to go over again later
with a D-SUB cable as he didn't actually have one). He's very pleased with his
purchase but I'm holding off cashing the cheque for a few days in case he has a
problem. I on the other hand have a minor problem in that many of my games
aren't completely happy in widescreen and many don't actually go above 1600 x
1200. I think I can live with it though.
I've been doing more documentation for most of the day. The heat is only
vaguely tolerable here with the humidity really noticable when you leave the
air-conditioned solace of the server room. Despite that I still went for the
normal run at lunch time and came back about half a stone lighter (or so it
seemed). Happily, the ice cream van was in the car park when we arrived back
so I was bought an orange ice lolly type deal which really helped take some of
the heat off.
The rest of the day has been spent watching the absolutely torrential downpour
which enveloped where I worked. From a few thunder and lightning bits and some
fairly heavy rain to something that can only be described as a lake with tiny
holes cut in it. Seriously I couldn't really see out of the window. That
lasted for about ten minutes before vanishing as quickly as it arrived. Looks
like I'll be able to get home without putting on waterproofs. Elaine doesn't
have any with her today anyway so it's good things are dry again. We're off
climbing this evening. Hopefully the rain will have taken some of the heat
out of the day before we get there.
[12:30] My new monitor has arrived! It is spankygorgeous.
Unfortunately I don't have a machine with a decent enough graphics card (DVI
out and with a CD-ROM to take the driver files to go up to 1920 x 1200) to try
it out properly. I have powered it up and run it at 1280 x 1024 on D-SUB and
verified that there is not one stuck pixel (as yet) so I'm very pleased. Of
course I have to wait until this evening to see it in all its glory. I'll be
dropping off the CRT this evening too with generous help from Cormac and his
[17:30] I've been writing documentation all afternoon after having come back
from lunch. Turns out that the UPS engineer who we were going to get to come
out didn't have any good contact names so had to go away again. Shame, we'll
have to have him back to show us what went wrong with our management card.
It's a bit annoying at the moment because we don't dare plug it into the net
again in case it goes wild and tries to shut everything down again.
The documentation on the other hand has been going according to plan. I've
printed out a first draft and left it on my line manager's desk for him to poke
holes in. Other than that I'm going to wait for my (monitor's) ride home and
[12/07/2005 09:30] Sorry, should have written this on the day but I was
hellishly busy. For most of the day I was working on the finishing touches to
my DR documentation and some other important stuff. Over lunch we went running
in the really rather hot weather and managed a reasonable time without melting
too much. However the real fun came at 16:08 when one of the linux mail
servers went absolutely batshit crazy and began spouting a whole load of EXT3
errors. It looks like something went briefly wrong with the actual RAID
controller but there's very little to go on. As it was I cycled over there as
soon as I could but found that my swipe card wasn't registered to work in the
doors there yet. I had to ring and wait for someone else to come along before
I could get access to the machine. Happily (worryingly) it came up perfectly
and didn't put a foot wrong while I was there. I'll do some more
The weekend was interesting. Getting dressed up for the wedding was a minor
inconvenience for seeing all my friends together in the same place. It was
very odd to see groups of friends I'd never expect to see together, together.
Beside that a great time was had and I took some reasonably good photos which
you can see in the relevant collection
for your delectation and delight. Of course, there are plenty of pictures of
Elaine in there too.
[17:25] Hurrah. A productive day. Not only did I do the whole 6.5 mile loop
without a pause (and wasn't dead by the end of it) but I've made real progress
on my disaster recovery document too, including picking apart the various
places databases and standby databases have ended up inside our setup. I've
got scripts in place that're almost finished, the rsync knows about them and
all I need to do is finished the scripts, repackage the scripts directory
tarball for potential future reinstallations and I should be done. I also
have to start collecting Dell service tag ids so we can work out which machines
are worth getting extended support on. Actually I have a whole pile of
documentation to write which is going to take an age. The DR one is the most
I think I mentioned that it's my ex's wedding tomorrow which I'm attending.
It's going to be great to see people I haven't seen in a long time in some
cases. I'll probably take some photos here and there. For now though, it's
late on a Friday night (work-wise) so I'm off home.
[17:10] I won't even bother with today's London news, you all know about it
from reputable sources. I on the other hand have been spending time helping
other sysadmins who use our server room for co-location to rack a few servers,
take away our old and spare D1000s and generally tidy up the office a bit. We
had an Institution-wide Solaris 10 workshop thingy which turned out to be a
reasonable lunch and then a disorganised question and answer session which
turned up some useful things but wasn't overly great as the use of a lunch
time. Especially so as I didn't get to go running. Still, probably a good
idea for the left achillies tendon, etc.
[17:10] Argh! Why do things have to be so complicated when it comes to,
y'know, Stuff. If it wasn't bad enough that I appear to have forgotten how to
write useful documentation, I'm beginning to hate the tnsnames.ora file and
just how many database servers/instances we have to juggle. Sometimes I'm very
glad I'm not a DBA and just a sensible sysadmin with other fish to fry. Of
course it pays to know a little about both.
Today has been odd as I didn't run. My left achillies tendon has been giving
me a little gyp recently so I think I need to perhaps not run every day for a
little while until it feels a bit better. Either way I'm climbing this evening.
That coupled with the exercises this morning which I hope to start doing again
every morning should help me get back to where I was two or so years ago
fitness wise. Work achieved today: minimal, chat had: lots.
[17:30] So I get in this morning and there's been some real problems with the
UPS and the SAN. Turns out that around midday yesterday the network card in
one of the UPSes (the one where I work most of the time) decided that there was
only one minute of battery time left and therefore all the clients would need
to be shut down. This they duly started to do, all twenty five of them. Why
the Windows boxes didn't I don't know, but that's something else to think
about. Anyway, the people who were here were able to stop the key most
essential boxes from shutting down, which was a good thing. Unfortunately
twenty other boxes powered down (cleanly). It took a fair while to sort
everything out, but there was no data lost so that's OK. I was called to find
out the UPS username and password but in the end they just yanked the network
cable to stop it doing anything else. There's nothing wrong with the UPS or
the battery so far as we can see. Very odd.
The SAN just threw another disk. Standard Seagate predicted spindle failure
(again). The Sun engineer is here again replacing that as well as connecting
up another complete array to do copying over to get us off all the Seagates
and incidently upgrade us to faster spindles anyway.
I've been editing tnsnames.ora files and working on the depths of the disaster
rcovery document, which is turning out to be a little more complicated because
of the use of Delphi applications with hardcoded database references. I'm sure
it's all solvable though. With time. For now I head off in the rain to meet
my slightly damp Elaine (who didn't take a waterproof to work today) before
heading out for food and cinema (Batman Begins).
[05/07/2005 - 15:55] Took the day off. Having spent the weekend either
relaxing or at a party with lots of fun people it was nice to be able to sleep
in on a Monday morning. Once up and about though we headed into town to try
and get some clothes together for this wedding thing next weekend. Once we'd
got Elaine something she cycled home to make sure it worked with the shoes she
wants to wear. I went trouser shopping. Of course I didn't buy anything until
Elaine was back. A second pair of eyes really does help sometimes. Of course
once we'd got my last bit of apparel Elaine needed shoes and accessories so I
went to have a sit down in Borders with a book (Nova, Samuel Delany) for a
while until she found me and we headed home.
During the day I got a call from work. Seems without me there things went to
hell very quickly. Not through my technical know-how but because when I'm
around things break less. More on this tomorrow.
[18:05] Again I'm very tired today. First thing this morning I and a couple of
others transported an old but still good Sun E3500 and associated A1000 to one
of the other machine rooms and got it up and running again. Six CPUs, two or
four gigs of RAM. Better than the machine it's replacing anyway. As a bonus
it sits the console server's console on it far better than the E250 it's
replacing. It's all good.
I didn't go running today because my left achillies tendon has been giving me a
bit of gyp. I wasn't too put out as my running partner cried off before me.
It's going to be odd next week running alone as he's on holiday. I'll have to
see how it goes. Maybe I'll get the whole route done in one go without a
pause. It's a shame I didn't go really because I'd put my shorts around the
back of the SAN cabinet to dry them out especially. They weren't more than
damp (no danger of them dripping on anything) but it was good to get
them properly dry. I'm sure it's not a proper use for a machine room, but it
won't have done any harm considering the size of the place.
This afternoon I buckled down and got stuck into the disaster recovery document
I'm supposed to be writing. It's pretty simple; three cases, five applications
to deal with. I just have to ensure that the more complex cases (those that
use tns_names.ora rather than configuration files) are dealt with. For the
moment though (and for Monday, as I'm holiday) I will head off and wish you a
happy and relaxing weekend. I'm off to a barbecue this Saturday and probably
clothes shopping for the wedding on Monday.