[13:20] This morning I got in nice and early to clear down the queue on the
mail server I was going to move. So, at atound 08:00 I was in and looking
through it for spam, bad emails and the like. Happily the machine was down
well before the planned time. The problem arose when we de-racked the machine
and found that it wasn't actually on rails at all and had been resting on the
machine below it. No-one know where the racking kit for it was. I put in an
order as quickly as I could but it wasn't going to help the move right then.
Happily once we'd got down to the other site we found a shelf which we mounted
in the right place and got the thing in place, connected and fired up. While
we were about we got me another card with a swipe strip on it so that I can
get into the other server rooms on the other sites without having to have
someone else with me. This necessitated me getting a new card, which also
meant I got to have an updated photo on it. I wish I'd shaved this morning.
I'm back at work now, but my line manager isn't in today so my meeting about
the hardest of the projects isn't going to take place after lunch. I guess
this means me and a colleague can get an rsync-to-backup-server thing in place
for the CVS server in place and running now that that's live too.
My new graphics card hasn't arrived yet. I'm hopeful it will arrive before
the end of the day. I spent the weekend getting my machine ready by shuffling
data to disks which won't be touched by the reinstall. With Elaine away in
Scotland visiting her father I've had the time (too much) to myself to get on
with consolidating my data and preparing things. For some odd reason my
machine in its current incarnation has developed an intermittent cross-browser
with displaying JPEGs. I'm hoping it'll be fixed by changing graphics card
and operating system.
[15:10] It's been a bitty day. I didn't have much to do this morning so I
solved the reason why my Dell servers sometimes hang when booting to serial
console. Turns out that if you have serial "redirection after boot" on in the
BIOS, grub and the BIOS fight it out until one of them suffers some kind of
timeout. Sometimes this results in a grub lockup, other times eventually you
get a grub console boot screen. I've turned it off on two of the four
machines affected and should be able to do the other two machines when they're
moved to their final resting places on the next two Mondays.
Five more servers have arrived for me to set up. I'm not sure what they're
going to be used for yet other than the web services guys are using them.
Setup is going to be a bit more complicated this time around as they're going
to be both live and development boxes. I like how things have been going here
recently; each project is slightly more complex than the one proceeding it.
At this rate I may be able to tackle The Big One without actually doing
something utterly catastrophic.
Elaine's off to her Dad's place in Scotland early tomorrow morning. I was
hoping to spend the weekend playing with my new GeForce 6600GT which allegedly
shipped yesterday afternoon... only Parcelforce don't have any record of it
being with them yet, so I don't think it's going to arrive today. Most likely
Monday then, which is OK as Elaine's not back until late on Tuesday.
I think I'll go and unbox those servers and see which are which. Some have
different amounts of RAM/CPUs/HDDs and I have no idea which is which.
[16:05] Grammar aside, I know now and have labeled the boxes accordingly. Of
course I can't do anything to them until I know what the developers want doing
with them. Frankly I'd like to go home considering how tired I feel at the
moment. I may actually leave at 16:30, just like the normals do.
[11:45] Elaine and I decided not to go climbing last night. We were both
absolutely knackered. I think it's the cold more than anything, and this snow
which falls but doesn't settle. I don't know what the road conditions are
going to be like half the time. Sometimes I think it's safer to run to work.
We chatted about stuff, sorted things out and generally caught up on the last
few weeks. It's been harder recently, with everything going on, to spend
proper time with each other just being comfortable and relaxed. I know I
haven't been quite doing things right and it's something I'm going to be
taking steps to address. Among the things I need to do is to actually think
less. Odd as this may sound this would be a major achievement for me and do
a lot of good. Seriously, don't ask.
Otherwise work is going well. I've implemented a
CVSNT CVS repository on a spare box here
and asked my sample developer to have a go with WinCVS as the client. It's
really rather good and everything looks to be stunningly compatible. There's
a huge plus to CVSNT which is that you can set repository compatibility such
that one may behave like a CVS 1.11.x server and another like a
bells-and-whistles CVSNT server supporting the latest and greatest switches,
flags and malarky. I'm modifying my build script to remove CVS (and RCS
apparently) and install CVSNT and associated scripts, configurations and
whatnot. Pending the meeting this afternoon I may rebuild the actual CVS box
to the new specifications.
[12:15] I hate to admit it but I couldn't stay away. I realised that if I
didn't get the BrightSTOR ARCServe Backup patch installed on the two servers
running that software at my old place pronto, there was probably going to be a
lot more for me to do (as the person with the most knowledge of the setup) if
it wasn't done and the machines were hacked. As a result (and owing to the
fact that I ran in in the snow this morning and didn't have a bike or a jacket)
I got a lift from one of the consultants, who has a Porsche Carrera 2, across
town to get it done. It didn't take long. I'm pretty sure the machines hadn't
been hacked, but there were some Discovery Service errors in each machine's
logs. Here's hoping. I've done all I can considering I don't even work there
any more. I really don't want to have to start going back there to help the
new guy get his head around things. For the moment though I'll contact
Networks and make sure that the place isn't serving warez and porn to the
world at large.
At lunch time today we have a semi-obligatory 'brown bag' seminar. This is
where someone from the department gives (what will actually be) an interesting
seminar on what their group actually does. I think this is a pretty good idea
as most of the time people here don't seem to have a great idea of what other
people are doing. The brown bag refers to the fact that we get some kind of
free lunch (TANSTAAFL) as it's over the lunch break.
I really need some new headphones. These in-ear buds just aren't up to the
[15:35] One of what I thought was one of the core development servers went
down at lunch time. It's the other half of a live service that's one of my
next projects here. It frightens the hell out of me as it's what can best be
described as a maze of twisty little chroots, all alike. I currently have no
idea how to adminster it, let alone recreate it in a new way on the transition
server I have to play with. I've requested some time with my line manager to
work out what exactly we know how to do and to read over what documentation
exists but it's a little complicated at present. I think it's going to need to
be simplified quite a bit before I go ahead and create the replacement system.
Right at the moment I'm tired, my back hurts and I want to go home and hug
Elaine for a while. Hopefully things will seem a bit more sensible tomorrow.
[15:35] Cormac and Steph wre still with us with their house due to be replumbed
for heating and stuff beginning next Monday. Most of Saturday was spent
lying around and generally recovering from the week. Well, that and because
Elaine and Cormac stayed up until 03:00 playing Burnout 3 on my Playstation 2.
We toddled off to have a play with the kites in the afternoon but found it
just a bit too windy so went for a walk in the nearby town instead and had a
look at some architecture. Cormac and Steph went out for a meal while Elaine
and I stayed home and ate some of the stuff we had left over in the house.
Sunday morning was all about the biggest fryup in quite a while. Once that
was over we went out to do some shopping. We managed to get a double Cat-5e
socket for the final bedroom as well as a bread bin, duvet and some network
tools from places all over town and still get back in time for the rugby at
15:00... which wasn't on because the BBC's site had it wrong and it's next
Sunday. Everyone sat around eating cookies while I did some patching before
realising that one of the sockets I had was for phone, rather than data. One
wuick trip later and everything was sorted. We spent the rest of the evening
watching more Scrubs episodes before going to bed under the new duvet we'd
Problem is that the duvet's material is very 'loud' when you rustle it (which
you can't help). As a result we've taken it back today and will be getting a
non-down one another time. Annoying, but sleeping's pretty tough when there's
a racket in your earhole.
Today I made sure that mail was working properly on one server and then downed
the other one and rebuilt it. I had the same problem with the NIC/switch not
autonegotiating quickly enough so dumped a sleep into the post-install build
script before it got copied to the machine. Everything was fine on buld then.
After a quick up2date and a reboot it was soon re-relaying mail again as if
nothing had happened. I love it when a plan comes together.
I took a trip into town at lunch time with the duvet, met Elaine and got it
returned before heading to my old workplace to make sure everything was fine
for my replacement's start date of the 28th. Now I'm back at work I'll be
doing some work on documentation for the remainder of the day and then heading
[13:55] I was sure it was later in the day than this when I decided I had time
to write today's entry. Just goes to show that some days have less going on
in them than others. Yesterday it turned out that Cormac's house needs a bit
of work done to it with regard to heating and stuff and it can't be done for
about a week. Consequently he and Steph are bunking at our place until such
time as it doesn't feel like the inside of a tomb at Cormac's house. It
should be a whole lot of fun.
Elaine went to the dentist this morning (which necessitated her getting up
early so that she got some time at work before the appointment, and hence I
got up early too). Turns out she's got an impacted wisdom tooth that won't
come out without a hospital appointment. Given that it doesn't seem to be
bothering her overmuch at the moment that's going to wait for a convenient
I've been bringing my documentation up to date this morning. I'm determined
to try and do it in parallel with whatever work I do here, updating as
required. Theoretically this should mean that there'll never be a lot to do
at one time and in the event of loss I should be able to reconstruct any lost
configuration files from the contents of the wiki.
Finally fed up of VLC not working for me I've installed mplayer, which seems
to work a whole lot better out of the box. Now all I need to do is work out
some way of getting it to work with Apple's movie trailers site. Other than
that I'll be doing a rebuild of the CVS server this afternoon to see if I can
bring it back to its current status as a working repository machine with
pserver (yes, I know it's insecure) enabled. I'm also in the market for a
new graphics card. Given my budget I think it's going to have to be a
[12:30] A colleague from the DBA group and I have been working hard on a CVS
setup that'll work with my CVS server. He, and others are going to have to use
WinCVS (a Win32 client) and it's not entirely flag-compatible with CVS 1.11 on
my linux box. This is a tad annoying.
In other news I think I may have found a solution to the networking issues with
my new server which I was having problems with yesterday. I've yet to try a
new rebuild as yet, I'll report back on that later once I've done one.
[13:30] One of the developers here has had a small belated birthday thingy and
brought in champagne, cake and orange juice. A Bucks Fizz in the middle of the
day is probably a bad idea. Especially before I have my lunch.
[17/02/2005 - 12:05] The guys from PC support came in this morning with the
PowerEdge 2650 that I wanted to swap my 2850 for. They were in a bit of a
rush, which suited me down to the ground. Not doing anything else right then
I got it racked up (heavy beast) and set up the initial RAID 1 mirror. For
some reason I couldn't set the hot spare from within the Adaptec BIOS, which
was a bit annoying. I also updated the motherboard and RAID BIOSes as well
for good measure.
Kickstart wasn't so good. For some unknown reason it wasn't anywhere near as
hands-off as the other machine had been. I had to intervene a few times.
The rest of the build proceeded OK until first reboot, at which point the
build script failed on the first few network transfers (a links --source and
some rpm installations via http). I had no real idea what was wrong but put it
down to either bad VLAN/firewall configurations or the fact that the machine
was plugged into a gigabit port on a Cisco 3750 and the other machine (which
built perfectly) was on a 100Mb port on another model (of Cisco). I suspected
extremely slow autonegotiation after the NIC came up on the machine. While I
had the machine installed I set up the hot spare using the afacli tool (now
successfully (manually) rpm'ed in). A second build on the same box failed in
a similar but slightly different manner. By the time I was in a position to
talk to Networks they'd gone home, so I went to. Climbing was extremely good
with some real advances made on some tricky routes. We were all famished by
the end so had some really greasy food and then went to bed.
[18:40] OK, today has been one of those days where you achieve a lot, things
go well and then at the end you make a fool of yourself on two international
mailing lists. One of those days. You know.
Anyway, this morning I got a good handle on how to get pserver working for CVS
to such a degree that I think I'm ready for the tests tomorrow with one of the
developers. This is a good thing as another developer emailed me today to ask
for access to it. It hasn't even been built, let alone publicised yet.
My new motherboard arrived for my home PC. I've put it to one side to take
home this evening. I burnt final ISOs for RedHat Enterprise Linux 4 (at a
blistering x8 speed. I swear there's something wrong with my CD writer) and
started an install. Looked fairly similar to RHEL3 to be honest.
On first boot the differences were obvious but I could still get it connected
to RedHat Networks and get it up to date (already way behind). The 2.6.x
kernel shows its advances over the 2.4.x kernel straight away. Unfortunately
this is where I started to show my lack of knowledge as I tried to recompile
it to get extra modules (specifically v4l) installed. After some posts to the
Taroon and Nahant lists someone mentioned that I might like to RTFM. A couple
of minutes later I was eating humble pie and passing on my perls of garnered
wisdom to those people on the list who'd asked me what I'd learnt. I think now
it's time to go home.
[16:40] Today has been all about getting things done. This morning the guy
who was set to do the garage roof texted me to say he could come today. This
meant that I got to hang around at home until he turned up (on time). Once he
had power from an extension in the garage I cycled to work. While at work I
have been able to reinstall someone's machine with RHEL3 u4 (fully patched)
rather than RH9 (unpatched), give the user Firefox and also navigate through
the twisty maze of RPM dependencies to get Workrave installed for her. This
is such a good thing that I'm going to put an article on it in the work wiki
so that others may follow in my footsteps.
I'm that nice.
For most of the afternoon I've been burning RHEL4 beta CDs to try out tomorrow
as well as getting firewall rules and host lists updated such that the final
build of my new mail servers. We now have one machine that has a route
through all the firewalls for SMTP in from everyone and out to everyone, and
one which only allows SMTP in from the cluster we're beginning to hub through
for virus and spam filtering and SMTP out to everyone. I've asked for the MX
records to be updated to point to the cluster which is hubbing for us and, as
soon as that change propagates everywhere, we'll be able to shut off SMTP to
the other machine and tighten up security a bit.
Once all of that is done I can shut down the listener on the older
installation, clear the queue and bring it down for about forty minutes to do
a fresh build. I must remember to upgrade the firmware too. Oh, and uncomment
the line in Big Brother's monitoring configuration.
For now though I'm going to go home and look at my new garage roof.
[08:50] Looks like my motherboard is toast. I'm in the market for a new one
now. Hopefully supporting 333MHZ FSB. I'm after an ASUS A7V8X
(not the A7V8X-X) but can't seem to find one at the moment. I came in to work
early in case I had to leave to be at home for the people coming to do the
garage roof, but they're not coming today owing to it raining yesterday.
[12:30] I've gone and ordered a ASUS A7V880 from eBuyer. Hopefully it'll
arrive next Wednesday.
[16:45] I've managed to get the DVI card working under linux now. If you want
details then email me. Suffice it to say it's not hard. However now I have
DVI working the quality appears to be worse than under analogue (which is
pretty good), so I've remained on analogue although now I'm on the right
driver which also allows 3D hardware acceleration. I think my work is done
here for today, so I'm off home to paint a radiator and get the computer room
back into a state where I can work on a PC without breaking bits off it.
[11/02/2005 - 08:20] Again the entry is a day late. Never mind. Most of the
morning was spent fine-tuning the build script to work with the final bits of
the puzzle. I'm pretty sure it all works as expected now.
Our flatscreens arrived in the morning and, excited, I powered down my machine
and installed the DVI add-in card. The machine came up, the BIOS displayed,
grub displayed (running RedHat Enterprise 3 u4) and I watched the kernel boot
messages scroll by. However, as soon as X started the screen went blank.
There were no error messages from the screen to say something was out of range.
I removed the DVI card and booted with analogue input and everything worked
perfectly. This is annoying. Currently I'm running on analogue rather than
DVI input and hoping someone can tell me what the solution is.
During this little episode I had to take time out to go to my old workplace and
show the replacement guy how everything worked. I think he was a little
overwhelmed by everything I had in place. I'm fairly sure he'll get a handle
on things eventually. After two hours (an hour over the length of time I'd
wanted to be there) I made my way over to another of my current job's sites to
help a coworker unrack and rerack a large Sun. After that it was back to
work, all in the pouring rain.
As a result of being behind on things I was late home but eager to quickly get
my new Athlon 3000+ XP installed in my A7V333 motherboard. I had a look at
the manual and the motherboard supports 333MHz RAM (which) I have and a
100/133 FSB. The CPU I've bought says it has a 333 FSB. I now don't know if
this is compatible or not.
Of course this is all academic as in the process of taking off the old heatsink
I think I managed to chip off two tiny, tiny little surface-mounted components
next to the CPU socket, so now the motherboard can't find any RAM. I left it
alone and went to spend time with Elaine and guests.
[15:40] I'm currently in the middle of a kickstart rebuild on my server at the
moment. This could be the final interation (scripts willing). I've added in a
whole load of new MySQL functionality, Big Brother monitoring, ssh keys and
authorized_keys files, updated exim configurations (kindly provided by the Mail
Group) and a whole host of other little things.
Now I just have to see if it builds. If I don't write anything else today we're
off to a curry tonight, after climbing.
[10:25] Turns out I can now boot a PowerEdge 2850 completely from serial console
now, including from kickstart. Which is nice. What's also nice is that
apparently my LCD monitor has arrived so I can get rid of this piddling little
17" goldfish bowl.
What isn't so good is that while I'm also rejoicing that my final server (for
the moment) has come in, it's another 2850, rather than the 2650 I was hoping
for. I know that Dell don't make the 2650 any more (damn them), but the other
two servers in my cluster are 2650s and I'd dearly love to use the same
configuration file for all three. Now, there used to be a third server which
was a 2650, but we gave it to another group before I got here because the
project was taking so long to get off the ground. I have to plead with that
group to take my brand, spanking new 2850 and swap it for their (my) 2650 which
has been running Windows for I don't know how long.
[17:40] It's been a bit busy again today. I've been working on getting my other
servers switched over to serial rather than VGA/keyboard/mouse. This has
necessitated a good bit of fettling in the machine room to get things in place.
It turns out that while I've been able to get on a bit further with my build
scripts I won't be able to do any more until the people I'm doing it for come
up with their complete list. In the meantime I can get Big Brother on and
working with help from my colleague and keep on at the people who have the 2650
I want. I dunno, maybe if I polish it and put a bow on it that might help.
Other than that I've been slogging through paperwork for buying more RedHat
[12:35] Of course when I came in this morning I solved the partitioning thing
in about thirty seconds (the docs don't match the implementation). I now have a
happily booting/installing/post-install configuring PowerEdge2850 running on
the latest megaraid2 driver for RHEL3u4 and currently everything seems fine.
The weekend was pretty good. We had a great Saturday and worked out a whole
load of things we need to and can do to the house. At some point we're going
to lose the box/computer room and make the landing really spacious. This will
necessitate a lot of building work, but it almost all needs doing anyway. That
will give me the excuse to recarpet everywhere upstairs. In the meantime Elaine
went out and got some point and we started painting the room. And what a
difference it made.
Sunday hit Elaine with a headache so she dealt with it and made some bread
while I finished painting the coving and walls in the computer room. I also
did some research on the whole partition thing and discovered why things weren't
working. I can't remember the last time I was eager for the weekend to be over
so I could get to work ang get things sorted.
This morning (after solving the partitioning problem) I started off in a BigIP
presentation meeting but left at the halfway point as I felt I could be more
useful elsewhere. I've already prodded the people I'm doing the first project
kickstart builds for to see if they've got a complete list of the things they
want added/changed and for them to talk to Central about mailhubbing. With
luck they'll get their arses in gear and we can get this put to bed.
I need to learn more about CVS administration this afternoon.
[07/02/2005 - 12:25] I really should get this done more on the day, rather than
the next day (or after the weekend). Friday was all about getting the new
PowerEdge 2850 with a PERC 4e/Di controller to boot from kickstart. It worked
fine from CD, but when I started doing it from floppy to a HTTP-based install
the installer couldn't find the RAID controller, and hence the disks. After
some pointless hacking about with options to initrd to do with "expert noprobe"
and similar I settled for adding one line to the 'pcitable' file on the driver
floppy to list the RAID controller. This proved to be the best, easiest and
simplest solution. If you want to know what I did, email me.
Of course then I couldn't get the install to run because it seemed unable to
remove the old partitions. I left work wondering what was up with 'clearpart'.
[16:00] What a day. I thought I'd take a look at eximon on the server I
recently handed over to the development team. Colour me extremely surprised
when the machine appeared to have mail queuing on it. To cut a long and rather
involved story short, some things had happened more quickly than people had
thought and certain changes had been made. Suffice it to say that we got the
listener shut down and opened up a hole in the right firewall to let the mail
back out again. That was fun.
I've got some extra stuff to do to the build (mainly getting mail hubbed
incoming from elsewhere which will involve an MX record change at some point)
but I'm basically waiting on other people for the moment. This means that I
can go back to my other project of getting the new CVS server up and running.
I should have had a call from RedHat Network today about purchasing, but that
doesn't seem to have happened yet. I think I'll call them after finishing this
Other than that I've worked out a bit more about how things work here, probably
annoyed a few people and confused a few others. I'm extremely tired now and
have only just had lunch. I'm looking forward to going home at 17:00 and having
hot chocolate with Elaine.
[03/02/2005 - 10:40] The days are just so busy at the moment. A far better
morning than yesterday. I spent most of the time getting the documentation of
my build process down on the wiki we have here for future generation to gaze at
in awe and wonder before racking up the new Dell 2850 we have and trying to get
its DRAC card set up. Once installed (I love Dell rack kits) I set the RAID to
initialising the two RAID 1 mirrors the box is going to have and went back to
working on a new version of exim for the build I've just done. Documenting as
I went I now have a very simple way to update exim and incorporate it into new
builds of the servers. It'll also be simplicity itself to drop it onto existing
boxes (built with my build script) to replace the current older version.
It then being late in the day I went home as the road to the climbing wall was
blocked by a few large accidents.
[09:20] I'm tired and stuff today. Sometimes it all just gets on top of you a
bit and you wonder just how worth it is doing what you're doing. Currently I
don't know whether I'm going to go climbing in London tonight or go home. I
may try and meet Elaine for lunch to say hi and stuff, try and shake this
I've got the second half of the perl course today, just like last week. I had
real trouble staying awake through yesterday's, however there's a chance I may
do better today, again, just like last week. I just wish I wasn't so down at
[19:00] Well, the day brightened considerably after the sad start. First of all
I had another useful set of conversations with those nice people at RedHat
Network Customer Support. I think I know why everything is as it is, including
why we seem to be down a couple of channel subscriptions, and why we have
Management entitlements in the way that we do.
In need of some company from the delicious Elaine I'd already send off an email
asking to meet her for lunch and so, at around 12:45 we met by the pond in
town and had a wonderful time feeding the ducks and talking about the last few
months. Once we'd had a good walk around I headed off to the last of my perl
seminars and Elaine went back to work. In the break I popped off to see Bob
and retrieved my Firewire drive and borrowed his USB2 drive to get my MP3s to
work (no Firewire at work). Once the course was over I headed home rather than
going climbing to keep Elaine company for the evening. We're having something
nice to eat and I'd rather have it freshly cooked than reheated when I got back