[01/02/2005 - 09:00] Spent the entire morning working on documenting the build
process I've been doing since I started here. Seems to be a fair amount of work
considering how out of my depth I feel sometimes.
The afternoon was spent at the first of the two sessions of more advanced perl
teaching. The second one's tomorrow. I introduced Elaine to A.C.C.'s 2010
which was showing on TV when she got back from shopping with Cormac and I
finally got home after going back to work after the course. Following that we
watched the first few episodes of Scrubs' series two.
[09:55] I have my meeting in five minutes. It's the first one I've called and
I'm a bit nervous in case I'm asked to lead it and/or it turns out there's tons
to do and I don't know how to. Of course, this project's been stalled for a
good few months so there's no real rush, but still...
I kickstarted the one spare target machine I have with the right specifications
as soon as I got in this morning. The procedure went perfectly, which was a
big weight off my mind. Now all I have to do is add in whatever else is
required and help with the data migration and the backup procedures.
[17:30] The meeting went really well. I didn't really lead it, which was fine,
and I had the opportunity to ask all the questions I needed and get all the
answers that were required. I've spent the rest of the day tweaking the install
scripts to deal with the different exim configurations (learning a bit more
about sed in the process) as well as making sure that the kickstart IPs have
been updated to deal with the fact that this place is VLANed to hell and back.
Of course, some of the really funky stuff broke when I reimaged a machine on the
new VLAN, but some fettling with sendmail and trying some manual SMTP to the
organisation's mail switch/hub seemed to work in the end and now things are
back on track. I really do like centralised scripting, it makes things a hell
of a lot easier when you're setting up multiple machines. As it's Friday I
think it's time to go home now. With luck Elaine and I will be going to Reading
this weekend to help Dunk with some interior decorating ideas. I know nothing
about this kind of thing, but Elaine goes nuts for it.
If I don't come in on Monday it's because Elaine killed me for that comment.
[28/01/2005 - 09:55] A rather successful day all told. Completely rewrote the
kickstart stuff to take advantage of the Jumpstart method of adding a script to
the rc chain and running it once on boot then having it delete itself. I even
managed to work with RedHat Network's activation keys system to subscribe the
target machines to a subchannel. It's all gone rather swimmingly thus far and
all I need to do now is have the meeting to talk about what else the
developers want from the build before they get it.
Apologies for these entries being very work-related at the moment, but it's
exciting for me to be doing something challenging at work finally.
[17:20] I'm rapidly going mad listening to the same tunes over and over again,
but the SSH server I need to be up to get to my MP3s is still down. With luck
I'll be able to get it working tomorrow.
I've been able to get some of the last few wrinkles out of the way with my
kickstarts to a point where I've arranged a meeting with the developers who need
the service the machines will provide on Friday morning. I expect that the
remainder of Friday will be spent changing and fiddling with my setup to
conform to whatever extra or different things they require. I'm not too fussed
though (at present) as this is What My Job Is.
Other than that my money for the website I was doing at the tail end of last
year has come through (not a moment too soon as I was down to very little owing
to one thing and another recently). It's enabled me to make some purchases
without going into my overdraft (something I hate doing). Off climbing this
[13:20] Slowly but surely I'm getting to a point where the Kickstart stuff is
complete, fully-featured and gives me exactly the type of machine installation
that won't need any tweaking from me once installed. The storing of certain
config scripts on the KS server means I get a whole load of extra configurabilty
for no extra work.
The camera went down this morning, not sure why. If it doesn't look like it's
updating you're more than welcome to email and tell me. It's a bit more
supportable these days since I moved from RH5.2 to RHEL3.
I'm off to the second part of the perl course this afternoon. In fact, I'll
be heading off in about twenty minutes or so.
[25/01/2005 - 08:40] Busy day, again. I may just start leaving that bit out,
because it's almost always true. The morning was spent working out why the
kickstart disk I had didn't boot the machine I had. This turned out to be
because the machine I had was my old Windows desktop box which is now surplus
to requirements on account of the Windows 2003 Terminal Services prototype we
have here and the wonders of Rdesktop. Anyway, having been a Windows box it
was on a completely different VLAN and behind who knows how many firewalls
compared to the linux workstation I now sit at. The IPs I had to play with
didn't work on that machine so I simply created a kickstart .cfg file for its
IP and everything was 'fine'.
I say 'fine' because I still need to do a lot of work on the actual kickstart
itself. However, now I have a first approximation of something that works I
can go about refining the install as well as having a meeting with the
developers this box is destined for to find out what they want on top of the
This afternoon is Part 1 of a Perl course, so I won't be here.
[17:10] Every day is just so busy here. It's great.
The rest of the day, after a pleasant lunch with Shaun and Cormac, was spent
updating the post-install section of the kickstart setup I'm working on. I've
got everything except the NTP stuff sorted (I think) now. Still to do on Monday
is an updated kickstart installation point (RHEL 3 AS update 4), a few more
little things and then I try it on the test box I have. That'll be the moment
of truth. As for now, I'm off home then to Ian's birthday curry. This weekend
should be all about more relaxation, potentially some serious reading of books
for pleasure and some late nights and even later mornings in bed.
I'm really enjoying work at the moment, although I can't help thinking that I'm
going to be confronted by some extremely difficult tasks fairly shortly. Thus
far things seem to be going my way. Apart from some freaky screenshot keyboard
combination I seem to be hitting in Gnome, the arrival of my PS2 keyboard and
mouse to USB dongle has allowed me to start using my natural keyboard again and
my wrists have stopped complaining. However, I've done a good day's work today
and it's time to be off.
[17:10] So today I embarked on a trip through RedHat RPM dependencies hell.
Luckily it seems that it's improved since I last did anything serious with it.
This was in an effort to get Dell's more recent AFA management tools installed
on the test build of the three-server setup I'm working on. Although I
eventually worked my way through the twisty maze of packages, all alike, I still
had to update the RAID card's firmware (and hence the BIOS too) before I could
run the applications in case something 'unsupported' happened. This turned out
to be the easiest part of the day. Three floppies written, a few minutes of
running executables from DOS and we were happy. I've kept the disks
(apparently gold-dust around here) for the other two machines. Of course, as
one of them was given to another project we're getting a new one to replace it.
That'll probably have up to date BIOS and other firmware already.
With the help of some helpful people on the web I also have some scripts which
should check the status of the array and even allow me to assign the hot spare
without any kind of action on my part. I'm still swaying between RAID1 and 5
for the machines.
I was treated to lunch by someone from another department who was interested in
my Windows/linux integration skills as he's been given a vaguely blank slate
to create a resaonable system where he works. Over food I helped answer some of
his more Windows-related questions and we sketched out some rough ideas of where
to go with Samba, OpenSSH and the like.
More RHEL kernel updates got pushed out this morning so I did a few more kernel
compilations (ostensibly for modules only, but the practice is good for getting
the sequence of events straight in my mind) and pushed out the updates to the
growing list of machines I have some control over.
Given my requirement for burning .iso images to CD for installation purposes I
spent a semi rage-inducing half an hour or so installing k3b (and associated
cdrdao stuff) and getting annoyed that the setup program (once installed)
wouldn't run unless I was actually running KDE as the window manager.
Even though I don't think I've been pushing myself particularly hard as yet, I
feel absolutely knackered at the moment although this only seems to manifest
itself as yawning rather than droopy eyes etc., except when I get home and
crash. Speaking of which, I think it's time to do so.
[15:55] Far too impatient to wait for the order I put in for a USB expansion
backplane thingy for my PS2 natural keyboard, and because my wrist is already
starting to play up a smidgen I splashed out £10 on one from an online
store and hope it arrives before the end of the week. It'll probably beat Dell
and their delivery time.
I've done another two kernel compilations (for the purpose of module
installations today owing to Redhat's release of a new kernel for Enterprise
Linux. I've also put a fair bit of research into communicating with Dell's
onboard RAID controllers. Of course, being part of RHN now any extra RPMs I
need I can get from there, but it takes a fair while to have them installed. I
can wait until tomorrow I think. Then it's all hands to the pumps and some
interesting statistics to be pulled out. I think I can get the SNMP stuff
running too and have emails being sent with a bit of work.
Elaine and I went to see Team America: World Police last night. Amusing,
childish and extremely coarse in places. Typical Parker and Stone fayre to be
honest, although some of the songs were quite catchy. Climbing tonight, with a
chance of kebabs afterwards. I bought some mango chutney especially.
When I say I, what I mean is I whinged enough times that Elaine got
some the next time she went to Tesco.
[16:45] Quick entry as I'm off to the cinema. Have had monster success with
machine configurations, getting things added on to other people's work queues
(to give me results to work with) and generally getting things to work on my
machine (like sound, should have plugged in headphones sooner!).
Another reasonably successful day over with I'm going to go to the cinema to
see Team America: World Police with Elaine. It's our first 'date' ever. Does
going to Tesco for thw weekly shop afterwards invalidate it?
[19:00] So today I decided that it might be worth working out whether or not
it was possible to add new modules to a running kernel on my machine. This
turned out to be a bit harder than I'd envisaged. Suffice it to say that I
need to remember the command 'make oldconfig' rather than 'make dep' in the
grand scheme of things. A happy by-product of this kernel frottage is that I
now have BOFHcam I back up and running (as you can see if you take a look at
the main menu).
I now consider myself to be far more clued into how this whole kernel thing
goes about happening (even if I'm a bit annoyed at RHEL3 and its shoddy
misuse of source files. As a result I'm now going home happy, rather than
[23:55] Talk about a long day. Sorry, I'm doing this from home before bed. I
was in reasonably early due to Elaine getting up to ensure she worked her hours
while taking some time to go to the gym during the day (when it's free for her).
Once in I started on some more RedHat Network stuff, trying to make some sense
out of the concepts I have to deal with. While I had most machines up to date
there were still some inconsistencies in the values for licenses, subscriptions
I took a break to go and find my new linux PC and managed to get a very helpful
guy from PC Support to bring it and a monitor down with it in short order. Of
course it was shy half a gig of memory and the PS/2 expansion card I'd need to
run my natural keyboard (Dell GX280) but these were both expected. I fully
intended to install it over lunch but ended up talking to Ian, who'd been in the
building for training and was due a tour of the machine room.
After lunch I installed the box and took the opportunity to install one of the
BTTV capture cards for BOFHcam to go back on the air. I also cleared its
operation with my two line managers, just in case. So long as it only points
at me and I let people know it's there, we're sorted. Of course, once I had
the machine installed I decided to register it with RHN. No dice. A
frustrating number of hours later and I'd discovered a new level of complexity
to the situation. Not only do you have licenses and entitlements. You also
have channel subscriptions, which also cost money. This is... annoying.
Anyway, tearing my hair out I finally discovered how to activate a
subscription. Of course, it being Friday the subscription didn't activate
until about ten minutes ago. This meant that I sent a fairly involved email
to RHN Technical Support and got to speak to the lovely lady at RHN-EMEA who
clued me in to a few few things and left me feeling that, should I have any
more problems, she'd be a very useful person to speak to again.
By that time it was about 18:30 so I went home, made some food and watched
five episodes of Scrubs back-to-back with Elaine. A reasonable end to the
week. All I need to do on Monday is work out why my monitor is not doing the
resolution I want, and why we have ten machines registered with RHN but eleven
[17:20] It's great, for the first time in a long time I'm consistently too busy
to do Journal updates during the day. I love it. Anyway, just before I go home
here's a bit of a run-down on what I've been doing today. As I left my bike
here last night (on account of being too busy to get home in time to be picked
up) I ran in this morning in almost exactly twenty minutes. I felt exceedingly
unfit so there's every chance of doing the route faster as I get fitter.
Anyway, when I got in I immediately got down to getting machines logged,
entitled and up to date with RedHat Networks. It's been a labour of love
sorting out who has what, what machines are 'off the map' in terms of being
known to RHN and not being more holey than some of those cheeses.
As things stand, at the end of the day, all machines bar one are fully patched
and happy. From here I can extend my domain out to the other machines which
have yet to feel my benevolent hand upon them. I will bring peace and secure
workstations and servers to this place during my tenure here, and it will be
Power-madness aside it's been a good day all things considered (especially
with the climbing last night which helped regrow some thicker skin on my hands)
and I'm actually looking forward to coming in tomorrow. Of course, a day in
sysadmining wouldn't be complete without rebooting a box for the first time and
not having it come back up again. Luckily I'd checked (twice) whether it was
used for anything (it wasn't) and after a few minutes of searching in the
server room I found it attached to a terminal server and was able to fsck it
and send it on its way.
[12:40] I went climbing in London last night. After five minutes it was
obvious how much skill I'd lost over Christmas and how less fit I was. Luckily
I was climbign with two friends I made at the wall who like to go in for
'interesting' problems, so I spent the evening regrowing calluses and getting
my balance back. Getting home through a head wind by bike from the station was
tiring so I was ready for bed when I got in some time around 23:20.
Of course, even though I couldn't get to sleep quickly I had to be up for 07:00
this morning to get to work in time for the 08:30 meeting I had with some of
the important heads of various departments here. After actually finding the
place the meeting was going to happen I settled down to try and keep track of
what was going on and not to end up in a situation where I didn't know what was
happening or what would be expected from me.
I have to say that I think I got a fairly good handle on the proceedings and am
fairly sure I know what I'm going to need to do in the coming weeks. Luckily
though there's a good concept of "of you don't know, tell someone, rather than
suffering in silence and buggering something up" so I'll be relying on that if
I need to do something outside of my current skill set.
[18:10] I now know very much more about RedHat Network and licensing than I
thought I ever would (this is going to be a stock phrase for a while, I think).
There's all kinds of confusion because we I've been assured we bought a certain
number of 'management' entitlements and yet we appear to have a whole
load more 'update' entitlements that no-one knows anything about.
According to the guy who used to be in my post before me this is an artifact of
RedHat Network such that if you connect a machine incorrectly in some fashion
it registers as a purchased-for-nothing update entitlement (presumably to allow
RedHat to prod you about it at a later date or something). So those 'update'
entitlements don't actually exist and will expire in a year from their
start date. In conjunction with that little mess there's the issue of the
actual 'license keys' we have which don't amount to the same number as those
machines we have registered. I think this just comes down to not registering
them via a new system so as to preserve their start dates in terms of expiry.
If I understand things correctly we have a good few server and workstation
licenses left and things should be OK. Maybe. For now I'm going
climbing in an effort to wind down.
[14:35] Still no linux box for me to work on as yet. It's a bit annoying to
have to open PuTTY every time I want to connect to one of the other boxes, but
it gets the job done. I've also been generating points for discussion (I hope)
for tomorrow's early morning meeting. I mean 08:30! What kind of
time is that? In an effort to inject some seriousness into my demeanour at
that time in the morning I will, for the first time in a few years, be wearing
trousers to work.
Don't all laugh at once.
So anyway, the rest of the day is consisting of getting more understanding of
what I'm going to be required to do here in the next few months, putting up
some of my posters, crib sheets and printouts from the last job (to make it
feel a bit more homely here) and finally been given some wallplugs and screwed
my whiteboard to the wall. Things are starting to come together here, at least
in my corner of the office.
I stayed a bit later last night working through a promising line of reasoning
with Shaun about mysterious banner printing with our Oracle applications. That
didn't come to anything but did prove to be an interesting journey into the
depths of Oracle Forms and may yet be useful in the future when other printer
problems appear. I've already received two emails from BOFHcam readers with
some helpful tips for my new tasks (thanks guys). All I need to do now is have
a meeting, get access to the boxes and start hacking about with things. One of
the mindset changes I'm going to have to get used to is that I'm now just the
person who looks after the machines. Someone else installed the applications,
someone else does stuff with the box. All I do is install and maintain. You'd
think this would probably leave my left empty of meaning. But no...
[16:50] We've just spent about fifty minutes trying to work out one of the
more complicated chrooted environments during a supposedly routine certificate
replacement (needed to bounce the server processes). For the first time I
felt vaguely useful as I tracked down where the startup and shutdown scripts
were found and called from. Unfortunately I don't have the experience to get
my explanations across to other people in the right way yet so things took a
little longer while other people came to the same conclusions as me. However
we're all on the same page now so things are sorting themselves out.
[15:15] Despite watching two episodes of Desperate Housewives (not a porn show)
on E4 last night, which didn't end until midnight, I still managed to get up
for a reasonable time this morning.
Of course on getting in to work I came to realise a little more the full impact
of what I've let myself in for here. I had a meeting with the head of
operations whom together with my input drew a rather complicated network/service
diagram embracing three different sites, two firewalls, and numerous different
live, staging and development servers.
It seems I'm responsible for working out a coherent kickstart scheme, reimaging
the machines, maintaining them and handing them off to the developers once
they're up and running. Coupled with a fairly complex set of chroot
environments and we're looking at something that, once I get on top of it all,
will be fairly funky.
I just need to get on top of it all.
Still no linux box or sensible-sized monitor as yet; working on a Windows
machine with a 17" screen which I've convinced to run at 1152x864. Slowly but
surely I'm getting a grip on what's required here, learning where all the
documentation my predecessor left is and filling in more gaps in my self-taught
education of linux and Solaris. All I need to do is just keep up.
[16:10] Today has been pretty good all things considered. We've been working
without our line manager today and I've been learning some good back-end stuff.
I don't think I could run the place on my own yet, but I'm answering questions
and making sense more than 90% of the time.
Printing issues have been cropping up and I've had to stick my nose in to more
PostScript and .PPD files that I ever thought I had to before now. Our line
manager is someone who works overhard some times so she's done a lot of the
sticky stuff. This means that we're playing catchup on what she's achieved but
also that we're keeping well up with the things we need to be doing.
I've been learning about NetBackup, more about lsof than I already knew (not
much!) and about ssh tunnels. Those are especially useful. I think for now,
with a recent change in place which should fix a good few of the printing
problems we'vwe been seeing, I'm going to go home and reflect upon the week.
With luck and a following wind, Elaine, Cormac, Steph and I will be going
climbing this weekend somewhere at some point, followed by some kind of
communal cooking effort at the place Steph is house sitting.
[09:45] I went home at lunch time yesterday to take a shift waiting for the
package to be delivered. Elaine headed off to work and got there at a
reasonable time. Naturally as soon as I started on something CityLink arrived
and practically threw the thing at me when I opened the door. I made him wait
for a signature while I opened the box to make sure everything was there.
Given that this was only my second day in the new job I decided that it might
be a good idea to head back to work and see what I could do to lighten the
load on my colleagues. It's odd being a PFY again but there's a certain
lightness of being in not being the one giving the orders, at least at the
This morning I'm learning about the printing system associated with some of
our applications. It's certainly far more involved that the Windows printing
system but vastly more powerful. I look forward to getting my hands dirtier
Cormac popped round last night to return my portable hard drive. I can now
get my Firefox bookmarks et al off it and onto my machine (soon to be machines
when the other one finally gets delivered) here. We chatted for a while and
played far too much Tekken before he headed off.
This morning in the spirit of just trying to get on with things I curtailed
some of my online activity.
[14:50] I think I've solved my first problem. Well not so much problem as
issue which turns out to be widely recognised and currently under investigation
by minds vastly more superior to my own. Oh, and I had my first delivery to
work, which is cool. Elaine should be vaguely pleased this evening.
I've also just been shown the helpdesk software. Joy of joys.
[13:00] I spent the morning in a very interesting meeting to do with
authentication methods for systems that're being developed elsewhere in the
building. Not really anything to do with my area of expertise but as I need
some kind of entrance into the way things work here it seemed like a sensible
thing to be in on, notwithstanding the fact that the team I've joined is going
to have to support whatever hardware and OS is required to make the systems run.
I'm pretty certain I kept up with what was happening and perhaps even made a
reasonable contribution at one or two points. Either way it's a completely
different working environment to the place I came from, and that's a good thing
in pretty much every way.
I still feel as though I'm not contributing to much yet (second day blues?). I
think this'll change as I get more on-board with how things run here. It's all
a bit strange and new, but my colleagues are great and, time permitting, are
very willing to help. Having Shaun here helps too as a friendly face.
As far as I can tell, Elaine's still at home waiting for the CityLink delivery
we were told would arrive before 12:00. Naturally it hasn't yet.
[13:30] Well, I'm here in my new job. Thus far it's not been too bad. There
aren't any PCs for me yet, the phone line hasn't been connected and I don't
have any work to do. Nevertheless I feel like I should be learning already
and trying to help with things. Not that there seems to be much happening that
isn't under control.
I've rearranged where my desk is so that I don't have my back to the door.
All that's on it at the moment is my keyboard and mouse from my old workplace,
a notepad which is due to be filled with things I need to remember/learn/keep
track of and a list of all the machines in the machine room and what they do.
Compared to the last place there are... a lot more. Still, things seem to be
fairly straightforward so I'm sure, with a little work, I should get a handle
on things within a few weeks. This afternoon I'm going to be adding a bit of
depth to my shell scripting while I wait for my PCs to turn up. This could be
more than a few days though.
[17:40] OK, I now have a PC (running Windows), a phone with a number, a desk
behind which I can sit and some idea of what I need to do when I come in
tomorrow, I think. I think it's time to go home and collapse.
[22:30] Today has mostly been about preparing the house for the new working
year (week), and getting some more things out of the garage and house and
into the loft. As well as consolidating a lot of the bedding (throwing out
a whole load of old stuff) we got to bin some of the less decent pillows,
duvets and sheets.
Elaine submerged herself in The Sims 2 for the rest of the day while I made
some progress in getting involved a bit more in one of the local climbing
clubs as part of my decision to do a lot more outdoor stuff this year, no
matter the weather. Hopefully I'll be able to drag Elaine along too after a
[03/01/2005 22:25] By the time we got up there was an SMS from Bob asking if
we wanted to see House of Flying Daggers at the cinema. Owing to the success
of Hero in our opinion we opted for an early afternoon showing so we could
head back and make a decent Sunday meal with the last of our Christmas food.
This was a duck and, after returning from the film we started work, only to
find we were a few cloves or garlic short. Naturally the one place we
expected to be open was closed.
Nevertheless, the duck cooked off very well and we have a wonderful final
Christmas-period dinner before lying back on the sofa to watch the last of
Elaine's Red Dwarf (series 5) DVD episodes.
[03/01/2005 22:15] Shaun and Linda woke us from our vaguely alcohol and late
night induced sleep with the offer of a walk around a nearby country park to
shake out the cobwebs a bit. After some mad dashing around and showering
we were ready with seconds to spare. Trailing a very active young boy
around and some cold winds is both fun and tiring. Elaine and I took some
time to birdwatch near one of the man-made lakes while Thomas played on the
swings and such before I took a turn with Thomas in the back papoose as we
explored some more of the area. Heading back to Shaun and Linda's house we
were treated to some wonderful hot crumpet action before settling down to
quietly watch a film and relax. As time went on we got a lift back home in
time to greet Andrea who was on her way back from a local football match.
Tea and mince pies followed before we were turned green with envy by Andrea's
tales of her experiences in Australia recently. We have to go
there one day, all it'll take is money. Lots of it.