[16:20] Very grateful to have a quiet day at work today. Pretty much nothing
went wrong, so I was able to catch up with things and generally recover from the
week somewhat. With Rachel arriving last night (which was lovely), and her
coming back again tonight (she took the day off for a long run and then trains
to see her parents and then back again) for the whole bank holiday weekend, I
think I might be feeling a whole lot more relaxed by Tuesday, when everything
starts up again. Lots happening next week, but hopefully much more
planned/organised that this week started out.
This weekend I'll be running fast tomorrow morning before/during/after parkrun,
then heading into town for some book browsing/shopping/reading, restaurant
voucher spending for lunch (and maybe even dinner), and then home for relaxing.
Then on Sunday we'll be running long and slow and hopefully enjoyably (that's
mandated in my training plan), and then doing lots of relaxing. And then on
Monday cycling out to a local National Trust (or English Heritage (or none of
the above)) property for a picnic, possibly with other people, too.
[17:35] When I got home last night the building work had been completed and
everything looks fine. It needs to rain fairly hard for a few minutes now
though to wash away all the brick dust from the angle grinder, though. I spent
a few minutes grubbing around behind the washing machine with some bleach and
water to try and deal with the leftover rat scuffing, although I did start by
sweeping out the last of the dessicated (and not quite so dessicated) rat
droppings. I'll push the washer back and pop on the kick/skirting board below
the cupboard when I get home this evening.
This morning I decided to get in early so I could begin the V2Ving of three
servers from a KVM host to the VMWare cluster. It actually all went really well
and I was done with all three servers before 10:30, so that was good. The
remainder of the day has been spent doing small but important other things and
being more than a little concerned about my inability to lose the excess I
always have around my waist, which seems like it's very, very slowly increasing,
despite all of the (I assumed) fat-burning exercise I do. The lunchtime run
today was anything but fat-burning as it was my now-weekly track/sprintervals
session. I thought it went really well, but I still don't feel like I can run a
marathon at anything close to my PB pace.
Anyway, it's home time now, via Tesco for an evening meal for Rachel and I. I'm
very much looking forward to her being home again, although she's not working
from home tomorrow but heading off to see her parents for a flying visit and
then coming home again some time in the evening. Either way, I hope tomorrow is
quiet for us both.
[17:55] I had a day of working from home yesterday as builders came to deal with
my rat incursion problem. After getting up early to run for an hour before they
arrived, they duly did (right on time) and got down to work. Within an hour or
so they'd discovered that the people who'd lived here before me had done a
shocking job with the plumbing/sewerage and had left things in a terrible state.
The pottery piping was broken and collapsed in on itself (leading to water from
the sink (and dishwasher when I use it) emptying pretty much straight into the
earth. And there was a hole in the foundations which it seems likely was the
rat ingress point into the house. They didn't take up the floor in the utility
area, but it seems likely there's an excavated void under there. Additional
surprises were the use of internal-only pipework to bridge gaps in the system,
and wrong-sized piping which was then part-filled with rubble because of that.
All of this was torn out and replaced with proper piping and a second gully for
the sole purpose of venting washing machine water. The repair of the pipework,
plus a large amount of hardcore, pea shingle, and concrete surrounding it, plus
concrete in the other end of the rat route inside the house, plus grates over
the gullys should see the end of rats in the house, and mean that I can get some
marigolds on tonight and once and for all bleach and clean where they've been.
The builders almost finished yesterday, but went home before everything was
reinstated. In the meantime I spent the day glued to my laptop putting in
pretty much a full day of work.
This morning I waited for the builders to arrive and made sure they got started
OK and then headed into work for another full and busy day. Lots has gone on,
but both of the major problems that appeared on Monday that I didn't solve on
the day both got sorted, which was great. Although one of them not by me, which
is great in a different-but-equally-great way. I had a number of meetings, one
of which contained a moment where I was thanked for my work, which was nice. I
just wish Management would get on with approving my getting some more money for
what I do.
I'm in semi-early tomorrow to V2V three servers, at the same time as
significantly downsizing disk capacity, RAM, and CPU usage. Hopefully that'll
all go according to plan. Rachel also arrives tomorrow evening, thank goodness.
I could really do with her being around to hug.
[18:10] I have had a horrible, stupid day. When I got into work there was a
mass of red lights on the monitoring system. It turns out that on Friday the
Networks people had (quite rightly) readdressed the core network switches to get
them managed on their network. Only those switches were the NTP servers for our
entire network. So every single one of my servers was complaining about being
unable to get NTP information. I spent the first two hours of my day going
through every single server getting them to use the newer, more permanent NTP
servers. Once that was done I started on all of the DRACs/LOMs/SPs and other
devices which have NTP settings. I'm fairly certain I haven't got to them all,
but the ones I haven't remembered probably don't matter all that much. Then,
towards the end of that, a DBA came in and told me that our most important
service was suffering from a RAID disk failure and what looked like
database corruption. Checking things out I saw that indeed a disk had gone
wrong, but it had gone so wrong that it was being reported as "removed" rather
than just failed, and potentially in breaking it'd caused the rest of the RAID
array to begin a creeping block corruption (known in this case as "punctured
raid stripes"). We had the same issue on another server last week, or the week
before, and I migrated that to a VM. Not something I could do in this case. A
major incident was triggered and the upshot is that standby/DR was invoked and
the DB is now up and running on another server. I've asked for some new disks
from our supplier, and they should arrive tomorrow (which isn't all that helpful
based on what I'll be telling you in a moment). While all of that was
going on our primary monitoring server (which runs a stupidly old version of
RHEL) decided to have a journalling I/O error and set its filesystem read-only.
Because of the age of the server and its hardware I had to cycle out between
major cloudbursts to reboot it and see if that brought it back. It seemed to
boot fairly happily, but by the time I got back to the office the filesystem was
read-only again. I've struggled to make any headway with it, and failed
spectacularly to create a bootable USB stick with some rescue software on it.
Frustratingly, I'm not actually in work tomorrow as I'll be home while someone
comes to (I hope) fix my rat infestation issue. Given my current luck it'll
turn out that what I was told wouldn't be required will be required and
a lot more money, time, and inconvenience is going to be necessary.
Oh, and Rachel and I visited her parents this weekend. As I want to go home
I'll keep it short. We had a lovely time with them, and also Rachel and I ran
20 miles on the Saturday morning while her parents waited in for someone from
the RAC to come out and reattach a bit of the underside of the car which
detached as we were coming home from a delicious meal out together on Friday
evening. We got the train home on Sunday lunchtime and then immediately cycled
out to a pub about 13 miles away to have a late lunch with two of Rachel's
friends (and their dogs). Once we got home we had a few hours of relaxation,
Indian takeaway, and television. Then it was bed time and getting up this
morning to face the waking nightmare that has been today.
I'm going home now, getting some food from Tesco and then hoping nothing else
goes too wrong while I'm away from the office tomorrow. Potentially for the
[12:05] A short entry for a short work day. Rachel arrived last night, and
there was much rejoicing (even a bit of dancing). We had a lovely evening and
meal, and then as we were getting up early this morning, went to be fairly
early, too. Sadly, I didn't have a stunning night's sleep, but it was at least
enough that I didn't feel totally terrible when we got up at around 06:30 this
morning. After getting prepared we headed out for our individual runs, doing
the first mile or so together before splitting off in different directions down
the tow path beside the river. I was back at the house when the hour was up,
Rachel a few minutes later. After showering and breakfast I headed into work
and have spent the morning working on a physical server which has needed to be
installed for a few weeks now. I may just about get it done before I have to
head home again to have lunch with Rachel before we get trains to see her
parents for the weekend. Tonight we have dinner out, tomorrow morning we're up
early for a 20 mile run, and then into the local town for lunch. After that
we'll be heading to a local National Trust property or similar for an amble and
stuff before going home. Sunday morning should hopefully consist of a bit of a
lie-in before breakfast and then trains home. As soon as we get home we'll be
off on bikes for a 13 mile cycle to see some of Rachel's friends for a very late
Sunday pub lunch. Once we're capable of doing so we'll cycle home again.
Hopefully the weather will be nice enough that we can take our time over both
cycles. There might then be a few hours of Sunday evening left before we have
to go to bed for the beginning of next week.
[16:20] A fairly useful day today. I even slept semi-well, too. But anyway, it
started with me getting up at 06:00 so I could get in early to reboot a VM that
wasn't seeing its extra RAM. Changing the guest OS type in VMWare was a bit odd
at the end of the procedure, but everything seems to have worked out in the end.
More minor tweaking may be required in the future. After that I found a decent
deal for my gas and electricity tariffs and began the procedure for switching to
it (back to E.ON from EDF, who I'd transfered to from E.ON a year ago). It'll
mean my smart meters regain their brains again, not that it really matters as
they did everything I needed anyway, even when dumb. Work-wise I was consulted
a few times on an email I sent before going home last night, but everything
seemed to be fairly straightforward, and I created a new user for my Red Hat
Satellite installation for someone who's going to be managing a different chunk
of it (read: Organization in Satellite parlance). Hopefully we won't tread on
each other's toes. I just don't want him to break anything.
Rachel arrives this evening, which'll be fab. I'm going to go home and spruce
the house and myself up a bit.
[18:05] An OK night's sleep, but still didn't feel 100% when I got up this
morning. At work I managed to get a reasonable number of things done, including
some coordination with my two contractors for getting the remedial building
work done to deal with my rat issue. To cut a long story short, it should be
getting done by the more expensive contractor next Tuesday, when I'll work from
home until it's done. Over lunchtime I went for a steady one hour run in the
pouring rain and ended up having the same issue with my trainers as I did with
the last pair in that the insoles/inserts start to slide around when the shoe is
very wet, and end up with permament folds in them. This was solved (I think, I
don't know as I went running again in them in the wet) by getting some
replacement inserts from the shop I bought the shoes from (they had a box of
spares in the back from shoes people had their own orthotics for) which had a
more textured underside. In any event, I called the shop this afternoon and
they're going to call the shoe manufacturer and see if they'll send out some of
the same textured inserts again for me. If that doesn't work, and in the
meantime as I'm not sure they'll arrive any time soon, I'm going to see if the
ones I got before (and are in my old shoes) are still usable.
In work news, my colleague and I have been working on some licensing
issues/software and have come to the conclusion that a number of KVM guests are
going to have to be moved to the VMWare cluster so we can get a read on the
resource allocations they have. The alternative is trying to shoehorn RHEV-M on
top of two bald RHEL6 KVM hosts and hoping that it works. This would
necessitate a whole load of work at the Red Hat Satellite end, as well as
working out how and where to install RHEV-M. The other alternative is buying
and setting up six new physical servers in short order, with low enough spec.
that they don't actually cost us more in licensing costs for the greater oomph
Anyway, I've been here far too long today and I have a moderately early start
tomorrow, so I'm going home.
[17:40] I'd class today as s light tick in the "successful" column. I got up at
a normal time, got to work safely, achieved a few useful things, helped my
colleague/team leader work through the installation of some really stupid IBM
licensing software (we're nearly sorted, after a day's on and off work; success
tomorrow, I think), had a good track/speed session (following a useful phone
conversation with my running coach last night), and managed to get through the
afternoon without anything breaking (either because I touched it, or it just
broke on its own). I even managed to tidy up a filling disk issue without
deleting anything important. Sadly, our monitoring system still looks pretty
lousy with red and yellow icons. I'll see what I can do about some of that
tomorrow, I think.
For now though I'm off home to worry more about the rats I heard under the
utility area floor again last night. I'm waiting on both of the builders I got
quotes from to tell me when they might be able to do the work we think needs
doing. I just hope that as and when it can be done it's all that actually does
need doing to solve my issue...
[17:45] So I tried to get to Croydon on Friday night, but the power went out and
that turned into a fairly long night for not going anyway. In the end I was
able to get home and just order a pizza (which while tasty, wasn't a great idea
for my stomach). I then had to get up pretty early on Saturday morning so I
could get trains to London and then to Kingston (meeting Rachel at Victoria
along the way). We got to the race HQ in plenty of time and then did the race
in fairly OK weather in the end. I got what I'd been hoping for, but not what I
wanted in terms of time. Rachel, of course, smashed it and did very well. We
headed back to Croydon by train and tram and had a nice afternoon and evening
together. On Sunday we both ran out along the Vanguard Way, me to turn around
after 1h15m, Rachel to head on and then onto the North Downs Way for a bit
until she met someone for lunch. I made it out and back fairly happily, had a
shower, had lunch, and then waited for Rachel to come home by train. Owing to
one thing and another she didn't get back until well past when we'd hoped she
would, but everything was OK in the end. More relaxing then took place.
This morning my train back to work was slightly delayed, which put me behind
somewhat in catching up with things. However, owing to not running today, and
most other things not breaking too much I've been able to catch up with
everything for the most part. I've got lots to do this evening, and I'm not
sure I'll get it all done, so some might have to wait until tomorrow.
[16:10] Got up early so I could get in and Get Stuff Done before other people
came in to bother me. This mostly succeeded. I've now had a day of getting a
reasonable amount of stuff done, found out that a lot of the errors I was seeing
on my recent Satellite upgrade are harmless (thank goodness), and even managed
to have my lunch at the right time of day.
Now I'm off to the train station to head to Croydon and see Rachel for the
weekend. We'll be running a half marathon tomorrow (in allegedly Very Bad
Weather), and then perhaps 2h30m of less painful running on the Sunday if we can
be bothered and the weather's OK.
[18:10] So this morning I got up early so I could be in work for 07:00. This
enabled me not only to get ready for the work with one of the DBAs moving the
service that had to remain up from the stricken server to its partner, but also
do some other work I'd been asked for. In the end the latter work got done, but
the person who'd asked for it didn't come in to stand behind me (as I was sure
he'd said he would). The former work also got done (eventually), which allowed
me to do a P2V of the filesystem to the VMWare cluster. This completed some
time around lunch time, but there were then numerous issues with firing up the
new virtual version of the host, which meant I didn't get to do my run until
gone 14:30, which was a bit annoying. Nevertheless is was good to get the
server I'd P2V'd running and stable. It's not doing everything it should yet,
but neither is is doing anything it shouldn't, which is more important in a way.
I really need to go home now though, as I may need to come in early tomorrow
morning and I could do with a bit of downtime at this point.
To be honest with you, at this point I'm feeling a little flat and lacking in
excitement about anything right now. Mentally and physically, I'm not really
interested in all that much at the moment. Hopefully this will change soon.
[08/08/2019 11:10] I'm writing this on Thursday morning as around lunch time
yesterday we had another evacuation due to a strong smell of gas. It's been
ascertained that the smell isn't coming from inside the building, but as far as
I'm aware no-one's managed to track down where it's coming from or how to stop
it. We'll have to see. In any event, rather than hang around for the all-clear
again I just went home. I thought I'd just be going home to relax and read all
afternoon rather than actually working, but that didn't turn out to be the case.
I'd replaced a disk in a RAID5 set when the evacuation happened as it seemed
like a sensible use of my time to cycle over to the server room where the work
needed doing whilest waiting for the situation to be dealt with. The work went
perfectly well, but when I got home to check that the RAID array was happy... it
wasn't. The disk had rebuilt, but the virtual disk still thought there was a
problem. Logging into the box I found that there was at least one punctured
RAID stripe. This is a Bad Thing that you're more than welcome to read up on if
you like, but suffice it to say the solution is to make sure you have backups,
blow away the virtual disk entirely, make sure all your physical disks are fine,
and then recreate the virtual disk. After that you can reinstall your operating
system and restore your data. This would be significantly time-consuming. So I
spent the afternoon and evening working out a plan to save everything on the
server, migrate the services that definitely needed to stay up, and decide how
to P2V it to the VMWare cluster (convert it to a virtual server). All this
while wondering how long the filesystem was going to last on the server before
it either corrupted totally, or went read-only. After a while I decided to make
a list on paper of everything I needed to consider and then make a conscious
effort to forget about it until the morning. This seemed to work as I slept
[18:15] Sooooooo, yesterday was interesting. After getting to work for around
10:00 due to trains back from Croydon being what they were I wasn't in work for
more than about an hour and a half before the fire alarm went off and we all
evacuated. After ten minutes outside it turned out that there had been a
strange smell in the building which people thought was gas. To cut a long story
short, after waiting around for over two hours, and eventually being allowed in
with an escort to get my bag so I could have lunch, we were sent home. Given
the weekend I had (more on that below) I ended up dozing badly on the sofa all
afternoon, having dinner, and then heading to bed at around 21:30.
The reason I was so tired is that on Saturday, after getting to Rachel's on the
Friday night, I ran a fast (although not as fast as Rachel) parkrun in quite
warm temperatures, and then had an afternoon of this and that. And then on
Sunday ran the Vanguard Way (trail) Marathon and despite running as hard as I
could on the way back (we ran with friend + dog on the way out at a much slower
pace), ended up 16 minutes behind Rachel over the half marathon distance back
to Lloyd Park. She's amazingly fast, and has such deep reserves of speed and
stamina these days, I'm in awe of her (and somewhat disappointed in my lack
thereof). Anyway, after doing that, hanging around for a bit to see friend +
dog back to the finish, having some chips and drinks, and feeding the dog, we
headed home and had a nice quiet afternoon and evening on the sofa before an
Indian and then bed.
Back to today and I've had a fairly useful day of stuff and things, although a
few emails have come in that I'm not entirely looking forward to dealing with.
I also had a meeting over my usual lunch/run time, so the run was pushed back to
a start time of 14:20, which meant not getting back to my desk to eat until
around 15:45 or so. That shouldn't happen often, though.
I'm going to head home now and try to have a relaxing evening. Tomorrow I need
to call a builder about fixing my drains/rat issue, and also contact my
buildings insurance people about the increase in the renewal quote.
[13:20] Gas leak suspected in building. Being sent home. More on the weekend,
today, and tomorrow... well, tomorrow.
[16:00] A gentle start to the day, getting up just after 07:00, doing the
morning routine, and then cycling to work. I did take in all the stuff I need
for this weekend in Croydon, as well as for kayaking this evening in London, and
some things Rachel left last weekend. Not to mention new tyres for the bike
Rachel had stolen from the station and just replaced last night with a very
similar bike I found on Gumtree. I did manage to drop the tyres (and the framed
picture Rachel had left at mine that I'd bought for her and she'd taken to show
her parents), but the damage to the frame wasn't too bad, so that's OK.
Rachel's sensibly told me to leave the tyres at work for another time as lugging
them through London this evening and having the porterage people deal with them
while we're kayaking just doesn't make sense. Especially as the 'new' bike has
better tyres than the old one that I originally bought them for. Next time.
At work today I've done some firewall work, collected data for a cracked
WordPress site, gone for a very gentle 45 minute run, showered, lunched, got a
photo printed a few times (first couple were the wrong size), and even managed
to answer some work emails. Now I'm off to the station to get a slightly
earlier train than usual so I can get across London in time to meet Rachel.
Going through Bank at the time of day I'll get there is going to be hellish.
[16:10] Not quite as early to bed last night as I would have liked, if I'm
honest. I still got a reasonable night's sleep and woke just before the alarm
this morning for an early start. I had a nice cycle into work with mostly
deserted streets, and was able to get the patching and rebooting (and firmware
updating) done without any real issues, which was great. That meant I was
better able to concentrate on some oddnesses with the load balancer and firewall
(separate problems) which I'm now able to say have both been sorted. One
required SNAT to be turned on for a rule, and the load balancer thing required
two steps of rule application and rule ordering before everything was as it was
supposed to be. Getting them both sorted was good though, and has left me with
time to work on banishing some warning lights from the monitoring system. Oh,
and today is worth noting as the day I got VMWare's vSphere/vCenter working with
Firefox and Flash again. Whether it's worth doing remains to be seen.
Otherwise, I think that's about it for today. I was in early, so I'm leaving
early. Tomorrow's Friday, and there should be a bit more going on, even if it's
after work rather than during.