Menu Members Guests LTQ
BOFHcam navigation bar

BOFHJournal

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019

July's Journal



23/08/2019 [16:20] Very grateful to have a quiet day at work today. Pretty much nothing went wrong, so I was able to catch up with things and generally recover from the week somewhat. With Rachel arriving last night (which was lovely), and her coming back again tonight (she took the day off for a long run and then trains to see her parents and then back again) for the whole bank holiday weekend, I think I might be feeling a whole lot more relaxed by Tuesday, when everything starts up again. Lots happening next week, but hopefully much more planned/organised that this week started out.

This weekend I'll be running fast tomorrow morning before/during/after parkrun, then heading into town for some book browsing/shopping/reading, restaurant voucher spending for lunch (and maybe even dinner), and then home for relaxing. Then on Sunday we'll be running long and slow and hopefully enjoyably (that's mandated in my training plan), and then doing lots of relaxing. And then on Monday cycling out to a local National Trust (or English Heritage (or none of the above)) property for a picnic, possibly with other people, too.

22/08/2019 [17:35] When I got home last night the building work had been completed and everything looks fine. It needs to rain fairly hard for a few minutes now though to wash away all the brick dust from the angle grinder, though. I spent a few minutes grubbing around behind the washing machine with some bleach and water to try and deal with the leftover rat scuffing, although I did start by sweeping out the last of the dessicated (and not quite so dessicated) rat droppings. I'll push the washer back and pop on the kick/skirting board below the cupboard when I get home this evening.

This morning I decided to get in early so I could begin the V2Ving of three servers from a KVM host to the VMWare cluster. It actually all went really well and I was done with all three servers before 10:30, so that was good. The remainder of the day has been spent doing small but important other things and being more than a little concerned about my inability to lose the excess I always have around my waist, which seems like it's very, very slowly increasing, despite all of the (I assumed) fat-burning exercise I do. The lunchtime run today was anything but fat-burning as it was my now-weekly track/sprintervals session. I thought it went really well, but I still don't feel like I can run a marathon at anything close to my PB pace.

Anyway, it's home time now, via Tesco for an evening meal for Rachel and I. I'm very much looking forward to her being home again, although she's not working from home tomorrow but heading off to see her parents for a flying visit and then coming home again some time in the evening. Either way, I hope tomorrow is quiet for us both.

21/08/2019 [17:55] I had a day of working from home yesterday as builders came to deal with my rat incursion problem. After getting up early to run for an hour before they arrived, they duly did (right on time) and got down to work. Within an hour or so they'd discovered that the people who'd lived here before me had done a shocking job with the plumbing/sewerage and had left things in a terrible state. The pottery piping was broken and collapsed in on itself (leading to water from the sink (and dishwasher when I use it) emptying pretty much straight into the earth. And there was a hole in the foundations which it seems likely was the rat ingress point into the house. They didn't take up the floor in the utility area, but it seems likely there's an excavated void under there. Additional surprises were the use of internal-only pipework to bridge gaps in the system, and wrong-sized piping which was then part-filled with rubble because of that. All of this was torn out and replaced with proper piping and a second gully for the sole purpose of venting washing machine water. The repair of the pipework, plus a large amount of hardcore, pea shingle, and concrete surrounding it, plus concrete in the other end of the rat route inside the house, plus grates over the gullys should see the end of rats in the house, and mean that I can get some marigolds on tonight and once and for all bleach and clean where they've been. The builders almost finished yesterday, but went home before everything was reinstated. In the meantime I spent the day glued to my laptop putting in pretty much a full day of work.

This morning I waited for the builders to arrive and made sure they got started OK and then headed into work for another full and busy day. Lots has gone on, but both of the major problems that appeared on Monday that I didn't solve on the day both got sorted, which was great. Although one of them not by me, which is great in a different-but-equally-great way. I had a number of meetings, one of which contained a moment where I was thanked for my work, which was nice. I just wish Management would get on with approving my getting some more money for what I do.

I'm in semi-early tomorrow to V2V three servers, at the same time as significantly downsizing disk capacity, RAM, and CPU usage. Hopefully that'll all go according to plan. Rachel also arrives tomorrow evening, thank goodness. I could really do with her being around to hug.

19/08/2019 [18:10] I have had a horrible, stupid day. When I got into work there was a mass of red lights on the monitoring system. It turns out that on Friday the Networks people had (quite rightly) readdressed the core network switches to get them managed on their network. Only those switches were the NTP servers for our entire network. So every single one of my servers was complaining about being unable to get NTP information. I spent the first two hours of my day going through every single server getting them to use the newer, more permanent NTP servers. Once that was done I started on all of the DRACs/LOMs/SPs and other devices which have NTP settings. I'm fairly certain I haven't got to them all, but the ones I haven't remembered probably don't matter all that much. Then, towards the end of that, a DBA came in and told me that our most important service was suffering from a RAID disk failure and what looked like database corruption. Checking things out I saw that indeed a disk had gone wrong, but it had gone so wrong that it was being reported as "removed" rather than just failed, and potentially in breaking it'd caused the rest of the RAID array to begin a creeping block corruption (known in this case as "punctured raid stripes"). We had the same issue on another server last week, or the week before, and I migrated that to a VM. Not something I could do in this case. A major incident was triggered and the upshot is that standby/DR was invoked and the DB is now up and running on another server. I've asked for some new disks from our supplier, and they should arrive tomorrow (which isn't all that helpful based on what I'll be telling you in a moment). While all of that was going on our primary monitoring server (which runs a stupidly old version of RHEL) decided to have a journalling I/O error and set its filesystem read-only. Because of the age of the server and its hardware I had to cycle out between major cloudbursts to reboot it and see if that brought it back. It seemed to boot fairly happily, but by the time I got back to the office the filesystem was read-only again. I've struggled to make any headway with it, and failed spectacularly to create a bootable USB stick with some rescue software on it.

Frustratingly, I'm not actually in work tomorrow as I'll be home while someone comes to (I hope) fix my rat infestation issue. Given my current luck it'll turn out that what I was told wouldn't be required will be required and a lot more money, time, and inconvenience is going to be necessary.

Oh, and Rachel and I visited her parents this weekend. As I want to go home I'll keep it short. We had a lovely time with them, and also Rachel and I ran 20 miles on the Saturday morning while her parents waited in for someone from the RAC to come out and reattach a bit of the underside of the car which detached as we were coming home from a delicious meal out together on Friday evening. We got the train home on Sunday lunchtime and then immediately cycled out to a pub about 13 miles away to have a late lunch with two of Rachel's friends (and their dogs). Once we got home we had a few hours of relaxation, Indian takeaway, and television. Then it was bed time and getting up this morning to face the waking nightmare that has been today.

I'm going home now, getting some food from Tesco and then hoping nothing else goes too wrong while I'm away from the office tomorrow. Potentially for the whole day.

16/08/2019 [12:05] A short entry for a short work day. Rachel arrived last night, and there was much rejoicing (even a bit of dancing). We had a lovely evening and meal, and then as we were getting up early this morning, went to be fairly early, too. Sadly, I didn't have a stunning night's sleep, but it was at least enough that I didn't feel totally terrible when we got up at around 06:30 this morning. After getting prepared we headed out for our individual runs, doing the first mile or so together before splitting off in different directions down the tow path beside the river. I was back at the house when the hour was up, Rachel a few minutes later. After showering and breakfast I headed into work and have spent the morning working on a physical server which has needed to be installed for a few weeks now. I may just about get it done before I have to head home again to have lunch with Rachel before we get trains to see her parents for the weekend. Tonight we have dinner out, tomorrow morning we're up early for a 20 mile run, and then into the local town for lunch. After that we'll be heading to a local National Trust property or similar for an amble and stuff before going home. Sunday morning should hopefully consist of a bit of a lie-in before breakfast and then trains home. As soon as we get home we'll be off on bikes for a 13 mile cycle to see some of Rachel's friends for a very late Sunday pub lunch. Once we're capable of doing so we'll cycle home again. Hopefully the weather will be nice enough that we can take our time over both cycles. There might then be a few hours of Sunday evening left before we have to go to bed for the beginning of next week.

15/08/2019 [16:20] A fairly useful day today. I even slept semi-well, too. But anyway, it started with me getting up at 06:00 so I could get in early to reboot a VM that wasn't seeing its extra RAM. Changing the guest OS type in VMWare was a bit odd at the end of the procedure, but everything seems to have worked out in the end. More minor tweaking may be required in the future. After that I found a decent deal for my gas and electricity tariffs and began the procedure for switching to it (back to E.ON from EDF, who I'd transfered to from E.ON a year ago). It'll mean my smart meters regain their brains again, not that it really matters as they did everything I needed anyway, even when dumb. Work-wise I was consulted a few times on an email I sent before going home last night, but everything seemed to be fairly straightforward, and I created a new user for my Red Hat Satellite installation for someone who's going to be managing a different chunk of it (read: Organization in Satellite parlance). Hopefully we won't tread on each other's toes. I just don't want him to break anything.

Rachel arrives this evening, which'll be fab. I'm going to go home and spruce the house and myself up a bit.

14/08/2019 [18:05] An OK night's sleep, but still didn't feel 100% when I got up this morning. At work I managed to get a reasonable number of things done, including some coordination with my two contractors for getting the remedial building work done to deal with my rat issue. To cut a long story short, it should be getting done by the more expensive contractor next Tuesday, when I'll work from home until it's done. Over lunchtime I went for a steady one hour run in the pouring rain and ended up having the same issue with my trainers as I did with the last pair in that the insoles/inserts start to slide around when the shoe is very wet, and end up with permament folds in them. This was solved (I think, I don't know as I went running again in them in the wet) by getting some replacement inserts from the shop I bought the shoes from (they had a box of spares in the back from shoes people had their own orthotics for) which had a more textured underside. In any event, I called the shop this afternoon and they're going to call the shoe manufacturer and see if they'll send out some of the same textured inserts again for me. If that doesn't work, and in the meantime as I'm not sure they'll arrive any time soon, I'm going to see if the ones I got before (and are in my old shoes) are still usable.

In work news, my colleague and I have been working on some licensing issues/software and have come to the conclusion that a number of KVM guests are going to have to be moved to the VMWare cluster so we can get a read on the resource allocations they have. The alternative is trying to shoehorn RHEV-M on top of two bald RHEL6 KVM hosts and hoping that it works. This would necessitate a whole load of work at the Red Hat Satellite end, as well as working out how and where to install RHEV-M. The other alternative is buying and setting up six new physical servers in short order, with low enough spec. that they don't actually cost us more in licensing costs for the greater oomph they'd provide.

Anyway, I've been here far too long today and I have a moderately early start tomorrow, so I'm going home.

13/08/2019 [17:40] I'd class today as s light tick in the "successful" column. I got up at a normal time, got to work safely, achieved a few useful things, helped my colleague/team leader work through the installation of some really stupid IBM licensing software (we're nearly sorted, after a day's on and off work; success tomorrow, I think), had a good track/speed session (following a useful phone conversation with my running coach last night), and managed to get through the afternoon without anything breaking (either because I touched it, or it just broke on its own). I even managed to tidy up a filling disk issue without deleting anything important. Sadly, our monitoring system still looks pretty lousy with red and yellow icons. I'll see what I can do about some of that tomorrow, I think.

For now though I'm off home to worry more about the rats I heard under the utility area floor again last night. I'm waiting on both of the builders I got quotes from to tell me when they might be able to do the work we think needs doing. I just hope that as and when it can be done it's all that actually does need doing to solve my issue...

12/08/2019 [17:45] So I tried to get to Croydon on Friday night, but the power went out and that turned into a fairly long night for not going anyway. In the end I was able to get home and just order a pizza (which while tasty, wasn't a great idea for my stomach). I then had to get up pretty early on Saturday morning so I could get trains to London and then to Kingston (meeting Rachel at Victoria along the way). We got to the race HQ in plenty of time and then did the race in fairly OK weather in the end. I got what I'd been hoping for, but not what I wanted in terms of time. Rachel, of course, smashed it and did very well. We headed back to Croydon by train and tram and had a nice afternoon and evening together. On Sunday we both ran out along the Vanguard Way, me to turn around after 1h15m, Rachel to head on and then onto the North Downs Way for a bit until she met someone for lunch. I made it out and back fairly happily, had a shower, had lunch, and then waited for Rachel to come home by train. Owing to one thing and another she didn't get back until well past when we'd hoped she would, but everything was OK in the end. More relaxing then took place.

This morning my train back to work was slightly delayed, which put me behind somewhat in catching up with things. However, owing to not running today, and most other things not breaking too much I've been able to catch up with everything for the most part. I've got lots to do this evening, and I'm not sure I'll get it all done, so some might have to wait until tomorrow.

09/08/2019 [16:10] Got up early so I could get in and Get Stuff Done before other people came in to bother me. This mostly succeeded. I've now had a day of getting a reasonable amount of stuff done, found out that a lot of the errors I was seeing on my recent Satellite upgrade are harmless (thank goodness), and even managed to have my lunch at the right time of day.

Now I'm off to the train station to head to Croydon and see Rachel for the weekend. We'll be running a half marathon tomorrow (in allegedly Very Bad Weather), and then perhaps 2h30m of less painful running on the Sunday if we can be bothered and the weather's OK.

08/08/2019 [18:10] So this morning I got up early so I could be in work for 07:00. This enabled me not only to get ready for the work with one of the DBAs moving the service that had to remain up from the stricken server to its partner, but also do some other work I'd been asked for. In the end the latter work got done, but the person who'd asked for it didn't come in to stand behind me (as I was sure he'd said he would). The former work also got done (eventually), which allowed me to do a P2V of the filesystem to the VMWare cluster. This completed some time around lunch time, but there were then numerous issues with firing up the new virtual version of the host, which meant I didn't get to do my run until gone 14:30, which was a bit annoying. Nevertheless is was good to get the server I'd P2V'd running and stable. It's not doing everything it should yet, but neither is is doing anything it shouldn't, which is more important in a way. I really need to go home now though, as I may need to come in early tomorrow morning and I could do with a bit of downtime at this point.

To be honest with you, at this point I'm feeling a little flat and lacking in excitement about anything right now. Mentally and physically, I'm not really interested in all that much at the moment. Hopefully this will change soon.

07/08/2019 [08/08/2019 11:10] I'm writing this on Thursday morning as around lunch time yesterday we had another evacuation due to a strong smell of gas. It's been ascertained that the smell isn't coming from inside the building, but as far as I'm aware no-one's managed to track down where it's coming from or how to stop it. We'll have to see. In any event, rather than hang around for the all-clear again I just went home. I thought I'd just be going home to relax and read all afternoon rather than actually working, but that didn't turn out to be the case. I'd replaced a disk in a RAID5 set when the evacuation happened as it seemed like a sensible use of my time to cycle over to the server room where the work needed doing whilest waiting for the situation to be dealt with. The work went perfectly well, but when I got home to check that the RAID array was happy... it wasn't. The disk had rebuilt, but the virtual disk still thought there was a problem. Logging into the box I found that there was at least one punctured RAID stripe. This is a Bad Thing that you're more than welcome to read up on if you like, but suffice it to say the solution is to make sure you have backups, blow away the virtual disk entirely, make sure all your physical disks are fine, and then recreate the virtual disk. After that you can reinstall your operating system and restore your data. This would be significantly time-consuming. So I spent the afternoon and evening working out a plan to save everything on the server, migrate the services that definitely needed to stay up, and decide how to P2V it to the VMWare cluster (convert it to a virtual server). All this while wondering how long the filesystem was going to last on the server before it either corrupted totally, or went read-only. After a while I decided to make a list on paper of everything I needed to consider and then make a conscious effort to forget about it until the morning. This seemed to work as I slept fairly well.

06/08/2019 [18:15] Sooooooo, yesterday was interesting. After getting to work for around 10:00 due to trains back from Croydon being what they were I wasn't in work for more than about an hour and a half before the fire alarm went off and we all evacuated. After ten minutes outside it turned out that there had been a strange smell in the building which people thought was gas. To cut a long story short, after waiting around for over two hours, and eventually being allowed in with an escort to get my bag so I could have lunch, we were sent home. Given the weekend I had (more on that below) I ended up dozing badly on the sofa all afternoon, having dinner, and then heading to bed at around 21:30.

The reason I was so tired is that on Saturday, after getting to Rachel's on the Friday night, I ran a fast (although not as fast as Rachel) parkrun in quite warm temperatures, and then had an afternoon of this and that. And then on Sunday ran the Vanguard Way (trail) Marathon and despite running as hard as I could on the way back (we ran with friend + dog on the way out at a much slower pace), ended up 16 minutes behind Rachel over the half marathon distance back to Lloyd Park. She's amazingly fast, and has such deep reserves of speed and stamina these days, I'm in awe of her (and somewhat disappointed in my lack thereof). Anyway, after doing that, hanging around for a bit to see friend + dog back to the finish, having some chips and drinks, and feeding the dog, we headed home and had a nice quiet afternoon and evening on the sofa before an Indian and then bed.

Back to today and I've had a fairly useful day of stuff and things, although a few emails have come in that I'm not entirely looking forward to dealing with. I also had a meeting over my usual lunch/run time, so the run was pushed back to a start time of 14:20, which meant not getting back to my desk to eat until around 15:45 or so. That shouldn't happen often, though.

I'm going to head home now and try to have a relaxing evening. Tomorrow I need to call a builder about fixing my drains/rat issue, and also contact my buildings insurance people about the increase in the renewal quote.

05/08/2019 [13:20] Gas leak suspected in building. Being sent home. More on the weekend, today, and tomorrow... well, tomorrow.

02/08/2019 [16:00] A gentle start to the day, getting up just after 07:00, doing the morning routine, and then cycling to work. I did take in all the stuff I need for this weekend in Croydon, as well as for kayaking this evening in London, and some things Rachel left last weekend. Not to mention new tyres for the bike Rachel had stolen from the station and just replaced last night with a very similar bike I found on Gumtree. I did manage to drop the tyres (and the framed picture Rachel had left at mine that I'd bought for her and she'd taken to show her parents), but the damage to the frame wasn't too bad, so that's OK. Rachel's sensibly told me to leave the tyres at work for another time as lugging them through London this evening and having the porterage people deal with them while we're kayaking just doesn't make sense. Especially as the 'new' bike has better tyres than the old one that I originally bought them for. Next time.

At work today I've done some firewall work, collected data for a cracked WordPress site, gone for a very gentle 45 minute run, showered, lunched, got a photo printed a few times (first couple were the wrong size), and even managed to answer some work emails. Now I'm off to the station to get a slightly earlier train than usual so I can get across London in time to meet Rachel. Going through Bank at the time of day I'll get there is going to be hellish.

01/08/2019 [16:10] Not quite as early to bed last night as I would have liked, if I'm honest. I still got a reasonable night's sleep and woke just before the alarm this morning for an early start. I had a nice cycle into work with mostly deserted streets, and was able to get the patching and rebooting (and firmware updating) done without any real issues, which was great. That meant I was better able to concentrate on some oddnesses with the load balancer and firewall (separate problems) which I'm now able to say have both been sorted. One required SNAT to be turned on for a rule, and the load balancer thing required two steps of rule application and rule ordering before everything was as it was supposed to be. Getting them both sorted was good though, and has left me with time to work on banishing some warning lights from the monitoring system. Oh, and today is worth noting as the day I got VMWare's vSphere/vCenter working with Firefox and Flash again. Whether it's worth doing remains to be seen.

Otherwise, I think that's about it for today. I was in early, so I'm leaving early. Tomorrow's Friday, and there should be a bit more going on, even if it's after work rather than during.