BOFHJournal - November 2019

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

October's Journal
December's Journal

29/11/2019 [14:05] Today is a great day, because it is sunny, I didn't have to get to work early, I had a great shave, I got a spare laptop to take to Rachel's to replace the one we use for connecting to a monitor to watch television on, and nothing untoward happened at work this morning. Most importantly though, I'm leaving work now and heading to see Rachel for the weekend. We're going to have a lovely time (I hope), and there'll be fun tonight and tomorrow morning, and running (also fun, of course) on Sunday. Rachel's playing hockey on Saturday, so I'll be left to my own devices then (so good job I have a laptop handy). See you all on Monday.

28/11/2019 [17:00] Early start for something I thought might take somewhere between 40 minutes to an hour. In the end it took about ten minutes, and I even arrived a bit early (before 08:00) to give myself more time. It was a disk volume size increase on the SAN, and it went far faster than I could have imagined, even with the data tiered up to the fast disks (SSDs). So anyway, that went pretty straightforwardly, with just a few niggles that caused me to have to depart from the script I'd written yesterday. Everything duly went into a Confluence page so anyone (me) needing to do it in the future might have an even smoother time of things. After that I had a morning of doing many small, but important, things. Lunchtime was spent running in the very very light drizzle, doing 1 minute hill reps (10 of them), with a 3km jog to and from the hill. While I think they went pretty well, my watch is telling me that session on session my VO2_Max is getting lower and lower. It's as if I'm somehow detraining rather than getting 'fitter'. I don't know what's going on, or whether the calculations it's doing are just wrong for some reason. Either was it's a bit disheartening to see. This afternoon I've basically just been doing things for other people (no real change there), and keeping the place running. The announcement for this year's "does this person deserve more money" round just came out via email and for the first time since my first proper job I asked my line manager to put me forward for it. Turns out he already had, which was lovely of him. On that note, I think I'm going to go home. Tomorrow is a half day (hurrah) because I'm off to see Rachel for the weekend, and we're doing something nice on Friday night/Saturday morning. Oh, I forgot to mention, on the cycle in yesterday morning I stopped off at my favourite bike shop to beg some tools and a few zip ties so I could repair my front mudguard, which had shaken loose. Hurrah for them, too.

27/11/2019 [17:30] Another morning when I woke up feeling tired. It's not like I'm going to bed late or anything. I do wonder if my body's taken a very long time to extract its revenge for the ultra marathon. Either way, it only seems to manifest as a slow start to the day. Once I'm up and running everything seems to be fine. Or at least, once I've cycled for a few minutes. Once I got to work most of the day was quiet and uneventful. It was only after about 17:00 that things started to get weird, with a firewall issue rearing its head in a way that no-one left in the office(s) could deal with. We've put in a work around until tomorrow morning when the person most clued-up on the firewalls gets back in to work, but we're not even sure he'll know what's going on. I'm in early tomorrow morning anyway to do something complicated and dangerous with a disk volume (although I do have the safety net of a SAN snapshot before I get started), so I'll be able to talk to him before most people get in and need to start using the thing we've disabled. But anyway, it's late and it's time to go home and relax for the evening. Once I've navigated the roads, at any rate. The amount of poor/badly lit cyclists on the roads at the moment is distressing. Every night I try to leave the building aiming to be calm and unaffected by them, but it's quite difficult.

26/11/2019 [17:25] I woke up shattered this morning. I have no real explanation as to why. I struggled out of bed and did my morning stuff, which seemed to wake me up somewhat. The cycle in to work did a bit more, but I've spent the rest of the day being glad that nothing too exciting or interesting has gone on. At lunch I did my hardest track session to date, I think. 10x800m, with 400m of jogging between each one. I didn't feel as strong as I have for other fast/interval sessions like this previously. Maybe I'm still recovering from the 50 miler two weekends ago. It's not outside the realms of possibility. Still, I got it done, had a quick shower, and was back at my desk fairly quickly. In the afternoon there was some kerfuffle about shutting down the finance system over the Christmas break to minimise our vulnerability surface, but lots of other services complained that their interfaces to it would break and things would pile up waiting to connect to it again. I suggested we just put a block on the traffic manager at the point that all client connections came in through the firewall/NAT (given that all the other services should connect via internal APIs/interfaces) and people are thinking that's a great idea they had no idea about. I despair sometimes as to how people think our IT works. In any event, it'll most likely be the accepted solution, and things will go a lot more smoothly because of it, I hope. Otherwise, I can't really think of much else that's gone on today. Oh, other than foolishly assigning myself an early start for a disk LUN size increase on Thursday morning. However, it should (I hope) be fairly straightforward. I'll be doing a dry run with my team leader tomorrow to ensure everything goes according to plan. For now though, I'm heading home.

25/11/2019 [17:15] We've had a very busy weekend these past three days. Friday started with a train journey to Rachel's parents, and after spending some time with them, a pretty hard hill session in their village which left me far tireder later on than I was expecting. Or it might have been the alcohol (which I almost never drink). Either way I was shattered. On Saturday morning, after a bit, we got the train back to London and Rachel's place where she headed off almost immediately to play hockey and I relaxed at home for a bit. In the evening we spent some time together, which was just what we needed, I think. On Sunday morning, after a bit of faffage we did a very good (if a little too hilly, if that's possible) run together, fitting in a stop at one of Rachel's friends' houses to collect something, and then we headed into central London so I could continue on home and Rachel could meet some friends for lunch and a catch up. Back at home I did the Tesco trip and then settled in to wait for Rachel, who arrived late in the evening. We had dinner, relaxed a bit, finally, and then went to bed. This morning Rachel was up and away shortly before her train (which was cancelled, although she got a later one, thankfully), and I headed into work without lunch as I was planning on having it in a local pub with Shaun. Sadly, he couldn't make it, so I had to buy a sad sandwich from the building next door. I'll fill up on evening meal food tonight when I get home. Work-wise there was a bit of progress with the fibre channel issues I've been having. Nothing positive, unless you count finding out that the FC switch the server with issues is connected to is generating errors for all the things connected to it. My team leader isn't in until Wednesday, so there's not a huge amount I can do until he's in and we've discussed things a bit more. If I'm honest, I'm at a bit of a low ebb at the moment. Hopefully it's just being tired after the big run the weekend before last. With a bit of care and attention there's a good chance I can use this week and the coming enjoyable things planned for next weekend to perk myself up again. Rachel (and I) deserves me at my best.

21/11/2019 [17:15] I was at a conference in London yesterday. I can't say it was worth the day of, if I'm honest. It also meant getting up early and having to go home again rather than to Rachel's afterwards because of work I needed to get done this morning, which was annoying, too. On the 'plus' side, I did get to go for a run when I got home to counterbalance sitting down all day being fairly bored out of my skull. So yes, that was yesterday. This morning I slept in to the amazingly late hour of 07:30, and then got up and did some work from home before then dashing into work to start on finishing building seven new Drupal servers, and then six more servers for the HR system. On top of that, the firewall upgrade I needed to be in for went amazingly wrong because of some major-but-unconsidered (I think, nothing to do with me) changes to how NAT works. This caused pretty much a whole day of woe for the person who did the upgrade, as well as me and other people who had to field issues from people around the building. I think we're pretty much at the last few things that need fixing, but they won't get done until Monday at this point as I and the main person responsible are on holiday tomorrow. I'll be with Rachel as we travel to see her parents for a flying visit, then be back home on Sunday (although in my house or Rachel's I'm not sure). So yes, see you on Monday.

19/11/2019 [14:05] I had a shocking night's sleep last night. Obviously a lot of the reason was down to knowing I had to be up very early this morning (05:10), but I was also far too hot, which was probably partially down to still recovering from the weekend's exertions. Either way I needed a shower to get myself going this morning, even if I didn't do any morning exercises and that lack, plus the food choices I've made over the last week meant that when I stood on the scales it looked like all of my dietary changes over the last few months had been practically wiped out. Disappointing, to say the least. I'll have to start all over again as of this evening when I go to Tesco. Although not this evening, but as soon as I've finished this journal entry, on account of my getting in to work for 06:00 this morning. The short version is that both bits of work went off without a hitch, which was great. They also took a whole lot less time than they were expected to, too. So anyway, I'm done, and heading home. Tomorrow I'm down in London for the day at a Veritas forum/open day/thingy. Then I hope to get home in time to do an hour's run before the end of the day. Given how things are scheduled I may have to do it in the dark.

18/11/2019 [17:15] So I had something of a busy weekend. On Friday morning I got up and finished packing everything, then made my way to the train station further away from my house as I was going to need to cycle to work on the Monday (today). From there it was a train journey to Croydon, where I found Rachel at home, working away as she does on a Friday. She finished around about the time I discovered that there was no way to get her old laptop to power on any more. Luckily, the HDMI->VGA converter that I bought her many months ago chose to start working after previously not doing, so we were still able to watch things on the monitor, rather than just the smaller laptop screen. We had a nice dinner, and then headed to bed. Horribly early on Saturday morning a friend from Rachel's running club collected us and we headed off to Wendover Woods. To cut a very, very long story short, it took us just a shade under 13 hours to run five ten mile laps through the Woods, up and down some extremely steep slopes, and for twenty miles of it, in the pitch dark (with head torches). It certainly wasn't easy, and for a considerable amount of time I didn't enjoy it. But we did it, and did it together, which was what the whole thing was about. Afterwards we were very grateful for a quick ride to a nearby Premier Inn where were got very little sleep due to arriving at around 23:30, having extremely painful leg muscles, and the heating/bedding not really working out. Still, it was far better than going all the way home, and there was a nice breakfast in the morning. We hobbled back to Croydon via a taxi to Berkhampstead to see one of Rachel's friends for a coffee, and then spent the rest of the day collapsed on the sofa, whimpering every time we had to stand up. We walked out to get a takeaway for the evening meal, then went to bed at a moderately sensible time. Getting up, packed, onto the train, and to work this morning was a bit of a struggle, but I managed it. However, once at my desk I realised lots of things had gone wrong on Friday and I hadn't been around to fix them. Consequently they'd continued to go wrong over the whole weekend, too. Satellite had messed up for some reason, there'd been an OOM error and the OS had killed one of the databases (rather than something far less consequential). I managed to fix that, and upped the memory allocted to the VM to hopefully stop it happening again. The FC card I changed on Thursday morning last week errored again (but only once), so I'm hopeful that won't happen again. If it does then we have some other things to try. There have been numerous other firewall, traffic manager, and backup script issues, but I seem to have solved most of them. I'm in at 05:50 tomorrow morning, which is insane, but my team leader and I are done some very important work that has to be finished by 09:00, so we're trying to make sure that happens. Right now though I need to get home on my station bike, with a large rucksack, in the dark. Then go out again and get dinner and lunch for tomorrow... Unless I don't bother and do the week's shop tomorrow afternoon when I most certainly will be leaving early.

14/11/2019 [15:00] A poor night's sleep last night. Almost certainly because I knew I had to be up early this morning. Also it rained quite hard in the early hours and that might have woken me up, too. Anyway, I was up at 06:00 and out of the door well before 07:00. It was still raining (although not quite as hard by then, so I didn't get too wet), dark, and breezy, so not the loveliest of conditions for cycling. I made it to the data centre in good time, got the approval to power down the server, swapped the FC card, and booted it back up again. I was a little disconcerted to find that the server noticed that the FC card had been swapped (as expected) and that it was going to update the firmware on it to the same version as the old card (a bit of a shock). This added ten minutes or so to the boot time, but that was OK as my team leader (doing the SAN and fabric reconfiguration) wasn't in the office yet. To cut a long story short, everything worked, I came back to the office and learned about the bits of reconfiguration I didn't previously know how to do, and then got on with my morning. Mostly the rest of the day has been dealing with bits and bobs that've cropped up, going for an early run at lunch time (thankfully it was barely even spitting outside by then), and then coming back in time to sit in on a meeting I didn't really need to be in. I'm about to head off for another meeting, and then from there to get my hair cut and head home. I have the day off tomorrow, and will be spending the first half of the morning packing for the 50 mile run on Saturday, and the second half getting the train to Croydon (off peak). Rachel is working from home so we'll have lunch together, I'll relax for the afternoon, we'll have dinner together, and then get an early night for a very early pick up the next morning. We'll be driven to the race, attempt to complete it before the 15 hour cut-off (finishing in the dark with head torches), have hot soup, get a taxi to a local Premier Inn, collapse, shower, collapse again, sleep, try not to miss breakfast the following morning, maybe have lunch with Rachel's local friend, and then head back to Croydon again. I'll come to work from Croydon on Monday.

13/11/2019 [15:10] Mostly today has been about coordinating everything for tomorrow morning's early work. Whether or not it all goes according to plan remains to be seen, but there shouldn't be any risk of data loss, just extended down time while I move parts in and out of the chassis, do reboots, and hope that my team leader is doing the right things with the SAN and Brocade fabric management tools. Last night I made the decision to change both sets of disc brake pads on my bike in the dark (although I do have external lights on the house). Having watched an instructional video on YouTube before I went home it actually went surprisingly smoothly, being slowed down only by some really fiddly packaging on the new pads and A-spring, and the fact that the pistons needed to be pushed apart quite a lot before I could get the rotor back in with the new pads in place. Otherwise it wasn't too much of a trial. About the same time as to replace some old-school brake blocks, and with no adjustments to make sure they touched the braking surface afterwards. Or at least I hope not! Heading off to the dentist shortly, and then home for an early day. I may even get home in daylight, if a little lighter of wallet.

12/11/2019 [17:05] Not a lot going on today. I've been trying to coordinate some work happening (potentially) on Thursday morning, but one person is working from home and the other is ill and won't be back in until Thursday morning. This presents a bit of a problem. Hopefully we can get things sorted tomorrow such that I don't need much/any help on Thursday. We'll have to see what happens. In other news I've kept the place running, just about, managed to get my track session done without getting rained on (although it was cold and windy), and generally dealt with all the stuff that goes on in the day. I'm going to head home now via Cormac's work place to drop off a hard disk and then go and watch videos on how to replace the pads in my disc brakes. Hopefully it isn't too hard and I can get it done tomorrow morning before work.

11/11/2019 [16:50] A fairly fab weekend, all things considered. Rachel finally had enough trains behave that she was able to get to me on Friday evening without more than an hour's delay. It was wonderful to be able to welcome her into the house and have her just relax. Sadly, the next morning she had to leave the house early so she could get back to Croydon to play hockey. Fantastically, her team won their game, which was great. While she was out I managed to get the sink in the shower room draining properly for the first time in years. The plug hole and U-bend didn't seem that blocked, but cleaning them both seemed to help enormously. Hurrah for that. I made a half-hearted attempt to start clearing away rotten/dried-up/damaged silicone sealant, but didn't really get very far with removing it, never mind replacing it. I should, as it means when any washing up is done now (and Rachel did a lot this weekend) there's even less protection against water getting where it shouldn't around the sink and drainer. What I did do, with the help of some toothpaste, was minimise the visibility of some of the scratches in my black glass/ceramic hob top. It turns out that something weird from the web actually works. I may need to have a few more goes, but some of the scratches are practically gone, and the biggest, deepest one is a lot less noticable now. Other than that, I didn't actually do all that much on Saturday other than go to Tesco. Rachel got home again late in the evening and we had a rather delicious takeaway Indian meal before bed. On Sunday morning we were remarkably lazy, but still managed to get out of bed and had done a two hour run before 12:30. After a rather nice lunch my mind's a blank as to what we got up to, but then later on Rachel did some excellent planning of our winter holiday/personal training camp in Austria (travel by train), came up with another great idea, and then managed to fit in some jam-making and make dinner. I helped where I could, but she's a one-woman powerhouse, so for the most part I just stood back and cheered her on. This morning, as usual, she headed to London for the week's work and I cycled into the office and got on with the usual job of keeping my part of this place running. I had two meetings today. In both of them I felt somewhat out of my depth, and in one I was required to show people that I have very scruffy writing when it comes to working on a whiteboard, but for the most part I think I contributed reasonably well in both. A new fibre channel card has arrived for our main finance system DB server, but I don't think it's going to get installed before Thursday morning (early). Hopefully that'll be OK.

08/11/2019 [16:05] Rachel didn't come last night as it was going to be far too much effort, time on trains, and she'd arrive too late for us to have anything approaching a decent evening together. Definitely the right choice, I think. She is coming this evening though, which is wonderful. She'll be off again tomorrow morning-ish to go and play hockey and then won't be back again until later on in the evening, but that's just the way it has to be. It'll give me a chance to maybe redo the silicone grouting in the kitchen (or not, we'll see how the mood takes me). On Sunday I've got a 2 hour run to do at three different pacings, so we'll see how that goes, also. Work-wise today I splurged on a lie-in and didn't get out of bed until close to 07:30. Did the morning routine, and even shaved in preparation for Rachel being in close proximity tonight. At work I attempted a reboot of the server I'd made some GRUB changes to, but they didn't work, so I had to cycle over and 'rescue' it into a bootable state. Back to the drawing board on that, I think. In the meantime there's no way to choose a kernel at boot time on two servers (only by setting the default in /boot/grub/grub.conf before rebooting). Other than that I've fielded a lot of emails, got a FC card set to be delivered on Monday, begun to work out how to deal with some failed DIMMs in a very old server (disappointingly, the same DIMMs we have hanging around are perfect... except for being buffered rather than not), and generally kept this place running. But now, home time, and washing up, tidying, putting things away, and awaiting Rachel's arrival.

07/11/2019 [17:00] I didn't sleep too well again last night. My dinner was fine, I went to bed at a sensible time, but after about 03:00 I just couldn't stay asleep for any length of time. Again, I think it was mainly down to knowing I needed to get up early. When my alarm went off I duly did get up, do the morning thing, and get to work in time to do the patching I needed to. Everything went as expected, even the fix I put in, which turned out to work, but remove some functionality I now need to test putting back into place. I've arranged to reboot a slightly less important server tomorrow to see if the thing I've tried works. We'll see how that goes. Since then the rest of the day has been one of fixing small things, helping web developers not to trigger our web application firewall, and going for my hill reps session at lunch time. Now I'm off home in the dark similar to the dark I arrived in to get dinner from Tesco and await Rachel's wonderful arrival.

06/11/2019 [17:05] Nothing really happened today. I don't really know what to tell you. I did get some stuff done for me by other people, and opened support calls with Dell and Red Hat about some fibre channel issues that've cropped up, but nothing's really come of those as yet. I think I'll go home and get ready for tomorrow morning's early start for patching. Hopefully the fix I pre-emptively put in after Tuesday's issue will mean tomorrow goes smoothly.

05/11/2019 [17:05] In early for patching this morning, everything was going well. Then I rebooted the only physical server being patched (both OS, BIOS, and RAID firmware patches) and... it didn't boot. Just stopped before the GRUB boot menu would appear. By happy happenstance, after trying the usual trick of redoing certain settings in the BIOS and RAID BIOS, I immediately remembered that there was another server (of a similar, if not identical marque) that I'd recently noticed a different in the /boot/grub/grub.cfg with regard to serial and console lines. Hoping very much that this was this issue (after a quick bit of searching online) I tested out a USB boot disk to see if it would do a 'rescue' mode and cycled over to the server room after checking out the keys. A quick boot into 'rescue mode', a fiddle with the configuration file and thankfully the server booted. I even had it back up and doing its database duty with two minutes to spare before the vulnerability period ended at 09:00. I'm very glad I got in as early as I did after all. As soon as I got back to my desk I edited the same file on the server I'll be patching on Thursday morning which is exactly the same make and model, with the same patches being applied to it. Only in this case it's the production database server, so it being down for any longer than necessary would be A Bad Thing. I'm still not sure why the server this morning choose this reboot to finally decide to do what it did, considering there are other servers with similar configurations which haven't had the same issue (yet). In any event, I hope I've proactively solved the issue for Thursday morning. In other news I did my track session at lunch time. It seemed to go pretty well, in that it was hard and tiring, but I managed to do it, and fairly well, if the GPS information is anything to go by. So that's good. Otherwise the afternoon's work has gone by with me solving a few small issues and generally making sure everything continues to tick over. Oh, I forgot to mention that the morning after the patching was mainly taken up with getting my workstations onto a whole new network/VLAN to solve the intractable issue of bandwidth fall-off for anything outside of the VLAN. None of the network people were able to fix it, so we've gone with moving me to another VLAN, which seems to have solved the issue perfectly. This is both gratifying for all involved, and seriously annoying as we don't know why there was an issue, and why this solves it. Given we're trying to move towards a whole new networking architecture anyway this is a thing we'd have been doing anyway, but it still rankles that we don't know what's going on. However, all my downloads and web sites now work as intended, so I'm happy at least. Now though I'm off home to enjoy the leftovers from Rachel's cooking on Sunday. No fireworks for me tonight as Rachel won't be here after all and I really can't be bothered this year if she's not here.

04/11/2019 [17:00] Rachel was here for most of the weekend, so it was lovely in that regard. The weather wasn't too great, and she had to go to a hockey game and then had train issues coming back again, so that wasn't so good. Otherwise I got to do some useful errands, bought some clear silicone sealant to redo bits of the kitchen with at some point soon, and went for a run on Sunday morning (although I don't think it went as well as my coach seems to, after having seen my GPS upload of it). Other than that I got a bit concerned about the dampness in my shower room, about the way some of my kitchen/dining room/lounge floor bows, and probably some other things I really shouldn't be concerned about if I was being sensible. But that's me all over, really. Rachel attempted to get to Bristol/Bath by train this morning, but they were crap, so she had to abort the planned meeting she had with some donors there. It also meant she doesn't get to spend an evening with one of her old friends who lives there now, too. So that's a bit poo. I left the house feeling odd about the cycle in to work, so wore my cycle helmet for my commute for the first time in a long time. While nothing happened, I did get the overwhelming sensation that everyone on the road was angry or frustrated. Maybe I was simply projecting, but I intend to take a lot more care until that feeling goes away again. Not that I don't tend to take maximum care every time I get on my bike anyway.

01/11/2019 [16:05] The day started so well. I woke up with Rachel, after a lovely evening with her, and one of the better nights of sleep I've had in week. Not perfect, mind, but not bad. Then getting up and going to work was fine, although there were plenty of bad cyclists and car drivers around... And then it all went downhill. Various things went wrong, various things that require me to be in early in the mornings got scheduled, I had to talk a delivery driver in from a location he'd ended up in, and then go and install what he'd delivered (a power supply unit). That last thing actually went OK and the server's happy again. But then it, and another server, started showing all kinds of multipath errors that I haven't been able to solve, and the person who knows more than me isn't in for the whole of next week. Which means I have to hope that nothing goes 'breakingly' wrong for the next nine days. Basically, the day's been quite stressful and I want to go home. This weekend Rachel will be here and not here depending on what she has to do, and how much she feels like coming back after heading back to Croydon to play hockey tomorrow. I'm running on Sunday, and generally relaxing on Saturday, I think. There may be food shopping. Also reading and television. Hopefully much time spent with Rachel.