BOFHJournal - March 2020

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

February's Journal
April's Journal

27/03/2020 [18:15] Sorry I haven't posted in a while. Honestly, working from home, and just the sheer depressive nature of this whole nationwide and global situation has a way of getting to me (and us all to some degree I'd have thought). Anyway here I am, sitting at my dining room table, as I have been for the last week, working away on a laptop with an external monitor for a bit more screen real estate, and a full-size keyboard and mouse. I've achieved quite a bit this week in terms of work stuff. Both Tuesday and Thursday mornings were early starts where I was doing complicated stuff to ensure that various services didn't run out of disk space. The tail end of last week I went back into the office twice to help set up additional VPN capacity, which included going out to server rooms to rack things up, as well. Since Monday morning, though, I've been home. I can't remember which morning it was - it was certainly at least a day before Rachel arrived, so it might have been last Friday - I couldn't sleep, so I made my way to Tesco for it's 06:00 opening and managed to get two of a few useful things, as well as some bread and other essentials. Once Rachel arrived she sensibly did some meal-planning and we've been fine since. This hasn't stopped me from worrying occasionally about us going out again to restock and there not being enough/any of what we need, but if I think about it rationally, there'll be enough of something, and we'll make meals with that. We're not going to go hungry, just potentially have to eat differently to in normal times. I did dig the old bread maker out of the storage space under the stairs and fire it up a few days ago. It made some crunching/grinding sounds, but produced a damned fine loaf with the flour I'd managed to get. We ate that in two days. This morning I attempted a second loaf and something plastic and obviously now-brittle broke utterly inside and the thing stopped kneading, making only gruesome crunching sounds instead. Luckily, this was part way through the 'second knead' so with its dying energy a second, perfect loaf, was created. We now have to rely on pre-made bread, or being able to get hold of more flour and having the actual real joy of making bread by hand. I might see if Tesco has a larger bread tin when we do finally go to Tesco, brave the queue (and chances of infection), and see what's still in stock. Rachel's obviously living here at the moment given the lock down. I know she'd really like to go to her home for a while at some point. I can't blame her. While I really want here to be her home too, it's not quite Our Home, and I know she's missing her place in Croydon. We have actually considered cycling the fairly long way to her house, and I've even gone as far as purchasing new tyres and some custom-made maps from the Ordnance Survey to get us there. It's just that I don't think it would be quite within the spirit of the law to be outside for that long. And if we had problems it might cause more issues for the already-stretched public services. We'll see what the future brings. Otherwise life continues to continue. We both put in a full working day. Me at the kitchen table, Rachel up in one of the bedrooms (which we had to deep clean as something in there gave her a serious allergy flare-up). We see each other occasionally, and I take her a coffee now and then. She has what we think is a light cold at the moment, I'm currently feeling fine. I'll continue to write journal entries as and when I feel like it. Probably not every day, but at least a few times a week, I hope. Between now and the next one there's a weekend, which won't feel all that different to this week that's passed, except that I probably won't turn on the computer all that much. We're hoping to go to Tesco on Monday, but I'm not sure when. Until the next entry, stay safe.

18/03/2020 [17:20] Well, this is it. We've been told not to come to work tomorrow. Working from home starts then. We'll have to see how that goes. Microsoft Teams is going to be the way ahead. I've got a headset to plug into my laptop, a 22" monitor from work, and I've redirected my work phone to my mobile. Now we have to see. In other news I'm saddened and angered by the amount of panic buying going on in the supermarkets. I'm very glad they've enforced limits on purchasing, although I know everyone's going to get around it by doing repeated shops, or going as a family and checking out separately. It really gets my goat how people can't just buy what they need for the short term and trust the supermarkets to do what they've been doing for years. And finally, work: Not a lot going on, thankfully. I've managed to sort the alert system out by working on as many things as possible today, leaving only a few 'regular' alerts outstanding. As people use our systems less (and some far more, but I don't look after those). For now though, I'm heading off. I'll see if I can update this from home (chances are good), and keep you appraised of how I'm doing. I'll admit I'm beginning to be concerned about the amount of people forecast to die, even in the most optimistic projections. And also just how long this working from home/lockdown is going to be in effect. I guess we'll see...

17/03/2020 [16:35] I went to Tesco last night and found many shelves empty. This was extremely galling given there's absolutely no need for panic buying of any kind. Luckily I was able to buy everything I normally would other than baking potatoes, washing up liquid, and the kind of apples I wanted (I got another kind). I went home and tried not to be frustrated by the stupidity of the masses. This morning, as I was due to be in work very early I went via Tesco for the potatoes and washing up liquid. It was amazing how busy the place was. Some shelves were already looking sparse again after a delivery last night at apparently 23:30 or so. I was told it'd been busy all through the night. Madness. Anyway, with what I needed and no more I cycled to the server room I needed to be at, coordinated with my colleague who'd shut down the service the server supported (as well as the server itself), and then headed in and swapped the FC card for the one I had with me. After powering the server back on I handed it back over to him and headed back to the office to do the disk/partition resize operation that was also on the list. Disappointingly, the process I'd painstakingly worked out didn't work on the server because the version of 'parted' installed on RHEL6 lacked the command that the version on RHEL7 (where I'd done all my testing) had. I tested another procedure on a low-priority server... and promptly broke it so badly I had to apologise to the developers who used it and rebuild it from scratch. Luckily it was an extremely unimportant server, so no real harm was done. Rebuilding it, and then using it to test the revised procedure for what I needed to do took up most of the morning and lunch time, so I didn't get to go for a run. At this point I probably won't go when I get home from work as I'm feeling just a little tired by this end of the day. I do have a working procedure now, and it should even deal with a spanner that could get thrown into the works by the process, so that's good. The work has been postponed to next Tuesday (and Thursday), but it's likely I'll do the work from home anyway given how things are going with the country-wide lockdown. I'm heading home now, where I may go for a short run, or just do the washing up and try not to get angry at the masses for panic buying. Cormac'll be over at some point with the monitor I've taken from work, so I may spend some time setting up a 'home office' somewhere. I will be in work tomorrow though, unless I'm told not to come in. In other news we're trying to embrace Microsoft Teams here. I'm not sure how well it's going. Especially as I feel like everyone who's working from home isn't quite doing as much as I am here in the office. I wish people would make a bit more effort to close a few helpdesk tickets.

16/03/2020 [17:05] I think currently, more than anything, what's getting to me about this virus situation is the panic-buying and the effect it's having on people (including the elderly). There's simply no reason for it. I'm going home after being away for half the weekend, and then to Tesco and I fully expect not to be able to buy everything I'd like to. And there's simply no reason for it. But anyway. Over the weekend Rachel and I were at my house until Saturday morning, after finding out both of our spring marathons were cancelled on Friday. So, rather than me do a 20 mile run on Saturday and follow Rachel to Croydon later on, we went together early on. I had to cycle to the main train station so my bike'd be in the right place for this morning, but otherwise all went well. Once in Croydon Rachel went to play hockey and I stayed home. After she came home again (after another loss, sadly) we had a lovely evening together. On Sunday Rachel ran in a local race and came first woman, which was amazing (also, I don't think I could have kept up with her), and I went for a half marathon distance run along the Vanguard Way. Sadly, it was a bit wet and muddy in places, but it's not like I need to keep my running shoes pristine any more. I'll probably have a new pair by the time races are happening again. Once we were both home, clean and warm, we had lunch, and then settled in for the rest of the day. This morning everything went as planned with regard to trains and things and I got to work to find that most of the office wasn't coming in, but was working from home for the day, or potentially for the forseeable future. A bit disappointing, but definitely for the best. I'm going to continue to come in for the time being as as more people work from home the safer it's actually going to be to be in the office. I'm in very early tomorrow morning to swap over an FC card and then make sure other stuff that other people are doing for me has happened correctly. More on that tomorrow. For now I'm off to brave Tesco and see what food is left for me to eat this week.

13/03/2020 [16:15] Rachel arrived last night, a little later than I was expecting, so I got a tad concerned until she walked in the back door. We had a lovely evening though, and even though I didn't sleep all that well it was wonderful to have her there. I left her in bed with a coffee this morning and cycled to work via a 'new patient' appointment at my new surgery. Nothing out of the ordinary there, other than them only letting people in the door after a conversation about the reason for my visit through the intercom. Virus minimisation, I guess. Anyway, after that I headed to work and have had a reasonable day of swapping in a replacement HDD, claiming a monitor to take home for home working, and generally keeping the place running. I'm about to head home now though, for a weekend of running, traveling to Croydon after my long run on Saturday, following Rachel who'll have gone much earlier for a hockey game, and then potentially cheering her on when she does a half marathon on the Sunday (a proper rest day for me after three days of running in a row). If the world hasn't ended, see you on Monday.

12/03/2020 [17:15] Our hardware vendor blinked first and sent us the FC card we asked for. Which is nice. On top of that I've requested a replacement HDD from our 3rd party support company for a disk which has gone in a RAID set in another server. Other things which have occured today is my testing of RHEL 8.1 for the first time, deleting lots of old ISOs in the VMWare cluster and replacing them with more up-to-date ones, going for a run on the track at lunch time and having a strong headwind for half of every lap, which didn't do my times any good, and generally keeping this place going by spinning as many plates as possible just enough so they all stay up. For some reason I started falling asleep on the sofa before 21:00 last night, so I just went to bed. I did wake up fairly early (but not that early), but was still a little tired. Maybe another early night tonight would be a good idea. Especially as I'm running, then going to an initial health-check appointment at my new surgery (complete with urine sample) before work. Rachel also arrives this evening, which is reason for much rejoicing. Which reminds me that I should go home, via Tesco for dinner, and then do the washing up before she arrives.

11/03/2020 [17:15] Nothing much to report today. I took my 'new' rear mudguard to work and did the drilling modification and fitting of it at lunch time. The morning was spent doing a few odd jobs and talking with John (who was doing one of his "keep in touch" days today) about LetsEncrypt issues we'd been having. We think we've got to the bottom of it now (it boils down to not tidying up after ourselves enough). Otherwise there hasn't been a lot on my plate, for once. I got two of the four new workstations I prepared out of the door, did some post-install configuration on two of them, and bothered a few developers about getting things moved off servers which should have been retired years ago. All that, and washing my hands a lot and eating too many doughnuts and cookies (2 and 1 respectively). I guess that's everything for today.

10/03/2020 [17:40] I achieved things today. Not necessarily the things I had planned to do, but other things. And I achieved them. Given the state of the organisation at the moment (and the world in general) I consider this a victory. I also got a reasonably hard run done at lunch time too, so there's that. Lots going on, managing to stay on top of it for the most part. Hopefully somewhat more will get done tomorrow, including sending a rocket up one of our hardware suppliers as they're refusing to replace an FC card with another brand as we think that'll fix the problem we're having. Instead I've got a crapload of work to do next Tuesday morning which will mean more downtime without (as far as I'm concerned) fixing the issue. So we'll need another period of downtime after that for when they actually agree with my technical assessment and send me what I need. But anyway, other than that I replaced my front mudguard on my bike this morning, and still need to drill a hole in the replacement rear one (both broke a while back and were being held together with zip ties) so that it fits my weird bike. However, now it's home time.

09/03/2020 [18:10] A fairly handy weekend, if I say so myself. I got a decent run done on Saturday, even though I felt a bit tired and not-keen. On top of that was washing up, tidying the house, doing a tiny bit of bike maintenance (more still to do), going to Tesco, and collecting Rachel and my race numbers for the Sunday race. Rachel arrived on Saturday evening and there was, as usual, much rejoicing. We had dinner, and an early bed. Sunday morning we cycled to the boat house to use it as a base for the race. That meant no queue for the loo (at least until we got onto the race site), and showers afterwards. The race itself was a little harder than I'd been hoping for considering the amount of training I'd put in. My coach told me not to go too quickly and aim for the time I would hope to do half of my target marathon in. I think I pretty much achieved that aim, coming in about three minutes behind Rachel, who'd been going for it, but I can't say it was a completely easy run that I would have been able to do two of back to back. We'll have to see what the next three weeks bring... including whether or not the marathon's cancelled if COVID-19 makes a bigger showing in this country. After the race we showered, and then headed home for a well-deserved lunch. The rest of the day was taken pretty easily, and we sloped off to bed at a moderately sensible time given it was a school night. Rachel left for the train station in the morning and I headed to work not too long afterwards so I could start a large V2V process such that it'd finish in time for me to stay at work, but not be here all evening. That finished about half an hour ago, but I remained a little longer trying to work out how to install RHEL7 Workstation on some new desktop boxes which only boot internal storage in UEFI mode. Luckily, my installer USB stick also speaks UEFI or I'd have been in a bit of a pickle. I've got one box of four done, but the others shouldn't take too long tomorrow, although I probably won't do them until the afternoon as I'm going to attempt to start another V2V in the morning, and then take the new guy over to watch me fumble around and do an OS install on a new physical server at one of the server rooms before lunch (when I'll be running a considerable amount of time at target race pace). Anyway, it's definitely home time now.

06/03/2020 [16:15] A slightly less frenetic day today, thank goodness. After a reasonable night's sleep I even lay in bed until 07:30, which is unheard of most days. After I got into work I went straight into a meeting, then a demo of some stuff, and only after that got down to reading my email and then taking the new hire through triaging the status system. Since then the day has mainly been about keeping things running, having a sudden monsterous urge for a Twix and taking that opportunity to go and get a bit of spring sunshine while I ate it, and getting someone to create a new VLAN/network/infrastructure so that I could create three new VMs which might mean we're a step closer to getting rid of nine RHEL5 servers (seven of them virtual guests) which should have been consigned to the scrap heap more than three years ago both for hardware and OS support reasons. Now though I'm going to go home early, which is even better than on time, and much better than my habitual late. This weekend Rachel arrives from working from her parents' home today on Saturday evening after I've done a run in the morning and then been to Tesco. We relax in the evening, then we have a half marathon on Sunday morning which I hope the weather will be reasonable for. I've been asked by my coach to go no faster than (full) marathon pace as a test for my A-race of the season. I hope I can manage to do it! The remainder of Sunday should be nice and relaxed, with hot soup, nice bread, and possibly an Indian takeaway in the evening.

05/03/2020 [17:40] The day's been so busy I can't actually remember everything that's happened. I haven't even uploaded my lunchtime run (track session, 8x800m) at time of writing this. It started early, with me coming in well before 08:00 to reboot our main finance database server as it'd lost half of its multipath connections to the SAN (via fibre channel). Sadly, although it came back cleanly and all the paths came back, almost immediately and throughout the day until we disabled the port on the FC switch it went back to erroring the same port as before, which potentially also caused slowdowns in the finance application. I've asked the hardware vendor to send us an Emulex card to replace the QLogic one we've got. We've already replaced the card once, the FC cable twice, the SFP at the switch end, and moved ports within the switch chassis. None of them have helped. It's only this port on this card on this switch. The other port on the card's connected to an identical switch and has no issues. We have other servers with the same card connected to the same switch(es) and they don't have any issues at all. It's very odd. On top of all of that have been a large variety of other issues. Two of my colleagues are graciously taking work off my hands, but they're not doing it in the same way I would (given my OCD), and as such I'm finding myself concerned about the problems they're causing. Nothing big, you understand, just things that could have been avoided if they had my obsessive attention to detail and doing things Just So. It probably doesn't matter, but it does. Anyway, I'm tired, so I'm going home.

04/03/2020 [18:00] I don't know how it got so late today, but here I am still at work. It's disappointing as I was awake until gone midnight last night making sure that a server with a 2TB disk finished converting from a KVM guest to a VMWare guest, and then booting up properly and starting all its services correctly. That and I was coughing a lot again because of my nose. Anyway, today has been a reasonable day, although I wish I was going home to find Rachel there (I won't, she's got a lot on). I ran at lunch time, coordinated other people doing their thing, and tried to hand off as much as possible to other people in my team because I'm just doing a little too much at the moment. It's just a bit frustrating when they don't fix things they've broken straight away, or make them worse. I'm going home now to hopefully relax and get an early night as I'm rebooting an important server tomorrow morning at 08:10 and I want to be a fit state to not mess it up. Hopefully this action will restore all of the half of the multipath connections which failed a few days ago. That sentence does make sense, if you understand the technology, honest.

03/03/2020 [17:35] A lot of tasks to do today, but none of them huge. I've been busy, but flitting from one thing to another more than anything else. Had a discussion with one of the network security.topology people about a new zone for servers, showed the new guy some more about how we do things here (but not as much as I would have liked), went for a 1 hour run at lunch time which was nice given that the dealt-with blisters on my feet seemed to be OK, and my legs weren't too unhappy about running after only a day's rest following a marathon distance. I'm probably still not drinking enough, but I can do better in the coming days. I replaced a HDD in a server hosting two KVM guests this morning, only to find that the server had a punctured RAID stripe (something I'd been idly thinking about recently). As a result I'm now in the process of migrating both guests (one at a time) off the host server and into our VMWare cluster. There's a hard limit on creating disks larger than 2TB (the original disk size is 3TB), so I had to have a conversation about that with the people who administer the guest in question, and they're OK with 2TB. I've set that V2V conversion going now, and it should be finished some time around 23:30 tonight. I might stay up and ensure it does come back, and then get everything running on it again. Otherwise I could do it first thing in the morning from home, too. I'm a little concerned that my workstation will reboot overnight and break the V2V process, which would be extremely annoying. I've done everything I can to stop that happening, but we'll see if Windows has been outsmarted or not later on tonight/tomorrow morning.

02/03/2020 [17:40] So I made a big mistake with my hydration for the marathon distance run we did on Sunday. Even though it was cold and we weren't running all that hard, by 24km my urine was extremely dark (really, pretty much watery coffee-coloured) and I was probably on the way to having problems. Luckily I was able to get some fluids inside of me and I never actually felt in any way bad (other than being tired because of the time on feet, etc.). It's something I haven't had happen since the very, very early days of doing adventure races, and I'm pretty peeved with myself for letting it happen again. It won't happen a third time. Other than that though the run was... OK, I guess. Nothing amazing, and I had been hoping to feel a little better and run a little faster, but we certainly didn't need to by any means. We probably would have if I hadn't held back once I realised my hydration situation. Anyway, once we'd finished and cycled home again we had a lovely afternoon and evening of relaxation at home, thoroughly deserved. Working backwards, Saturday was pretty good, other than me waking early with coughing again, but then sort of lazing around in bed until gone 09:00. I finally got up, pottered around a bit, and then did my run, which went pretty well all things considered. Shower, lunch, Tesco, housework, and then Rachel arrived, which was brilliant. I guess that's all the major points from the weekend. This morning we seriously considered taking a sickie/working from home, but Rachel had to go to work for a face-to-face meeting, so I headed in myself too. I've shown my mentoree some more of what we do here (he doesn't seem too horrified or amused as yet, so maybe we're doing OK), done some useful work, and nursed along a few things I still haven't been able to fix yet. Hopefully one thing'll be fixed tomorrow with a replacement part, and another on Thursday with an early start for an OS reboot. We'll see.