BOFHJournal - July 2005

1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024

June's Journal
August's Journal

29/07/2005 [16:55] Ooops forgot to do an entry for today. Well, nothing happened except that I went for a run and nearly killed myself and put a lot of other things right again after yesterday, which is excellent! Now to go home.

28/07/2005 [17:40] What with one thing and another I've not achieved a whole lot today work-wise. The morning passed in a haze of UPs communication errors (my fault) and tnsnames.ora problems. The first of these issues was down to an out of date install package with SSL security still enabled being installed by our jumpstart process. That was probably my fault as I'd not told the person wot does the installs where to get the new one. The second thing was brought to light by one of the developers who was using the recently installed Oracle 10g client I'd stuck on one of those machines I keep talking about. Seems he had a Perl application which needed DBI::Oracle. Attempting to use it before it was installed he found that not only could he not get to the machine, Oracle denied all knowledge of it anyway. Turns out the tnsnames.ora file was way way out of date. Fixing that, getting the firewall opened and actually installing the correct Perl module took a little while but all seems fine now. This afternoon I treated myself to an icecream from the van which comes round because frankly I could do with the sugar buzz. It didn't work, being very much of the Mr Whippy school of ingredients (mostly air). I hate email, that is all. I can see things coming down around my ears unless technology improves. I'm quite tired trying to keep things together, doing my increased exercise stuff and holding down a job that's by turns completely dull and requires me to write reams of documentation or is completely insane-busy-all-hands-to-the-pump. I need a break, or at least some more fun to counterbalance the frustration at work.

27/07/2005 [11:00] Well, I did an unaided install of the new software this morning in pretty much thirty-seven minutes. It went flawlessly. I'm still unsure as to whether this is a good thing or not. I remain poised for the other shoe dropping. My guess is that it'll be next year when we roll out and go live and use a completely different version of the applications which are probably far less tested. Not only that we'll be splitting the system into DB, application and web front end, rather than having them all on the same machine. That'll make things a bit more exciting. That and needing to use mod_jk and tomcat. Thus far though everything has been pretty great and I'm pleased. The consultant has been helpful and allowed us enough rope to explore for ourselves, the documentation has been very complete and even mentions when we may get warnings and/or errors and the software seems to be fairly straightforward and doesn't have an interface which sucks utterly. I don't know how I'm managing to stay sane. A colleague's gone to power down and bring back an old E250 and D1000 which are buried deep in one of our Dell racks. I'm not sure if he's going to be able to get them out without damaging himself, the rack, the kit or the other stuff in there. Still, he's rebooted a machine for me to which froze last night during a regular update cycle. Oh, and I've sorted a backup issue on one of my servers as well. [17:25] Today seems to have been an extremely good day for all kinds of reasons. There's been some fun and laughter online, a failing Samba update thingy has been solved with some excellent firewall diagnosis, more progress on my running (wasn't nearly so tired today afterwards, probably because it was cold enough to see my breath (in July, in the middle of the day!)) and a couple of bits of documentation if not knocked on the head, then severely brushed with something heavy. Of course we have a new Samba problem that I currently have no idea how to fix. One of our linux boxes has been connecting to a pair of Windows shares for years now. Recently I built up another box to replace it and got around to mapping the shares as mount points. When you go to certain directories and issue an 'ls' command you get nothing back, not even the '.' or '..' files. Repeated execution of 'ls' may sometimes give an incomplete files listing. When I look at dmesg it contains a load of these errors: "smb_proc_readdir_long: name=, result=-2, rcls=1, err=123" which I'm pretty sure is to do with long file names and lots of files in the same directory. Only until someone tells me differently nothing's changed in those directories in months. This happens with RH 9 and RHEL 3 with Samba 3.0.14a and 3.0.9 respectively. Bit stuck for ideas at the moment. Google seems pretty unforthcoming. Other than that little blot on the landscape things are going swimmingly and I'm looking forward to things happening in the future which may be extremely fun! No climbing this evening as our driver appears to have pulled a muscle or trapped a nerve or something. Probably a good idea that I don't climb given I'm running 10km every other day now and doing a reasonable weights session every morning.

26/07/2005 [17:25] Things have been far better today. Aside from being very tired this morning and only doing a shorter workout before showering (although as recommended by Elaine I've added oblique crunches back into the routine to try and get rid of my handles. Ouch) I got in in good time and was able to do all the things I needed to before starting on watching the remainder of the testing for the software we're test installing for the new HR/payroll system. What was cool is that it all seemed to go off without a hitch after the DBA I'm working with had corrected some accidental typos in the tnsnames.ora files. That went on for most of the morning with some exhaustive testing of the client side almost sending me to sleep at one point. Very much out of my zone of demarcation. We broke for lunch and wandered over to the place across the road for some refueling and ended up talking about motorcycles for most of the time. I'm not sure the consultant was really a petrolhead but he seemed to get into it. I'm not either to be honest but the two DBAs who came along are very much so so it was fun. The afternoon's been spent watching the DBA do an install of an entire instance of the software with me riding shotgun (watching for typos/brainos) and the consultant making comments if appropriate. Everything went without a hitch, which in itself is a little strange. We ran through everything including patching and upgrading a remote thick client. All seemed fine. I'm really waiting for the other shoe to drop to be honest. The instructions are complete, they list common problems, even show when you should expect to see warnings and what they'll look like and and and and it all seems to work. I'm officially freaked out, but am not going to say it's all working until later in the year when we actually bin this version and jump to a completely new version that doesn't use a thick client but just uses a web browser for the client interface. At that point we're probably going to split the DB server from the application server from the web server (currently all on one (two (for testing different instances)) machine(s)). Hmm, hope you can parse those brackets correctly. Anyway, it's been a very interesting day as far as work is concerned, and there's been some progress on other fronts too, which is always good. I'm quite tired now so I'm going to go home and maybe upload some pictures of my brain...

25/07/2005 [17:40] What a day. I've been sitting (well standing) watching someone do an install of the software I've got to install myself tomorrow. It wasn't so much dire as I wanted to get in there, start doing things at twice the pace. In the end I just let it go as I was too tired to bother getting worked up. I ran at lunch time. That was hard too. Running alone and getting stitches isn't fun when you're tired. We completely failed to either get shorts or do gardening on the weekend. On Saturday we ended up going for a long walk in the countryside and on Sunday it rained so Elaine went out and got some wood for a shoe stand for the hall. She made it too, and only made one small mistake which she's going to fix shortly. I took some photos which came out reasonably well considering the subject. All in all it was a pretty good weekend for relaxing. Of course Elaine managed to end the weekend more tired than when she started due to some very very late (early) nights (morning). But it's not anything she can't handle. I need to go home now as I've been promised something and I don't want to miss out!

22/07/2005 [17:20] Definite progress on all scores today I think. Not only with the documentation but the networking I've been doing. The running was pretty good too at lunch time. I missed someone's leaving do but by all accounts his speech was a little odd and I didn't know him that well either. So, ce la vie. I've got to deal with his application running unattended from Monday too. If there are any problems I'm going to have to talk to someone else who only has a vague understanding of the code. Still, best foot forward and all that. This weekend I'll be gardening on one of the days and shopping for new shorts and running gear on the other. I really need some new ones and Elaine is beginning to tell me off for wearing ones with holes in. We'll probably be online a bit too, talking to people and playing around. All hail broadband! Depending on the weather this evening we're either going to a barbecue or get vaguely tipsy at home. Have to see what happens. Have a good weekend.

21/07/2005 [16:45] Still writing lots of documentation at the moment. It's not terribly inspiring but it is essential in the even that someone forgets how to do something with all the systems I've already set up here. It's actually useful in that it shows me where I've left holes in the things I've done that no-one else would think of, but I'd Just Do because it seemed obvious. Documentation is a fine art and I have to say that that having a wiki to to put it all on in some kind of semi-organised form is essential in a place this strangely organised. I took a break from it at lunch time to get my hair cut and head home for a shower and shave. I feel a lot better with less hair on my head. I don't think I'll get quite so hot tomorrow when we run again. Speaking of which, our friend who came on Wednesday is cycling lots on the weekend so he's not coming tomorrow which will be something of a relief as we may drop the pace a little and not end up dying when we arrive back. After I'd got cleaned up I headed into town to pay in some cheques and then went to look for the place I'm supposed to take the UPS engineer on Monday to park his car. Once I'd found that I had a quick (or what I thought would be a quick) trip to make sure my swipe card would work in all the doors it'd need to on Monday. Of course it didn't so I went to see the person I was supposed to. Naturally he wasn't in. After about half an hour of waiting I decided to swipe my card through a reader a few times and informed the receptionist of the time, my name and what readers the card needed to work for. Hopefully he'll look at the logs and do the neccessary. Turns out though that on my return to work that the new project I'm lead sysadmin for starts in earnest on Monday and I'm not going to be able to be anywhere but at work. This means I've got to hand off the UPS stuff to someone else. I don't mind, but the UPSes have kind of been my baby. It's good that someone else will get something of a handle on them too, in case I'm not around at some point when things need doing. It's getting close to home time here again and I'm actually full of energy for once. It's a great feeling and I hope to take advantage of it.

20/07/2005 [17:20] Have been arranging a visit from the UPS engineer to our two main APC UPSes for next Monday. The car parking around here is insane so it's going to be kind of a mess getting from one server room to the other. Still, has to be done. One of the UPSes has been erroring for months (perhaps years) but it's only since I got my hands on it that the emailing of warning has been getting to people and that's what's spurred them into action to book an engineer. We went for a run today with a new running mate. He's far, far fitter than I and finished the 6.4 mile loop we do easily two or three minutes ahead of me. It wasn't embarassing as I know he's far better than me and has run at least two marathons before. I think training with him will help me tremendously and I look forward to keeping up with him in a few weeks or months of running. Chatting online today has been fun and Elaine's been involved too. Had a whale of a time. I have to admit that I'm using MSN, although in my defence I am using a linux client over an SSH tunnel, so I'm not totally without technical credit. I'm going to attempt to climb this evening despite having completely worn myself out at lunch time. I certainly feel a lot fitter than I did two months ago, but I'm also a lot tireder in the evening. I think it helps me relax a bit more which is good for Elaine as she's working very hard in her job at the moment and needs the evenings and weekends to relax and recharge a bit. Evenings are for talking about the day and saying hi to people.

19/07/2005 [17:00] Not much today. More documentation. Managed to get the PC Support (Windows admin) people to get the updated MGE UPS software on their machines finally. It's only taken four months or so. Still, never mind. Kiting tonight.

18/07/2005 [17:20] Elaine went out with work colleagues on Friday night so I stayed home, chatted on line and drank my way through some Southern Comfort in comfort. Elaine came home late on and we had a very enjoyable time chatting to people while happily merry. I ended up needing to stay up in the toilet for a little while whereas Elaine went straight to bed. This somehow meant that in the morning I was fine and dandy and Elaine had to sleep for a lot longer and still felt a bit under the weather. We have a lovely day spending it doing nothing at all. Sunday was better as we spent the morning and early afternoon relaxing even more before Cormac and Steph came over to take us to their place to help clear some of the jungle that is their back garden. Armed with shears, a scythe and an axe, not to mention some very thick gloves we set forth into the tree-like brambles and managed to clear a way to the garden proper. Within two hours we'd made some serious progress and decided to stop and have a curry at the place three doors down. That was wonderful after the heat of the day and the work we'd been doing. Following that we just headed home and went to bed. This morning we were woken at about 04:52 by a neighbour across the road who decided to start up and drive away his diesel 4x4 while our window was open. I don't think I slept properly again afterwards. Today I've been doing some useful chatting to people and wondering how Elaine's part in the interview process went (she was testing the applicants' ADS knowledge). Documentation has been happening (as it always does at the moment) and I'm making some progress with Oracle client installations on some of the web developers servers. I don't think they're going to be finished any time soon. Especially as I'm trying to find two packages that're allegedly in RHEL 3's RPM list yet I can't find them anywhere. If anyone knows the whereabouts of mysql-jdbc and mysql-jdbc-tomcat as referenced in the release notes for U1 of RHEL 3 I'll owe them a pint or two. Hopefully tonight though I'll forget about my searching, wailing and gnashing of teeth and Elaine and I can chat about the day, say hi to some people online and relax some more too. The run today was good. I got my running partner to make it all the way around without stopping. All we need to do now is get the pace up a bit. I invited a few more people out on Wednesday including someone who's probably far fitter than I. I wonder how that's going to go.

15/07/2005 [17:05] All in all it's not been too bad a week. In the last few days I've made some real progress with something I've been working on with Elaine and am very hopeful we'll move it further over the next few days. Today though has been mainly about fighting with the current hairy dog that is the bespoke backups system. I've been adding five machines to the setup and hating every minute of it. If I hadn't been following the instructions which I painstakingly consigned to the twiki many moons ago I'd have shuffled off this mortal coil and gone to join the choir invisible. The sooner we move to Veritas' NetBackup the better. My running partner was off doing Cisco exams this morning so didn't make it back in time for the lunchtime run. This meant I was free to go at my own (slightly faster) pace and do the whole 6.5 mile circuit in one go. The heat is unrelenting on the back part of the loop but I managed it, again. Go me. I was intending to go to a party this weekend but owing to one thing and another I can't, so I'll be at home, doing the things I need to do, playing with my monitor and getting some reading done where possible. I'm looking forward to not running for two days and not doing weights on Sunday morning, but other than that this whole exercise thing is doing me a whole lot of good. My defintion's coming back, my lung capacity is going up again and I feel a lot better about myself and what I can do. All I have to do now is do the things I want to do. Everyone have a great weekend and stuff and I'll see you bright and early on Monday morning. Well I would if any of you actually looked at the web cam any more rather than just going to LiveJournal to read this. Come on, I'm not that ugly. Load it up and notice that it looks at the other side of my face in this job than in did in the last.

14/07/2005 [17:45] Most of the day has been spent trying not to melt. It's hot here, damned hot, real hot. My shorts you could cook things in them. OK, enough of the 'Good Morning Vietnam' quotes. Today I've been doing more documentation, discussing Orcallator and generally working very slowly towards arranging something of benefit for a few people. It's early days yet but could work out. Last night was great. Aside from getting home and having a great shower and cool down with Elaine we watched Mutiplicity (a film I've not seen before) with Cormac and drove home some chinese food from our current favourite place. Once that'd finished we watched another Family Guy from season four before shooing Cormac out of the door so we could get an early night for once. I was still tired in the morning. Must be the heat. We didn't go climbing, as you may have noticed. Turns out Cormac wasn't going anyway but the thing he was going to do got cancelled too. I don't think going for a Thai red curry for lunch was really the best idea. It wasn't terribly hot but by the time I'd finished I was certainly a lot warmer. Like I said, probably not the best idea. As things stand I should probably go home and play with my monitor a bit more before people turn up. I took some pictures this morning which're on the gallery if you're interested.

13/07/2005 [16:35] The monitor is absolutely beautiful when in its native resolution and being fed by a good DVI signal. Elaine was suitably in awe and is probably thinking of ways to bump me off so she can have it. We took my Iiyama CRT around to Elaine's friend's house (although Elaine had to go over again later with a D-SUB cable as he didn't actually have one). He's very pleased with his purchase but I'm holding off cashing the cheque for a few days in case he has a problem. I on the other hand have a minor problem in that many of my games aren't completely happy in widescreen and many don't actually go above 1600 x 1200. I think I can live with it though. I've been doing more documentation for most of the day. The heat is only vaguely tolerable here with the humidity really noticable when you leave the air-conditioned solace of the server room. Despite that I still went for the normal run at lunch time and came back about half a stone lighter (or so it seemed). Happily, the ice cream van was in the car park when we arrived back so I was bought an orange ice lolly type deal which really helped take some of the heat off. The rest of the day has been spent watching the absolutely torrential downpour which enveloped where I worked. From a few thunder and lightning bits and some fairly heavy rain to something that can only be described as a lake with tiny holes cut in it. Seriously I couldn't really see out of the window. That lasted for about ten minutes before vanishing as quickly as it arrived. Looks like I'll be able to get home without putting on waterproofs. Elaine doesn't have any with her today anyway so it's good things are dry again. We're off climbing this evening. Hopefully the rain will have taken some of the heat out of the day before we get there.

12/07/2005 [12:30] My new monitor has arrived! It is spankygorgeous. Unfortunately I don't have a machine with a decent enough graphics card (DVI out and with a CD-ROM to take the driver files to go up to 1920 x 1200) to try it out properly. I have powered it up and run it at 1280 x 1024 on D-SUB and verified that there is not one stuck pixel (as yet) so I'm very pleased. Of course I have to wait until this evening to see it in all its glory. I'll be dropping off the CRT this evening too with generous help from Cormac and his car. [17:30] I've been writing documentation all afternoon after having come back from lunch. Turns out that the UPS engineer who we were going to get to come out didn't have any good contact names so had to go away again. Shame, we'll have to have him back to show us what went wrong with our management card. It's a bit annoying at the moment because we don't dare plug it into the net again in case it goes wild and tries to shut everything down again. The documentation on the other hand has been going according to plan. I've printed out a first draft and left it on my line manager's desk for him to poke holes in. Other than that I'm going to wait for my (monitor's) ride home and play!

11/07/2005 [12/07/2005 09:30] Sorry, should have written this on the day but I was hellishly busy. For most of the day I was working on the finishing touches to my DR documentation and some other important stuff. Over lunch we went running in the really rather hot weather and managed a reasonable time without melting too much. However the real fun came at 16:08 when one of the linux mail servers went absolutely batshit crazy and began spouting a whole load of EXT3 errors. It looks like something went briefly wrong with the actual RAID controller but there's very little to go on. As it was I cycled over there as soon as I could but found that my swipe card wasn't registered to work in the doors there yet. I had to ring and wait for someone else to come along before I could get access to the machine. Happily (worryingly) it came up perfectly and didn't put a foot wrong while I was there. I'll do some more investigations later. The weekend was interesting. Getting dressed up for the wedding was a minor inconvenience for seeing all my friends together in the same place. It was very odd to see groups of friends I'd never expect to see together, together. Beside that a great time was had and I took some reasonably good photos which you can see in the relevant collection for your delectation and delight. Of course, there are plenty of pictures of Elaine in there too.

08/07/2005 [17:25] Hurrah. A productive day. Not only did I do the whole 6.5 mile loop without a pause (and wasn't dead by the end of it) but I've made real progress on my disaster recovery document too, including picking apart the various places databases and standby databases have ended up inside our setup. I've got scripts in place that're almost finished, the rsync knows about them and all I need to do is finished the scripts, repackage the scripts directory tarball for potential future reinstallations and I should be done. I also have to start collecting Dell service tag ids so we can work out which machines are worth getting extended support on. Actually I have a whole pile of documentation to write which is going to take an age. The DR one is the most important though. I think I mentioned that it's my ex's wedding tomorrow which I'm attending. It's going to be great to see people I haven't seen in a long time in some cases. I'll probably take some photos here and there. For now though, it's late on a Friday night (work-wise) so I'm off home.

07/07/2005 [17:10] I won't even bother with today's London news, you all know about it from reputable sources. I on the other hand have been spending time helping other sysadmins who use our server room for co-location to rack a few servers, take away our old and spare D1000s and generally tidy up the office a bit. We had an Institution-wide Solaris 10 workshop thingy which turned out to be a reasonable lunch and then a disorganised question and answer session which turned up some useful things but wasn't overly great as the use of a lunch time. Especially so as I didn't get to go running. Still, probably a good idea for the left achillies tendon, etc.

06/07/2005 [17:10] Argh! Why do things have to be so complicated when it comes to, y'know, Stuff. If it wasn't bad enough that I appear to have forgotten how to write useful documentation, I'm beginning to hate the tnsnames.ora file and just how many database servers/instances we have to juggle. Sometimes I'm very glad I'm not a DBA and just a sensible sysadmin with other fish to fry. Of course it pays to know a little about both. Today has been odd as I didn't run. My left achillies tendon has been giving me a little gyp recently so I think I need to perhaps not run every day for a little while until it feels a bit better. Either way I'm climbing this evening. That coupled with the exercises this morning which I hope to start doing again every morning should help me get back to where I was two or so years ago fitness wise. Work achieved today: minimal, chat had: lots.

05/07/2005 [17:30] So I get in this morning and there's been some real problems with the UPS and the SAN. Turns out that around midday yesterday the network card in one of the UPSes (the one where I work most of the time) decided that there was only one minute of battery time left and therefore all the clients would need to be shut down. This they duly started to do, all twenty five of them. Why the Windows boxes didn't I don't know, but that's something else to think about. Anyway, the people who were here were able to stop the key most essential boxes from shutting down, which was a good thing. Unfortunately twenty other boxes powered down (cleanly). It took a fair while to sort everything out, but there was no data lost so that's OK. I was called to find out the UPS username and password but in the end they just yanked the network cable to stop it doing anything else. There's nothing wrong with the UPS or the battery so far as we can see. Very odd. The SAN just threw another disk. Standard Seagate predicted spindle failure (again). The Sun engineer is here again replacing that as well as connecting up another complete array to do copying over to get us off all the Seagates and incidently upgrade us to faster spindles anyway. I've been editing tnsnames.ora files and working on the depths of the disaster rcovery document, which is turning out to be a little more complicated because of the use of Delphi applications with hardcoded database references. I'm sure it's all solvable though. With time. For now I head off in the rain to meet my slightly damp Elaine (who didn't take a waterproof to work today) before heading out for food and cinema (Batman Begins).

04/07/2005 [05/07/2005 - 15:55] Took the day off. Having spent the weekend either relaxing or at a party with lots of fun people it was nice to be able to sleep in on a Monday morning. Once up and about though we headed into town to try and get some clothes together for this wedding thing next weekend. Once we'd got Elaine something she cycled home to make sure it worked with the shoes she wants to wear. I went trouser shopping. Of course I didn't buy anything until Elaine was back. A second pair of eyes really does help sometimes. Of course once we'd got my last bit of apparel Elaine needed shoes and accessories so I went to have a sit down in Borders with a book (Nova, Samuel Delany) for a while until she found me and we headed home. During the day I got a call from work. Seems without me there things went to hell very quickly. Not through my technical know-how but because when I'm around things break less. More on this tomorrow.

01/07/2005 [18:05] Again I'm very tired today. First thing this morning I and a couple of others transported an old but still good Sun E3500 and associated A1000 to one of the other machine rooms and got it up and running again. Six CPUs, two or four gigs of RAM. Better than the machine it's replacing anyway. As a bonus it sits the console server's console on it far better than the E250 it's replacing. It's all good. I didn't go running today because my left achillies tendon has been giving me a bit of gyp. I wasn't too put out as my running partner cried off before me. It's going to be odd next week running alone as he's on holiday. I'll have to see how it goes. Maybe I'll get the whole route done in one go without a pause. It's a shame I didn't go really because I'd put my shorts around the back of the SAN cabinet to dry them out especially. They weren't more than damp (no danger of them dripping on anything) but it was good to get them properly dry. I'm sure it's not a proper use for a machine room, but it won't have done any harm considering the size of the place. This afternoon I buckled down and got stuck into the disaster recovery document I'm supposed to be writing. It's pretty simple; three cases, five applications to deal with. I just have to ensure that the more complex cases (those that use tns_names.ora rather than configuration files) are dealt with. For the moment though (and for Monday, as I'm holiday) I will head off and wish you a happy and relaxing weekend. I'm off to a barbecue this Saturday and probably clothes shopping for the wedding on Monday.