This has been a very, very long week. Its gone so fast that its almost a blur. On Tuesday the address lookup system we used expired, it has a built in self destruct if new files aren't loaded within a cut off period. It isn't normally a problem, simple insert the update CD and run a script. The only problem was nobody could locate the CD. No problem though the customer has a support contract so they can download the update, just one quick phonecall... The tech who called up received a recorded message telling him to a download site. I thought this sounded a little suspect, and I really thought so when the download speed was so slow, only attributable to the number of people downloading.
Along side the lack of automated address lookups strange errors started to be reported. With a greater and greater frequency a rather weird "Unexpected operating system error" kept appearing all over the place. More and more users reported the error, I couldn't really explain it and contacted so of my colleagues. They all suggested either hardware going wrong or corrupt data.
The problems escalated at pace, the first reports came in the morning when I got in at 9 but by 1 the system had ground to a halt. That meant the entire companies main CRM system stopped running during the middle of a busy day. I was sat at my desk surrounded by a group of staff and managers peering over my shoulder and asking when it was going to be fixed. Only I didn't have a clue what the problem is.
Actually I am doing myself a disservice, I did manage to locate the problem within a reasonable time, the main data file for customer enquiries was corrupted. I also managed to convince a colleague to get me a copy of the address lookup software. A reboot restore and data copy later and we were able to get users back onto the system.
Unfortunately the backup system was broken, so after this frantic day I had to go home, then log in after hours and copy back the live system to the backup. It took until 3am, I was like a ghost coaxing the copy.
I got the morning off to recover, I even managed to go swimming. Unfortunately the problems well still reoccurring, though they were more isolated. I spent the afternoon looking into it on the phone to various people.
It bothered me all night, I went round to Martinas to watch a film. I was so tired i found it really hard to concentrate on the film. To be honest I didn't enjoy the film much at all. It was called a cock and bull story its based on some famous book. Martina and Annika seemed to enjoy it having read the book at university. I found it to knowing and too much packed full of the usual lovees of the British comedy scene.
Today I really hunkered down on the errors, and finally found another data file which had corruption for a number of cases. I felt really good about locating and finding a solution to the problem even though it meant more out of hours work.
I felt much better about going out for lunch having a solution under my belt. I met Sarah at the Slug and lettuce. We had a nice chat about houses
I stayed up tonight cleansing the data of all the bad records. I still have no real idea what caused the initial problem, it seems to have been the names software maybe corrupting memory space, but I cant prove it. I suppose unless it happens again I will never know, maybe that is best.
I got called into see one of the managers, he started by complimenting the way I handled the problems on Tuesday. It was nice to hear, though the overarching message was one of we have more work to do and we need you to concentrate to get it done. One moment was very pleasing he questioned the wisdom of putting me in the middle of the room, whether it was leading to me getting distracted.
I got a valentines card which was somewhat surprising to be honest. In fact I think its only the second one I have ever received. The thing is, the postal stamp is south east, I don't know anyone in the South East. Well at least no single women, at least that I can think of.
No comments:
Post a Comment