View Full Version : 10-hour Downtime


Taluntain
Thu, 27th Sep '07, 12:37pm
We've been down for the past 10 hours or so (SP and most of our hosted sites), for reasons yet undetermined. I'm looking into it to see if I can figure out why.

The Magister
Thu, 27th Sep '07, 5:00pm
I wondered about that. The internet is a queer beast indeed :borg:

Taluntain
Thu, 27th Sep '07, 5:40pm
Oh, the Internet's got nothing to do with it. What happened was on our server. ;)

Taluntain
Fri, 28th Sep '07, 12:41am
The server (and the site along with it) will be going down for a diagnostic in about 4.5 hours from now (5AM GMT+1).

A rough estimate of the extent of this planned downtime is up to 4 hours, though normally this takes much less time than that.

Sorry for the inconvenience - hopefully everything will turn out fine.

chevalier
Fri, 28th Sep '07, 1:27am
Thanks for picking the least problematic hours you probably could. ;)

Taluntain
Fri, 28th Sep '07, 12:49pm
Yea, well, the diagnostic took much longer than expected.

And unfortunately, the results are the worst possible. The server hard drive is bad and will need to be replaced. For the time being it was put back, but we need to replace it as soon as possible.

Which means going through an identical process as 2 months ago when we moved to this server, only this time we won't have the benefit of setting most things up on the new server before hand, but will need to do it all on the fly, since this will all happen on the same server.

This, of course, comes at the worst possible time due to my work on the transition to the new message board software, but that's Murphy's.

I'll keep you all informed of the developments as things progress.

Kitrax
Fri, 28th Sep '07, 3:59pm
Thanks for picking the least problematic hours you probably could. Says you! The down time made my 2 hour break so boring that I actually did some homework! :eek:

@Tal - Wouldn't a RAID 5 setup ease the pain of such events? I know they can get expensive, but just think of the headache it would save. :rolling:

Nakia
Fri, 28th Sep '07, 4:28pm
Tal, I almost had a nervous breakdown last night. All I got was a 'run-time error'. Guess it was because the whole server was down.

First thing I did this morning at 8AM ET was log on to see if SP was back. Thank Freya it was.

I like Kitrax's idea. I could scrunch up a few bucks to help pay for it. How much would a RAID 5 cost?

Taluntain
Fri, 28th Sep '07, 5:28pm
It's really only the cost that prevents me from ordering at least a mirroring (RAID 1) setup. But mirroring alone would cost an extra $40 a month, which is an expense I can't justify, unfortunately. The board transition alone has already cost me about $400 thus far, and will probably be another $200 by the time it's final.

I don't intend to put up a call for donations to help with that yet though - not until it's finished and online, which given the current situation, will take a while longer than expected.

But no worries, the hard drive will be replaced for free (since it's failed due to a hardware malfunction), it's just that this means about a week of unplanned extra work in configuring the server all over again and transferring SP and all the hosted sites and reconfiguring them.

All in all, a small price to pay - the fact that the data is still intact is what matters most.

Kitrax
Sat, 29th Sep '07, 7:18am
Hey...sicne we're talking out the new boards...

Do you have an idea of how it's going to look? Even if you pay a web designer to try and match the current view of the boards, I'm sure there will be differences... :rolling:

Taluntain
Sat, 29th Sep '07, 1:50pm
The basic, unchanged look is somewhat similar to UBB's unmodified look (not that anyone here has ever seen that). :D

It probably won't be possile to completely match our current style, but I think I should be able to do a relatively good approximation without any external help. Our style here is pretty simple, after all.

Taluntain
Thu, 4th Oct '07, 8:23pm
If everything goes according to plan, we'll be moving to the new hard drive tomorrow. There shouldn't be much full downtime, but the boards will be inaccessible somewhat longer than the site while I set them back up again. And, of course, while I back things up and wait for the drive replacement.

Taluntain
Sat, 6th Oct '07, 6:47pm
Testing, testing...

I haven't gone through the whole lengthy procedure to transfer the boards properly since it might not be necessary in this case, but we won't really know unless we start seeing problems like errors and such pop up here. Please test as many functions of the boards as you can, and report any problems here.

Takara
Sat, 6th Oct '07, 6:51pm
Will do. Thanks Tal, good work. :thumb:

Harbourboy
Sat, 6th Oct '07, 7:57pm
Nothing seems to be going wrong so far.