I Wanna Be The Forums!

Please login or register.

Login with username, password and session length
Advanced search  

News:

Gaiden 1.2 patch released. Click here to download!

Author Topic: Drive Failure and Incoming Downtime  (Read 12020 times)

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Drive Failure and Incoming Downtime
« on: November 12, 2012, 10:07:55 pm »

So some nutso drive problems are going on with the pyoko server so there IS going to be some downtime. I'm currently backing stuff up, so if things fail, they will be backed up from this point. So any comments made from this point on have no certainty of appearing after all this is dealt with. Hopefully I won't lose anything meaningful during all this. Either way: Expect downtime. Possibly a lot.

edit: Alright we're alive again. Should be fine. There might be downtime somewhere down the road when the secondary bad disk gets replaced, but that'll happen whenever it happens and hopefully should make for shorter downtime.
« Last Edit: November 15, 2012, 04:31:36 pm by Kayin »
Logged

Berry Punch

  • Kitty's BFF
  • Sr. Member
  • ****
  • Posts: 374
  • 40 proof noggererer.
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #1 on: November 12, 2012, 10:47:06 pm »

No RAID setup on the server? Meh.
Logged
Download this [ABANDONED] shit that nobody cares about: I wanna be PHAT!

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #2 on: November 13, 2012, 01:50:51 am »

RAID doesn't necessarily protect you from Data Loss (since RAID can back up flawed data). So right now the raid is broken and the surviving disk is being checked for consistency and bad sectors, but there can still be some corrupted files. There's some fishy stuff goin' on that the Admin is checking out.

But yeah, RAID isn't magic armor.
Logged

Berry Punch

  • Kitty's BFF
  • Sr. Member
  • ****
  • Posts: 374
  • 40 proof noggererer.
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #3 on: November 13, 2012, 02:11:11 am »

Well at least it looks like you had a RAID setup and some sort of backup ready. Otherwise that would just suck. Hope it gets fixed without any issues though.
Logged
Download this [ABANDONED] shit that nobody cares about: I wanna be PHAT!

Ellipsis

  • Ellipsis
  • The Guy
  • *****
  • Posts: 1907
    • View Profile
    • If Not Equal Limited
Re: Drive Failure and Incoming Downtime
« Reply #4 on: November 13, 2012, 09:01:32 am »

Ah. Sucks to here that. But at least hopefully there should be some sort of backup from the sounds of it. That's good to hear.
What on earth am I meant to do while the forum is down? This is going to be the most productive time of my life.
Kayin, any idea roughly how much downtime to expect? Days, weeks, months?
Logged

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #5 on: November 13, 2012, 03:34:19 pm »

Hours, though if the admin can't do everything in one go, it might be out for a day. Both drives are displaying errors. To quote him. "Two of the same drives from the same batch, in a RAID config getting the same usage". The good news is this doesn't sound like a catastrophic sudden onset of bad sectors. Or maybe it isn't on the so called "good" drive". I'm trying not to be a huge pest. But either way, the drive in the best condition is NOT a candidate for being in the new install. So both drives will be replaced and all the VMs, including mine, will be saved as disk images and then restored. He'll be doing this when he gets back from work most likely and will probably take the better part of the evening.

I personally have backups of my mySQL tables so in the worst case I can eventually get stuff back together if somehow everything fails, but our Admin is awesome and has been doing co-lo stuff for years. I think part of this and the warning about Dataloss isn't because it's necessarily likely, but he treats it with an admin's expected level of severity and professionalism.

Edit: Little more info. Both drives are exhibiting bad sectors but the situation isn't dire like I said. But it is a sign that the drives have to be replaced sooner than later and that'll happen tonight. No known data lose and hopefully nothing will go wrong during the transfer.
« Last Edit: November 13, 2012, 05:24:43 pm by Kayin »
Logged

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #6 on: November 14, 2012, 10:28:52 pm »

okay it didn't happen last night. It's happening 11:45 tonight. So in like 2 hours.
Logged

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #7 on: November 14, 2012, 10:46:04 pm »

Also downtime SHOULD be minimal if all goes well. Admin (Maou) is replacing one of the harddrives and letting the raid sync. Once it's clear everything is okay he'll eventually replace the second drive. If things muck up, expect longer down time.

This also means if things don't muck up, they'll be no data rollbacks.
Logged

Kayin

  • Well.. They're more like giant cherries...
  • Administrator
  • The Guy
  • *****
  • Posts: 1254
    • View Profile
Re: Drive Failure and Incoming Downtime
« Reply #8 on: November 15, 2012, 04:31:48 pm »

ALL DONE
Logged

Ellipsis

  • Ellipsis
  • The Guy
  • *****
  • Posts: 1907
    • View Profile
    • If Not Equal Limited
Re: Drive Failure and Incoming Downtime
« Reply #9 on: November 15, 2012, 10:28:10 pm »

Ah good. It's nice to have the forum back!
Glad all the data was OK and it's good to know it's also backed up too.
Logged