Sonic and Sega Retro Message Board: SeanieB - Viewing Profile - Sonic and Sega Retro Message Board

Sites are all coming back up! We're on the new server you all helped us fund after the old one started to fail after over 6 years of service. Good life for heavily used consumer-grade kit!

We're on a new server running (mostly) server grade components and on considerably more modern components (to fit our budget)

Tech specs:

Supermicro X10SAE (LGA1150) platform
Intel Xeon 1231-V3 (3.4 GHz 4 cores/8 threads) CPU
32GB Crucial (4x 8GB) 1600 MHz ECC (unreg) RAM
2 240GB Mushkin SSDs in RAID1 for the operating system and SQL database disk storage
2 1TB WDC RE4 Enterprise hard drives in RAID1 for all the files
4TB Buffalo external drive for backups over USB 3

It's good to be back after pretty much a few days of work between Scarred Sun and I, Consider this a bug reports thread as we will probably need to tweak things, expect minor downtime over the next few days while I get everything straightened out.

The wikis are pretty broken. We're only looking for reports of major functionality issues with them. white screens, upload failures etc

Edit: I went ahead and switched the new webserver in. Let me know in this thread or in PM if you find breakages.

Welcome back!

We've got our shiny new OS install and HDD's in place, and everything seems to be working perfectly.

Repo and Scans should be back over the weekend.

If you find any bugs around the site, please report them here.

Hey there, everyone --

I'm just gonna write up a quick thing about today's chain of events.

We've been tracking a dying pair of the original Samsung 500GB SATA drives inside retro's web server for a few days now. Around 1:30-2:00 AM Pacific this morning, the server dropped off the face of the earth. I wasn't notified about this because my Nagios was only set to notify me of Retro's status by e-mail and pager text-message during work hours.

I'm not entirely sure if it was a coincidence, a bit more disk failure, software failure, or maybe a hack gone wrong, but no logs of the problem were made, and the OS on the machine was definitely damaged in some way.

For those familiar with Linux, conventional System V style init uses a series of init scripts to accomplish the boot sequence, and handling of run level requests. For a still undiscovered reason, on initial run level, not all of the daemons start, but if SSH is adjusted to start immediately following networking, manually changing run level to one of the other unoccupied multiuser run levels on Debian will start the remaining daemons and allow logins to work as normal from the console.

I verified the drives with my personal copy of Spinrite and they did turn up a couple more errors, but no outright failure. RAM tests also passed.

Some of Retro's ancillary public and private services have been disabled until the restore to the new drives and OS is complete. Additional backup plans are being carried out to make sure we're prepared for catastrophe. Expect a lot of slowdown in the next 24 hours. I will be setting my Nagios to report problems 24/7 until Retro goes back to normal.

New drives have been ordered, and tentatively we hope this process can get started by Wednesday night, but that's an estimate. Expect another full day's downtime at that time, but I will give full warning. I am going to fully test the new drives before putting them into service, I think it would be prudent to spare some time for confidence in the stability of the new drives.

As far as I know at this time no information was lost in this incident. A backup finished very close to the time of failure, which may have also been a root cause.

Repo, FTP, the Scans service, and a couple of under utilized user accounts have been disabled for security, and will reappear later this evening or some time tomorrow.

I'm willing to accept questions in this thread

Thanks for your understanding.

Hello all --

After an extensive upgrade, the Repo is back in service, but the theme is reverted and the integration to the forum is in an unknown state.

Existing accounts should still work fine, but any accounts created after ~Jan 26 will not be integrated into the Hg frontend. If you can not access with your forum account, information (AND ARE A >=TECH MEMBER) PM me.

Read-only access is public, and you do not need to log in.

Cinossu and I are working to get it back to its' normal state.