Hi! Sorry no one's heard much from the Admin team in a while about the performance issues on the server/site and the outages and general sluggishness. I've been monitoring and keeping things kind of alive for a while but SS has been going through some intense personal stuff, but I got the green light to kinda take the helm over this server stuff to get things a lot more stable and running better. The gist of it is almost all of the storage devices in the Retro box are close to 50,000 hours of flying time, which is when conventional spinning disks lose performance and start to be ready to die again. One disk has died completely in the OS RAID so we have no redundancy if the remaining OS disk were to die. A site this heavy will also run like shit off of a single disk. I donated 4TB of offsite backup space off of my SAN in Los Angeles so we at least have something if that disk were to croak. I don't think we can afford to replace the whole server at this time, and it will run well if we give it some TLC so I am going to buy some parts out of my pocket, have them sent to the DC and have the DC staff overhaul the server. Full disclosure, here's what I'm buying and what I've spent: https://p.lostsig.net/EZNiD5.png $1,212 USD buys us good quality enterprise grade SSDs that can actually handle the OS and database disks for 2-3 years instead of instantly dying, some new spinning disks for the datastore and a NVMe SSD to use as a ZFS L2ARC cache I will also donate 64GB of ECC DDR3 from my personal stash to also allow for extra cache RAM The RAM I have is not compatible :c SS may be able to give me some of the Patreon/recurring donation funds for this. Whatever is left I will see if anyone in the community wants to help me donate to recoup, anything left will just be my personal donation for the next couple years, and if I go over $1212 with donations I will refund people. If you want to help me recoup some of these costs: Paypal: [email protected] / Cash App: $seanieb64 / Venmo @seanieb64 This should allow everything to run way better and take a lot of stress off of everyone so we're not constantly wondering when all the storage is gonna randomly die. Thanks for your time!
Thanks to Gerbilsoft for $169 :v Thanks to Armonte for the $5 Thanks to Larry for the $20 Thanks to Anonymous for $100 Thanks to Chaos Hedgie for $200 Thanks to BiggestSonicFan for $20 Thanks to SoNick for $20 Thanks to Mykonos for $20 Thanks to Andrew for $5 Thanks to Rgamer2009 for $100 Thanks to Bill for $10 Thanks to Sorachi for $100 Thanks to CartridgeCulture for $20 Thanks to Andrew for $50 Total remaining is now $382. Everything is ordered and I'm going to drop RAM in the mail The RAM I have is not compatible :c
Correct. I have a full backup that runs every seven days I'm hosting in LA. The whole server could die and I could just host the website out of that infrastructure.
The DC has received all the parts. I will schedule everything for Friday afternoon to get everything installed then Friday night to get everything reinstalled so we'll have some downtime on Friday night
Excellent, Im sure all will agree we won't mind the downtime. I hope this will help the site with less downtime in the future!
Question for years we've been essentially "turning off" PDF thumbnails for performance reasons - will these upgrades give us enough headroom to be ~pretty~ again? Sega Saturn/Magazine articles would be one of the more demanding pages.
We're coming back online slowly. The wikis are massive [1.9TB / 1,487,121 files] so I split the migration up into four parallel processes and brought the forum back online. Somehow even with it running like mad it can still load the forum really quickly. Unsure about PDF generation, someone will probably have to experiment and find out if it's fast enough to help. We still have the same CPU and that honestly might be the bottleneck there. Still owe some on the parts if anyone else is able to help out. Appreciate it. Right now the forum's the only thing that's up. I'll update as more parts of the community come up.
I'm sorry if this has already beed answered before, but is there a way to support the servers, like a Patreon page or something? Would that help keep the servers always up? I don't usually come to this forum, so I don't really know what you guys discuss here, but I have a gaming channel on YouTube that's focused on SEGA and I navigate on SEGA Retro basically every day (I always give you guys credits, of course). Every time the site is down (and sadly, this happens quite often), it's a nightmare, because sometimes I would need to use the search function to search for specific words, and WayBackMachine can't help me with that, only specific pages. That said, I'd gladly pay something like 5 dollars monthly if this helps maintaining the server in a good shape, because sometimes, pages would refuse to load quite often and it's always troublesome. I'm sure other people wouldn't mind to donate a few dollars per month to keep the servers always up. Again, sorry if you guys already have something like this or if this has been answered before. I honestly just don't know.
I believe there is a patreon, but I don't have access to those funds since I financed this one directly
Forgive me if this question is ignorant; I've only ever set up and maintained static websites, not any multi-user platform things like a forum, but does it make more sense to roll your own server (and assume the risks thereof) rather than to simply pay a big-name hosting service that has scalability and redundancy built into the product? I'm assuming DIY is costing less overall, but I don't know much peace of mind there would be to not have to worry about parts failing and disks going bad.
A capex every few years and a colo fee less than $100 a month is easier to handle either out of pocket or from donations for SS than what would be the equivalent of probably $500+ a month on a public cloud, especially since we'd need ~4 TB of block storage and a minimum of 16gb of RAM For instance, the lowest common demonminator public cloud platform, [OVH USA] if we crammed everything on a single image, with a VM the size of our current machine and 4TB of block storage would be $388.72 per month without paying for ingress or egress bandwidth,or backup services. Wouldn't even be a capability or capacity upgrade, and would be a downgrade in storage. To match what we just bought would be $573.04 a month without counting backup or ingrees/egress fees.
At this time, all the sites should be back up. I'm sure there'll be some bugs/glitches but just let us know.
