don't click here

Server repairs & upgrades

Discussion in 'Announcements' started by SeanieB, Sep 25, 2021.

  1. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    Hi! Sorry no one's heard much from the Admin team in a while about the performance issues on the server/site and the outages and general sluggishness. I've been monitoring and keeping things kind of alive for a while but SS has been going through some intense personal stuff, but I got the green light to kinda take the helm over this server stuff to get things a lot more stable and running better.

    The gist of it is almost all of the storage devices in the Retro box are close to 50,000 hours of flying time, which is when conventional spinning disks lose performance and start to be ready to die again. One disk has died completely in the OS RAID so we have no redundancy if the remaining OS disk were to die. A site this heavy will also run like shit off of a single disk.

    I donated 4TB of offsite backup space off of my SAN in Los Angeles so we at least have something if that disk were to croak.

    I don't think we can afford to replace the whole server at this time, and it will run well if we give it some TLC so I am going to buy some parts out of my pocket, have them sent to the DC and have the DC staff overhaul the server.

    Full disclosure, here's what I'm buying and what I've spent:

    https://p.lostsig.net/EZNiD5.png

    $1,212 USD buys us good quality enterprise grade SSDs that can actually handle the OS and database disks for 2-3 years instead of instantly dying, some new spinning disks for the datastore and a NVMe SSD to use as a ZFS L2ARC cache

    I will also donate 64GB of ECC DDR3 from my personal stash to also allow for extra cache RAM The RAM I have is not compatible :c

    SS may be able to give me some of the Patreon/recurring donation funds for this. Whatever is left I will see if anyone in the community wants to help me donate to recoup, anything left will just be my personal donation for the next couple years, and if I go over $1212 with donations I will refund people. If you want to help me recoup some of these costs:

    Paypal: [email protected] / Cash App: $seanieb64 / Venmo @seanieb64

    This should allow everything to run way better and take a lot of stress off of everyone so we're not constantly wondering when all the storage is gonna randomly die. Thanks for your time!
     
    Last edited: Sep 25, 2021
  2. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    Thanks to Gerbilsoft for $169 :v
    Thanks to Armonte for the $5
    Thanks to Larry for the $20
    Thanks to Anonymous for $100
    Thanks to Chaos Hedgie for $200
    Thanks to BiggestSonicFan for $20
    Thanks to SoNick for $20
    Thanks to Mykonos for $20
    Thanks to Andrew for $5
    Thanks to Rgamer2009 for $100
    Thanks to Bill for $10
    Thanks to Sorachi for $100
    Thanks to CartridgeCulture for $20
    Thanks to Andrew for $50

    Total remaining is now $382. Everything is ordered and I'm going to drop RAM in the mail The RAM I have is not compatible :c
     
    Last edited: Oct 11, 2021
  3. PicklePower

    PicklePower

    Wiki Sysop
    632
    26
    28
    We do have (complete/tested/offsite) wiki and forum backups, right?
     
  4. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    Correct. I have a full backup that runs every seven days I'm hosting in LA. The whole server could die and I could just host the website out of that infrastructure.
     
  5. MykonosFan

    MykonosFan

    MODE CHANGE. Moderator
    3,746
    550
    93
    Wish I could part with more, but sent $20 your way to help with this.
     
  6. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    The DC has received all the parts. I will schedule everything for Friday afternoon to get everything installed then Friday night to get everything reinstalled so we'll have some downtime on Friday night
     
  7. Excellent, Im sure all will agree we won't mind the downtime. I hope this will help the site with less downtime in the future!
     
  8. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,543
    2,465
    93
    Northumberland, UK
    steamboat wiki
    Question

    for years we've been essentially "turning off" PDF thumbnails for performance reasons - will these upgrades give us enough headroom to be ~pretty~ again?

    Sega Saturn/Magazine articles would be one of the more demanding pages.
     
  9. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    We're coming back online slowly. The wikis are massive [1.9TB / 1,487,121 files] so I split the migration up into four parallel processes and brought the forum back online. Somehow even with it running like mad it can still load the forum really quickly. Unsure about PDF generation, someone will probably have to experiment and find out if it's fast enough to help. We still have the same CPU and that honestly might be the bottleneck there.

    Still owe some on the parts if anyone else is able to help out. Appreciate it.

    Right now the forum's the only thing that's up. I'll update as more parts of the community come up.
     
  10. filipeandrade

    filipeandrade

    Member
    9
    0
    1
    I'm sorry if this has already beed answered before, but is there a way to support the servers, like a Patreon page or something? Would that help keep the servers always up?

    I don't usually come to this forum, so I don't really know what you guys discuss here, but I have a gaming channel on YouTube that's focused on SEGA and I navigate on SEGA Retro basically every day (I always give you guys credits, of course). Every time the site is down (and sadly, this happens quite often), it's a nightmare, because sometimes I would need to use the search function to search for specific words, and WayBackMachine can't help me with that, only specific pages.

    That said, I'd gladly pay something like 5 dollars monthly if this helps maintaining the server in a good shape, because sometimes, pages would refuse to load quite often and it's always troublesome. I'm sure other people wouldn't mind to donate a few dollars per month to keep the servers always up.

    Again, sorry if you guys already have something like this or if this has been answered before. I honestly just don't know.
     
  11. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    I believe there is a patreon, but I don't have access to those funds since I financed this one directly
     
  12. Forgive me if this question is ignorant; I've only ever set up and maintained static websites, not any multi-user platform things like a forum, but does it make more sense to roll your own server (and assume the risks thereof) rather than to simply pay a big-name hosting service that has scalability and redundancy built into the product? I'm assuming DIY is costing less overall, but I don't know much peace of mind there would be to not have to worry about parts failing and disks going bad.
     
  13. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    A capex every few years and a colo fee less than $100 a month is easier to handle either out of pocket or from donations for SS than what would be the equivalent of probably $500+ a month on a public cloud, especially since we'd need ~4 TB of block storage and a minimum of 16gb of RAM

    For instance, the lowest common demonminator public cloud platform, [OVH USA] if we crammed everything on a single image, with a VM the size of our current machine and 4TB of block storage would be $388.72 per month without paying for ingress or egress bandwidth,or backup services. Wouldn't even be a capability or capacity upgrade, and would be a downgrade in storage. To match what we just bought would be $573.04 a month without counting backup or ingrees/egress fees.
     
    Last edited: Oct 10, 2021
    • Informative Informative x 5
    • List
  14. Got it, thanks!
     
  15. SeanieB

    SeanieB

    Chief Server Monkey Administrator
    457
    38
    28
    San Diego, CA
    Fixing Sonic Retro
    At this time, all the sites should be back up. I'm sure there'll be some bugs/glitches but just let us know.
     
  16. cartridgeculture

    cartridgeculture

    Wiki Contributor Member
    Thank you for all your hard work SeanieB :)
     
    • Agree Agree x 4
    • Like Like x 1
    • List
  17. Brainulator

    Brainulator

    Regular garden-variety member Member
    Glad things worked out OK!