Sonic and Sega Retro Message Board: Don't read the front page? You suck. - Sonic and Sega Retro Message Board

Jump to content

Hey there, Guest!  (Log In · Register) Help
  • 4 Pages +
  • 1
  • 2
  • 3
  • 4
    Locked
    Locked Forum

Don't read the front page? You suck.

#31 User is offline Andlabs 

Posted 29 March 2013 - 12:57 PM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
Thanks; that settles that, and I'll just remove the redirect following. The Sonic part can wait.

Now for the other side: the pages indexed in the database but not here. Mainly this consists of accessories and console pages. Should we start tracking these too? If so, should they be listed with the games or separate? Or will I need to add a flag that tells us not to track scans of those, or not have those use Scanbox at all? Thanks.

#32 User is offline Black Squirrel 

Posted 29 March 2013 - 04:34 PM

  • wikiman
  • Posts: 3589
  • Joined: 27-December 03
  • Gender:Male
  • Location:Northumberland, England
  • Project:thinking of a better member title
  • Wiki edits:20,569
Accessories are worth doing, though it will be tricky to work out what constitutes a "complete" set of scans. Most boxes will have six sides that need scanning, but some may only need four or five, since there's not much consistency amongst third-party stuff. The other point to make is not all accessories will have scans - bundled controllers is the prime example

Consoles are more awkward. tbh I'm not keen on how we're documenting all the different models/bundles right now - the solution I came up with a few months ago kind-of works... but I'm not convinced shoving everything on one page is the best plan, particularly if we start getting detailed info on contents and release dates and prices. It will almost certainly confuse the tracker too in its current state, and we're also not very consistent right now.

#33 User is offline Andlabs 

Posted 30 March 2013 - 04:11 AM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
I'm not entirely sure either, but the scenario is this:
In order for a scanbox to be listed, the console= value in the template has to match the category name exactly, and right now the categories are hardcoded to be Cateogry:(console)_games. I could alter things to show everything on one page and use color coding to separate accessories from games, but I would have to make things more complicated to merge multiple lists (I have a potential idea that I could get to later).
Or we could put the accessories and games on separate pages, say "Mega Drive games" and "Mega Drive accessories", etc. The issue then is that the console= wouldn't necessarily match, and I would need extra information if I wanted to make the front page faster, because now I have no way of differentiating an accessory from a game. Of course, I could use string comparisons on the table...

I'll probably map out something tomorrow morning, but I know it will involve no longer automatically trying to discern the list of consoles through a filter and just specifying the list of consoles manually. Until now I enjoyed the luxury of the tracker automatically producing the list of consoles to track, but albums already give us special-casing and it's going to get worse as I add more things (see above). In that case, I would have my configuration file have a pair (category name, scanbox= field) and I could take the latter approach... I'll map it out tomorrow

#34 User is offline Andlabs 

Posted 30 March 2013 - 07:12 PM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
Okay, let's see: let's say I manually state which categories to watch using pairs of the form
     { "category name": "scanbox= text" }
For the front page, I would need to do the following
  • Read all scanboxes
  • Read all page names for all categories
  • For each scanbox, determine the values and add them to the stats for the category it belongs to
  • Produce the full report
For individual report pages, I would need to do the following
  • Read all scanboxes for the given console= value
  • Read all page names from the category
  • For each scanbox, determine the values and add them to the stats
  • Produce the full report
My issue is whether or not this would require me to duplicate code, and what data structures I would need to use to make this sortable (for the report page). However, this looks attractive, and if I use a map[category string]ScanSet I think I can avoid the need to use separate code outright (just pass in all the category names or just the one category name as a list...).

I'll just do that, if everybody thinks it'll do. The only cost is that this list needs to be edited if we add or drop categories, but since it will be in a configuration file, this won't be a hassle. In that case, here's what the category list will be:
Spoiler
Accessories without indiviudal scans can be dealt with individually; I have Template:NoScans for that... This is also a proposal for the system variations pages: put them all in a category like Category:Mega Drive systems, put the system variation name in the region= field (yes this is clumsier but avoids needing to have multiple entries or one category), and then it would be covered too. I can also edit this list to cover tings like our video scans...

This leaves a number of scanboxes (such as the ones for the Football Manager titles) that are marked PC/Mac; not sure how to handle those apart from splitting nad having duplicates :/ Any other ideas? Also should the two games for PC Engine Super CD-ROM² games (Gain Ground SX and Bonanza Bros.) have the scanbox say (and thus be listed as) PC Engine CD-ROM² games?
This post has been edited by Andlabs: 31 March 2013 - 01:30 AM

#35 User is offline Scarred Sun 

Posted 31 March 2013 - 09:54 AM

  • Rise and shine, ya sleepy layabouts.
  • Posts: 3693
  • Joined: 06-February 05
  • Gender:Female
  • Location:SD/LA/SF
  • Project:Staying woke
  • Wiki edits:36,091
Reminder of my request for a manual column.

#36 User is offline Andlabs 

Posted 31 March 2013 - 10:25 AM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
Yes, this is all leading up to the manual column. I want to get this polished and fast before breaking it more =P
This post has been edited by Andlabs: 31 March 2013 - 10:28 AM

#37 User is offline Andlabs 

Posted 01 April 2013 - 03:49 AM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
Well I went ahead and made the switch on the scan server, and everything's MUCH faster now :) The front page takes between 1 and 3 seconds to load, whereas it used to take between 4 and 6. The only cost was that I had to split the Mega Anser and Miracle Piano pages into one for the accessory and one for the software because the system can't deal with a page being in both (and that would require a fair amount of change). I also changed it to use one database connection per page request rather than one globally, so there shouldn't be issues should the MySQL server drop and come back.

I'll be polishing the system off either all day tomorrow or over the next few days (depending on how long it takes) and at the end I'll add tracking manual scans to the mix.

Do you want to go with my proposal for dealing with consoles themselves? Is there anything I am missing from these categories? Did any games or categories disappear? Thanks.
This post has been edited by Andlabs: 01 April 2013 - 04:03 AM

#38 User is offline Black Squirrel 

Posted 01 April 2013 - 05:52 AM

  • wikiman
  • Posts: 3589
  • Joined: 27-December 03
  • Gender:Male
  • Location:Northumberland, England
  • Project:thinking of a better member title
  • Wiki edits:20,569
answers

- there aren't any 32X/CD 32X accessories. Or any Mega CD 32X consoles in boxes because that would make too much sense
- copera should be merged with Pico
- Pico accessories do exist

also might I suggest different tables for accessories/consoles for the sake of readibility

#39 User is offline Andlabs 

Posted 01 April 2013 - 02:00 PM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
Thanks. It turns out all the Copera games are already listed under Pico, so that settles that. As for splitting by type, I'll see what I can do after settling everything else.

#40 User is offline Andlabs 

Posted 02 July 2013 - 09:07 PM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
The scan tracker is back up, and now tracks manuals too! Once again, feel free to report any bugs or make any suggestions (suggestions on how to not make it horizontally scroll while not line wrapping the console list would be nice). Also covers gaming books, since Black Squirrel recently did a bunch of them.

The front page may take a few seconds to load. I'm not really sure anymore why it's not running as fast as it did two months ago, but whatever; at this point I imagine the data load will eventually make it not really matter :/ (maybe?...)
This post has been edited by Andlabs: 02 July 2013 - 09:09 PM

#41 User is offline Andlabs 

Posted 03 July 2013 - 11:27 PM

  • 「いっきまーす」
  • Posts: 2175
  • Joined: 11-July 08
  • Gender:Male
  • Project:Writing my own MD/Genesis sound driver :D
  • Wiki edits:7,061
And finally, all the speed issues have been sorted out. The front page has gone from a max of 9 seconds several months ago to at most 650 milliseconds to load. Everything now runs nicely under a second, even the front page! No need for caching; this is still in real time. Now let's get this show on the road!



For those who wonder: when I was grabbing the list of scanboxes and whether or not files were good or not, I was making one query per game/file (respectively). This meant lots and lots of database hits, which was making everything slow. I got rid of the scanbox mess a while ago (with Metal_Man88's help), but had kept the one query per file check until now. Lesson: don't hit the database too often. There are two issues here, but they are both irrelevant for now (and hopefully forever). The first issue is that this may mean the server is going to eat up memory (but I have a garbage collector which I hope will come in when the page has fully rendered and all the reference to everything are dropped) and have string operations all over the place, but 650ms max for the front page shows me this isn't a problem. The second issue is that because all information is transferred on all pages, even individual console pages, the individual console pages are going to take slightly longer to load because they pull in data they do not use. However, again, this is a negligible change (~100ms or so, I think).
This post has been edited by Andlabs: 03 July 2013 - 11:29 PM

#42 User is offline SeanieB 

Posted 03 July 2013 - 11:44 PM

  • errno -1 (Not system error)
  • Posts: 435
  • Joined: 26-February 08
  • Gender:Male
  • Location:San Diego, CA
  • Project:Fixing Sonic Retro
  • Wiki edits:12
I've also been doing a ton of server-side optimizations, I think I've basically done all I can with all the existing systems. But I think I made out pretty well. I'm probably just gonna be doing mostly maintenance at this point, everything else would come out in the form of replacing the forum software or transitioning the DB engine type, and a few other things I just don't really want to go near on my own. (like updating the Retro wiki)

#43 User is offline Metal Man88 

Posted 04 July 2013 - 12:12 AM

  • Time Traveller
  • Posts: 2138
  • Joined: 06-March 06
  • Gender:Male
  • Wiki edits:46
Ah, I've been wanting to see the database stuff in action for a while now. Happy to see the idea I had worked.

#44 User is offline Ashram 

Posted 25 November 2013 - 06:37 AM

  • Posts: 1985
  • Joined: 22-February 09
  • Gender:Male
  • Location:California
  • Wiki edits:13
Out of curiosity, how long has it been since the NTSC Genesis list was updated? My friend has a bunch of cartridges that we might need from that list, but I don't know if we already have them.

#45 User is offline Master Emerald 

Posted 25 November 2013 - 07:04 AM

  • Posts: 3001
  • Joined: 14-December 07
  • Gender:Male
  • Location:Rio de Janeiro - Brazil
  • Project:College
  • Wiki edits:22
When will the Marble thing go to the front page?

  • 4 Pages +
  • 1
  • 2
  • 3
  • 4
    Locked
    Locked Forum

1 User(s) are reading this topic
0 members, 1 guests, 0 anonymous users