[Andlabs]

Thanks; that settles that, and I'll just remove the redirect following. The Sonic part can wait.

Now for the other side: the pages indexed in the database but not here. Mainly this consists of accessories and console pages. Should we start tracking these too? If so, should they be listed with the games or separate? Or will I need to add a flag that tells us not to track scans of those, or not have those use Scanbox at all? Thanks.

[Black Squirrel]

Accessories are worth doing, though it will be tricky to work out what constitutes a "complete" set of scans. Most boxes will have six sides that need scanning, but some may only need four or five, since there's not much consistency amongst third-party stuff. The other point to make is not all accessories will have scans - bundled controllers is the prime example

Consoles are more awkward. tbh I'm not keen on how we're documenting all the different models/bundles right now - the solution I came up with a few months ago kind-of works... but I'm not convinced shoving everything on one page is the best plan, particularly if we start getting detailed info on contents and release dates and prices. It will almost certainly confuse the tracker too in its current state, and we're also not very consistent right now.

[Andlabs]

I'm not entirely sure either, but the scenario is this:
In order for a scanbox to be listed, the console= value in the template has to match the category name exactly, and right now the categories are hardcoded to be Cateogry:(console)_games. I could alter things to show everything on one page and use color coding to separate accessories from games, but I would have to make things more complicated to merge multiple lists (I have a potential idea that I could get to later).
Or we could put the accessories and games on separate pages, say "Mega Drive games" and "Mega Drive accessories", etc. The issue then is that the console= wouldn't necessarily match, and I would need extra information if I wanted to make the front page faster, because now I have no way of differentiating an accessory from a game. Of course, I could use string comparisons on the table...

I'll probably map out something tomorrow morning, but I know it will involve no longer automatically trying to discern the list of consoles through a filter and just specifying the list of consoles manually. Until now I enjoyed the luxury of the tracker automatically producing the list of consoles to track, but albums already give us special-casing and it's going to get worse as I add more things (see above). In that case, I would have my configuration file have a pair (category name, scanbox= field) and I could take the latter approach... I'll map it out tomorrow

[Andlabs]

Okay, let's see: let's say I manually state which categories to watch using pairs of the form

     { "category name": "scanbox= text" }

For the front page, I would need to do the following

• Read all scanboxes
  
• Read all page names for all categories
  
• For each scanbox, determine the values and add them to the stats for the category it belongs to
  
• Produce the full report

For individual report pages, I would need to do the following

• Read all scanboxes for the given console= value
  
• Read all page names from the category
  
• For each scanbox, determine the values and add them to the stats
  
• Produce the full report

My issue is whether or not this would require me to duplicate code, and what data structures I would need to use to make this sortable (for the report page). However, this looks attractive, and if I use a map[category string]ScanSet I think I can avoid the need to use separate code outright (just pass in all the category names or just the one category name as a list...).

I'll just do that, if everybody thinks it'll do. The only cost is that this list needs to be edited if we add or drop categories, but since it will be in a configuration file, this won't be a hassle. In that case, here's what the category list will be:

Spoiler

32X accessories (?? - I doubt any exist)
32X games
32X systems
Albums
Amiga games
Amstrad CPC games
Apple II games
Atari 2600 games
Atari 5200 games
Atari 8-bit family games
Atari ST games
Atomiswave games
Aurora games
BBC Micro games
Beena accessories
Beena games
Beena systems
Chihiro games
Colecovision games
Commodore 64 games
Copera games (?? - should this just be merged with Pico?)
DOS games
Dragon 32 games
Dreamcast accessories
Dreamcast games
Dreamcast systems
Europa-R games
FM Towns games
Fujitsu FM-7 games
G80 games
Game Boy Advance games
Game Boy Color games
Game Gear accessories
Game Gear games
Game Gear systems
Game.com games
GameCube games
Hikaru games
Intellivision games
LCD games
Laserdisc hardware games
Lindbergh games
MSX games
Mac games
Master System accessories
Master System games
Master System systems
Mega CD accessories
Mega CD games
Mega CD systems
Mega CD 32X accessories (?? - I doubt any exist)
Mega CD 32X games
Mega CD 32X systems (?? - I doubt any such distributions exist)
Mega Drive accessories
Mega Drive games
Mega Drive systems
Mega LD accessories
Mega LD games
Mega LD systems (?? - assuming the Mega LD was sold separately)
Mega Play games
Mega-Tech games
N-Gage games
NAOMI games
NAOMI 2 games
NEC PC-6001 games
NEC PC-8001 games
NEC PC-8801 games
NEC PC-9801 games
NES games
Neo Geo Pocket Color games
Nintendo 3DS games
Nintendo DS games
PC accessories (Sega Logistics Service-manufactured controllers)
PC games
PC Engine CD-ROM² games
Palm OS games
Pico accessories (?? - were there any?)
Pico games
Pico systems
PlayStation games
PlayStation 2 accessories (Sega Logistics Service-manufactured controllers)
PlayStation 2 games
PlayStation 3 games
PlayStation 3 systems (?? - various Yakuza bundles in Japan)
PlayStation Portable games
PlayStation Vita games
Pocket PC games
RingEdge games
RingEdge 2 games
RingWide games
SC-3000 accessories
SC-3000 games
SC-3000 systems
SG-1000 accessories
SG-1000 games
SG-1000 systems
Saturn accessories
Saturn games
Saturn systems
Sega Titan Video games
Sharp MZ games
Sharp X1 games
Sharp X68000 games
System 24 games
System E games
TI-99/4A games
TRS-80 games
TRS-80 CoCo games
Triforce games
TurboGrafx-16 games
VFD games (?? - actually do we need this or should we just have LCD? or at all?)
VIC-20 games
Wii games
Wii U games
WonderSwan games
Xbox games
Xbox 360 games
ZX Spectrum games

Accessories without indiviudal scans can be dealt with individually; I have Template:NoScans for that... This is also a proposal for the system variations pages: put them all in a category like Category:Mega Drive systems, put the system variation name in the region= field (yes this is clumsier but avoids needing to have multiple entries or one category), and then it would be covered too. I can also edit this list to cover tings like our video scans...

This leaves a number of scanboxes (such as the ones for the Football Manager titles) that are marked PC/Mac; not sure how to handle those apart from splitting nad having duplicates :/ Any other ideas? Also should the two games for PC Engine Super CD-ROM² games (Gain Ground SX and Bonanza Bros.) have the scanbox say (and thus be listed as) PC Engine CD-ROM² games?

This post has been edited by Andlabs: 31 March 2013 - 01:30 AM

[Scarred Sun]

Reminder of my request for a manual column.

[Andlabs]

Yes, this is all leading up to the manual column. I want to get this polished and fast before breaking it more =P

This post has been edited by Andlabs: 31 March 2013 - 10:28 AM

[Andlabs]

Well I went ahead and made the switch on the scan server, and everything's MUCH faster now The front page takes between 1 and 3 seconds to load, whereas it used to take between 4 and 6. The only cost was that I had to split the Mega Anser and Miracle Piano pages into one for the accessory and one for the software because the system can't deal with a page being in both (and that would require a fair amount of change). I also changed it to use one database connection per page request rather than one globally, so there shouldn't be issues should the MySQL server drop and come back.

I'll be polishing the system off either all day tomorrow or over the next few days (depending on how long it takes) and at the end I'll add tracking manual scans to the mix.

Do you want to go with my proposal for dealing with consoles themselves? Is there anything I am missing from these categories? Did any games or categories disappear? Thanks.

This post has been edited by Andlabs: 01 April 2013 - 04:03 AM

[Black Squirrel]

answers

- there aren't any 32X/CD 32X accessories. Or any Mega CD 32X consoles in boxes because that would make too much sense
- copera should be merged with Pico
- Pico accessories do exist

also might I suggest different tables for accessories/consoles for the sake of readibility

[Andlabs]

Thanks. It turns out all the Copera games are already listed under Pico, so that settles that. As for splitting by type, I'll see what I can do after settling everything else.

[Andlabs]

The scan tracker is back up, and now tracks manuals too! Once again, feel free to report any bugs or make any suggestions (suggestions on how to not make it horizontally scroll while not line wrapping the console list would be nice). Also covers gaming books, since Black Squirrel recently did a bunch of them.

The front page may take a few seconds to load. I'm not really sure anymore why it's not running as fast as it did two months ago, but whatever; at this point I imagine the data load will eventually make it not really matter :/ (maybe?...)

This post has been edited by Andlabs: 02 July 2013 - 09:09 PM

[Andlabs]

And finally, all the speed issues have been sorted out. The front page has gone from a max of 9 seconds several months ago to at most 650 milliseconds to load. Everything now runs nicely under a second, even the front page! No need for caching; this is still in real time. Now let's get this show on the road!



For those who wonder: when I was grabbing the list of scanboxes and whether or not files were good or not, I was making one query per game/file (respectively). This meant lots and lots of database hits, which was making everything slow. I got rid of the scanbox mess a while ago (with Metal_Man88's help), but had kept the one query per file check until now. Lesson: don't hit the database too often. There are two issues here, but they are both irrelevant for now (and hopefully forever). The first issue is that this may mean the server is going to eat up memory (but I have a garbage collector which I hope will come in when the page has fully rendered and all the reference to everything are dropped) and have string operations all over the place, but 650ms max for the front page shows me this isn't a problem. The second issue is that because all information is transferred on all pages, even individual console pages, the individual console pages are going to take slightly longer to load because they pull in data they do not use. However, again, this is a negligible change (~100ms or so, I think).

This post has been edited by Andlabs: 03 July 2013 - 11:29 PM

[SeanieB]

I've also been doing a ton of server-side optimizations, I think I've basically done all I can with all the existing systems. But I think I made out pretty well. I'm probably just gonna be doing mostly maintenance at this point, everything else would come out in the form of replacing the forum software or transitioning the DB engine type, and a few other things I just don't really want to go near on my own. (like updating the Retro wiki)

[Metal Man88]

Ah, I've been wanting to see the database stuff in action for a while now. Happy to see the idea I had worked.

[Ashram]

Out of curiosity, how long has it been since the NTSC Genesis list was updated? My friend has a bunch of cartridges that we might need from that list, but I don't know if we already have them.

[Master Emerald]

When will the Marble thing go to the front page?