Don't read the front page? You suck.

Discussion in 'Announcements' started by Scarred Sun, Aug 19, 2012.

  1. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    In the next day or two I'm going to try to optimize how segaret_scans talks to the MediaWiki database, so expect intermittent failures.
     
  2. flarn2006

    flarn2006

    Member Member
    264
    0
    0
    SA2 Cheat Table
  3. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Working on refactoring the scan system before adding manuals to the fray; should I just remove the Incomplete stat and mark all incomplete scans with Bad? It just makes the logic more complex, however there are a handful of games where I put item text with no item. If I should remove Incomplete, what should we do about those?

    EDIT - looks like some of my refactoring is causing MySQL to shit itself, so the scan system will be down until further notice; the question above is still open
     
  4. Black Squirrel

    Black Squirrel

    Fifty paces towards the sun, cooking beets. Wiki Sysop
    4,947
    0
    18
    Northumberland, England
    The soil knows all.
    I think it's fine as it is, although it's still picking up systems that will never have box scans. "Incomplete" doesn't turn up that often, but it can still be useful

    That's assuming you're planning to treat manuals as something different from "media", otherwise most of the games on this site will be marked as "incomplete"
     
  5. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    I was just about to get to the invalid systems part. And right; SS was talking about a manual column so I will probably do that.

    The program seems to have fixed itself after I fixed a snag elsewhere; if the database breaks again please let me know
     
  6. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Okay let's talk about optimization.

    Metal_Man88 and I have been working to improve the SQL performance, which has been the big bottleneck so far. SS and I put up KwikiData, which removes the need to manually parse the MediaWiki to get at the scanbox data. After that point, the only way to make things faster is to grab all the scanboxes at once. While this makes individual console pages faster, there's now incongruency between the old way (to get the list of games by seeing what's in what category) and the list in the table. I want to solve this incongruency before moving on so that nothing gets lost in the conversion. This will also help speed up the front page: if I can make it so both methods return identical lists, or the latter method returns more releases that we want to put in, I could make the front page code process everything in one go rather than one console at a time. (This is ging to be a series of questions. Please bear with me on this.)

    Right now the only thing left on the category side are Sonic games and albums, redirects, and a handful of exceptions. I can't cross wikis without adding a lot of code and another database and KwikiData instance, so what should we do about Sonic games and albums, just let them slide or copy over all the files?

    The redirects are another issue that depends on the answer to the above question.

    The other things left:
    • Are 3D Space Harrier and 3D Super Hang-On eShop games?
    • How should we scanbox the SC-3000 magazine BASIC games like Candy Kid?
    • Does the Hikaru arcade board use removable media (ROM carts, etc.)? The most I could find was an auction for a Planet Harriers JP PCB that suggested it did not
    • What about the Aurora board? I couldn't find any PCBs for this board and System 16 doesn't have info
    • What about Sega G80 and Sega System E and Europa (Sega Rally 3 and GRID) boards?
    • Do we know if Company of Heroes 2 will receive a retail release yet? What about the Japanese version of Rusty Hearts?
    • Is Godsrule: War of Mortals for PC confirmed to be web browser only?

    Thanks.
     
  7. Black Squirrel

    Black Squirrel

    Fifty paces towards the sun, cooking beets. Wiki Sysop
    4,947
    0
    18
    Northumberland, England
    The soil knows all.
    (I never thought your system was slow (at least not compared to the wikis themselves))

    yes
    there's no boxes to scan, so yeah, nothing to do

    as an aside, these games can be created through SC-3000 emulators (spoilers blueMSX is weirdly one of the better options) but it's extremely tedious to use the BASIC prompt (especially when syntax errors ruin everything) . If there's a way of converting modern txt files (or whatever) into SC-3000 compatible disk images (in which the games can be booted by an emulator), that would be sweet - these are games which don't need dumping to be preserved so I'm surprised there hasn't been a drive to build up a collection (it's the same for dozens of computers from back in the day too)
     
  8. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    I'm not sure if that depends on our experience or not. I'm connecting to Sega Reetro directly (rather than through CloudFlare) so it's running at an ok speed for me. And while I do agree the individual console pages on my scan system are fast, it's the front page that is not...

    Thanks.

    So just not have these with a scanbox? Ok then...

    I wonder if we could OCR these magazine pages, shove the result in a text file, and somehow hack an emulator to manually type in the text file contents, which we could then save to a disc image... Bonus points if we can get the key-in to stop if bASIC reports an error


    I'll also ask some of the above questions on the Sega Retro forum, but the Sonic question is something that should be addresseed so I can figure out what to do about the other redirects.(see next post)
     
  9. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    On secound thought I'll ask the redirect question instead since I found there are only three cases.

    Right now the scan system code follows redirects for titles that are different on one system than the page for all other systems have. These redirects are categorized (most, if not all, by me :/ ), so they show up in the scan database. As the releases template now automatically categorizes pages, this means these games are now listed twice both on our category page and in the scan system. If I choose to get rid of following redirects entirely, these games, which are still categorized, will show up as having no scans. As it turns out, there are only three such redirect pages:What should we do about these? Should we just drop the categories from the redirect pages? If so, I can also get rid of the redirect following code outright, and the Sonic games will continue to show up in the scan database as not having scans available. Or should we do something different entirely? Thanks.


    Also I should probably stop calling it the "scan system" and start calling it a "scan tracker"; it sounds less awkward but :/
     
  10. Black Squirrel

    Black Squirrel

    Fifty paces towards the sun, cooking beets. Wiki Sysop
    4,947
    0
    18
    Northumberland, England
    The soil knows all.
    I wouldn't bother categorising redirects and tracking them - it makes things slightly inaccurate, but it just adds to the workload for very little gain.

    Strictly speaking, unless you're listing every release under its localised name things are inaccurate anyway - for example, German scans of The Lion King don't exist - for maximum accuracy it would have to be listed as Der König der Löwen. You'd have to make five exceptions on that page alone, and since most games vary between regions, you'd have to go through almost every game page on Sega Retro.

    I might jump the gun and give Zoom 909 its own page, so that'll address that issue
     
  11. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Thanks; that settles that, and I'll just remove the redirect following. The Sonic part can wait.

    Now for the other side: the pages indexed in the database but not here. Mainly this consists of accessories and console pages. Should we start tracking these too? If so, should they be listed with the games or separate? Or will I need to add a flag that tells us not to track scans of those, or not have those use Scanbox at all? Thanks.
     
  12. Black Squirrel

    Black Squirrel

    Fifty paces towards the sun, cooking beets. Wiki Sysop
    4,947
    0
    18
    Northumberland, England
    The soil knows all.
    Accessories are worth doing, though it will be tricky to work out what constitutes a "complete" set of scans. Most boxes will have six sides that need scanning, but some may only need four or five, since there's not much consistency amongst third-party stuff. The other point to make is not all accessories will have scans - bundled controllers is the prime example

    Consoles are more awkward. tbh I'm not keen on how we're documenting all the different models/bundles right now - the solution I came up with a few months ago kind-of works... but I'm not convinced shoving everything on one page is the best plan, particularly if we start getting detailed info on contents and release dates and prices. It will almost certainly confuse the tracker too in its current state, and we're also not very consistent right now.
     
  13. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    I'm not entirely sure either, but the scenario is this:
    In order for a scanbox to be listed, the console= value in the template has to match the category name exactly, and right now the categories are hardcoded to be Cateogry:(console)_games. I could alter things to show everything on one page and use color coding to separate accessories from games, but I would have to make things more complicated to merge multiple lists (I have a potential idea that I could get to later).
    Or we could put the accessories and games on separate pages, say "Mega Drive games" and "Mega Drive accessories", etc. The issue then is that the console= wouldn't necessarily match, and I would need extra information if I wanted to make the front page faster, because now I have no way of differentiating an accessory from a game. Of course, I could use string comparisons on the table...

    I'll probably map out something tomorrow morning, but I know it will involve no longer automatically trying to discern the list of consoles through a filter and just specifying the list of consoles manually. Until now I enjoyed the luxury of the tracker automatically producing the list of consoles to track, but albums already give us special-casing and it's going to get worse as I add more things (see above). In that case, I would have my configuration file have a pair (category name, scanbox= field) and I could take the latter approach... I'll map it out tomorrow
     
  14. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Okay, let's see: let's say I manually state which categories to watch using pairs of the form
    Code (Text):
    1.      { "category name": "scanbox= text" }
    For the front page, I would need to do the following
    1. Read all scanboxes
    2. Read all page names for all categories
    3. For each scanbox, determine the values and add them to the stats for the category it belongs to
    4. Produce the full report
    For individual report pages, I would need to do the following
    1. Read all scanboxes for the given console= value
    2. Read all page names from the category
    3. For each scanbox, determine the values and add them to the stats
    4. Produce the full report
    My issue is whether or not this would require me to duplicate code, and what data structures I would need to use to make this sortable (for the report page). However, this looks attractive, and if I use a map[category string]ScanSet I think I can avoid the need to use separate code outright (just pass in all the category names or just the one category name as a list...).

    I'll just do that, if everybody thinks it'll do. The only cost is that this list needs to be edited if we add or drop categories, but since it will be in a configuration file, this won't be a hassle. In that case, here's what the category list will be:
    Code (Text):
    1. 32X accessories (?? - I doubt any exist)
    2. 32X games
    3. 32X systems
    4. Albums
    5. Amiga games
    6. Amstrad CPC games
    7. Apple II games
    8. Atari 2600 games
    9. Atari 5200 games
    10. Atari 8-bit family games
    11. Atari ST games
    12. Atomiswave games
    13. Aurora games
    14. BBC Micro games
    15. Beena accessories
    16. Beena games
    17. Beena systems
    18. Chihiro games
    19. Colecovision games
    20. Commodore 64 games
    21. Copera games (?? - should this just be merged with Pico?)
    22. DOS games
    23. Dragon 32 games
    24. Dreamcast accessories
    25. Dreamcast games
    26. Dreamcast systems
    27. Europa-R games
    28. FM Towns games
    29. Fujitsu FM-7 games
    30. G80 games
    31. Game Boy Advance games
    32. Game Boy Color games
    33. Game Gear accessories
    34. Game Gear games
    35. Game Gear systems
    36. Game.com games
    37. GameCube games
    38. Hikaru games
    39. Intellivision games
    40. LCD games
    41. Laserdisc hardware games
    42. Lindbergh games
    43. MSX games
    44. Mac games
    45. Master System accessories
    46. Master System games
    47. Master System systems
    48. Mega CD accessories
    49. Mega CD games
    50. Mega CD systems
    51. Mega CD 32X accessories (?? - I doubt any exist)
    52. Mega CD 32X games
    53. Mega CD 32X systems (?? - I doubt any such distributions exist)
    54. Mega Drive accessories
    55. Mega Drive games
    56. Mega Drive systems
    57. Mega LD accessories
    58. Mega LD games
    59. Mega LD systems (?? - assuming the Mega LD was sold separately)
    60. Mega Play games
    61. Mega-Tech games
    62. N-Gage games
    63. NAOMI games
    64. NAOMI 2 games
    65. NEC PC-6001 games
    66. NEC PC-8001 games
    67. NEC PC-8801 games
    68. NEC PC-9801 games
    69. NES games
    70. Neo Geo Pocket Color games
    71. Nintendo 3DS games
    72. Nintendo DS games
    73. PC accessories (Sega Logistics Service-manufactured controllers)
    74. PC games
    75. PC Engine CD-ROM² games
    76. Palm OS games
    77. Pico accessories (?? - were there any?)
    78. Pico games
    79. Pico systems
    80. PlayStation games
    81. PlayStation 2 accessories (Sega Logistics Service-manufactured controllers)
    82. PlayStation 2 games
    83. PlayStation 3 games
    84. PlayStation 3 systems (?? - various Yakuza bundles in Japan)
    85. PlayStation Portable games
    86. PlayStation Vita games
    87. Pocket PC games
    88. RingEdge games
    89. RingEdge 2 games
    90. RingWide games
    91. SC-3000 accessories
    92. SC-3000 games
    93. SC-3000 systems
    94. SG-1000 accessories
    95. SG-1000 games
    96. SG-1000 systems
    97. Saturn accessories
    98. Saturn games
    99. Saturn systems
    100. Sega Titan Video games
    101. Sharp MZ games
    102. Sharp X1 games
    103. Sharp X68000 games
    104. System 24 games
    105. System E games
    106. TI-99/4A games
    107. TRS-80 games
    108. TRS-80 CoCo games
    109. Triforce games
    110. TurboGrafx-16 games
    111. VFD games (?? - actually do we need this or should we just have LCD? or at all?)
    112. VIC-20 games
    113. Wii games
    114. Wii U games
    115. WonderSwan games
    116. Xbox games
    117. Xbox 360 games
    118. ZX Spectrum games
    Accessories without indiviudal scans can be dealt with individually; I have Template:NoScans for that... This is also a proposal for the system variations pages: put them all in a category like Category:Mega Drive systems, put the system variation name in the region= field (yes this is clumsier but avoids needing to have multiple entries or one category), and then it would be covered too. I can also edit this list to cover tings like our video scans...

    This leaves a number of scanboxes (such as the ones for the Football Manager titles) that are marked PC/Mac; not sure how to handle those apart from splitting nad having duplicates :/ Any other ideas? Also should the two games for PC Engine Super CD-ROM² games (Gain Ground SX and Bonanza Bros.) have the scanbox say (and thus be listed as) PC Engine CD-ROM² games?
     
  15. Scarred Sun

    Scarred Sun

    Be who you needed when you were younger Administrator
    7,542
    14
    76
    Tower 8 ️
    Welp, this.
    Reminder of my request for a manual column.
     
  16. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Yes, this is all leading up to the manual column. I want to get this polished and fast before breaking it more =P
     
  17. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Well I went ahead and made the switch on the scan server, and everything's MUCH faster now :) The front page takes between 1 and 3 seconds to load, whereas it used to take between 4 and 6. The only cost was that I had to split the Mega Anser and Miracle Piano pages into one for the accessory and one for the software because the system can't deal with a page being in both (and that would require a fair amount of change). I also changed it to use one database connection per page request rather than one globally, so there shouldn't be issues should the MySQL server drop and come back.

    I'll be polishing the system off either all day tomorrow or over the next few days (depending on how long it takes) and at the end I'll add tracking manual scans to the mix.

    Do you want to go with my proposal for dealing with consoles themselves? Is there anything I am missing from these categories? Did any games or categories disappear? Thanks.
     
  18. Black Squirrel

    Black Squirrel

    Fifty paces towards the sun, cooking beets. Wiki Sysop
    4,947
    0
    18
    Northumberland, England
    The soil knows all.
    answers

    - there aren't any 32X/CD 32X accessories. Or any Mega CD 32X consoles in boxes because that would make too much sense
    - copera should be merged with Pico
    - Pico accessories do exist

    also might I suggest different tables for accessories/consoles for the sake of readibility
     
  19. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    Thanks. It turns out all the Copera games are already listed under Pico, so that settles that. As for splitting by type, I'll see what I can do after settling everything else.
     
  20. Andlabs

    Andlabs

    「いっきまーす」 Wiki Sysop
    2,175
    0
    0
    Writing my own MD/Genesis sound driver :D
    The scan tracker is back up, and now tracks manuals too! Once again, feel free to report any bugs or make any suggestions (suggestions on how to not make it horizontally scroll while not line wrapping the console list would be nice). Also covers gaming books, since Black Squirrel recently did a bunch of them.

    The front page may take a few seconds to load. I'm not really sure anymore why it's not running as fast as it did two months ago, but whatever; at this point I imagine the data load will eventually make it not really matter :/ (maybe?...)