In the next day or two I'm going to try to optimize how segaret_scans talks to the MediaWiki database, so expect intermittent failures.
I probably have more, but these are the three I could find. Home Alone 2: Lost in New York Frogger Cal Ripken Jr. Baseball
Working on refactoring the scan system before adding manuals to the fray; should I just remove the Incomplete stat and mark all incomplete scans with Bad? It just makes the logic more complex, however there are a handful of games where I put item text with no item. If I should remove Incomplete, what should we do about those? EDIT - looks like some of my refactoring is causing MySQL to shit itself, so the scan system will be down until further notice; the question above is still open
I think it's fine as it is, although it's still picking up systems that will never have box scans. "Incomplete" doesn't turn up that often, but it can still be useful That's assuming you're planning to treat manuals as something different from "media", otherwise most of the games on this site will be marked as "incomplete"
I was just about to get to the invalid systems part. And right; SS was talking about a manual column so I will probably do that. The program seems to have fixed itself after I fixed a snag elsewhere; if the database breaks again please let me know
Okay let's talk about optimization. Metal_Man88 and I have been working to improve the SQL performance, which has been the big bottleneck so far. SS and I put up KwikiData, which removes the need to manually parse the MediaWiki to get at the scanbox data. After that point, the only way to make things faster is to grab all the scanboxes at once. While this makes individual console pages faster, there's now incongruency between the old way (to get the list of games by seeing what's in what category) and the list in the table. I want to solve this incongruency before moving on so that nothing gets lost in the conversion. This will also help speed up the front page: if I can make it so both methods return identical lists, or the latter method returns more releases that we want to put in, I could make the front page code process everything in one go rather than one console at a time. (This is ging to be a series of questions. Please bear with me on this.) Right now the only thing left on the category side are Sonic games and albums, redirects, and a handful of exceptions. I can't cross wikis without adding a lot of code and another database and KwikiData instance, so what should we do about Sonic games and albums, just let them slide or copy over all the files? The redirects are another issue that depends on the answer to the above question. The other things left: Are 3D Space Harrier and 3D Super Hang-On eShop games? How should we scanbox the SC-3000 magazine BASIC games like Candy Kid? Does the Hikaru arcade board use removable media (ROM carts, etc.)? The most I could find was an auction for a Planet Harriers JP PCB that suggested it did not What about the Aurora board? I couldn't find any PCBs for this board and System 16 doesn't have info What about Sega G80 and Sega System E and Europa (Sega Rally 3 and GRID) boards? Do we know if Company of Heroes 2 will receive a retail release yet? What about the Japanese version of Rusty Hearts? Is Godsrule: War of Mortals for PC confirmed to be web browser only? Thanks.
(I never thought your system was slow (at least not compared to the wikis themselves)) yes there's no boxes to scan, so yeah, nothing to do as an aside, these games can be created through SC-3000 emulators (spoilers blueMSX is weirdly one of the better options) but it's extremely tedious to use the BASIC prompt (especially when syntax errors ruin everything) . If there's a way of converting modern txt files (or whatever) into SC-3000 compatible disk images (in which the games can be booted by an emulator), that would be sweet - these are games which don't need dumping to be preserved so I'm surprised there hasn't been a drive to build up a collection (it's the same for dozens of computers from back in the day too)
I'm not sure if that depends on our experience or not. I'm connecting to Sega Reetro directly (rather than through CloudFlare) so it's running at an ok speed for me. And while I do agree the individual console pages on my scan system are fast, it's the front page that is not... Thanks. So just not have these with a scanbox? Ok then... I wonder if we could OCR these magazine pages, shove the result in a text file, and somehow hack an emulator to manually type in the text file contents, which we could then save to a disc image... Bonus points if we can get the key-in to stop if bASIC reports an error I'll also ask some of the above questions on the Sega Retro forum, but the Sonic question is something that should be addresseed so I can figure out what to do about the other redirects.(see next post)
On secound thought I'll ask the redirect question instead since I found there are only three cases. Right now the scan system code follows redirects for titles that are different on one system than the page for all other systems have. These redirects are categorized (most, if not all, by me :/ ), so they show up in the scan database. As the releases template now automatically categorizes pages, this means these games are now listed twice both on our category page and in the scan system. If I choose to get rid of following redirects entirely, these games, which are still categorized, will show up as having no scans. As it turns out, there are only three such redirect pages: Puyo Puyo Sun Ketteiban (the PS1 port, whose last release was published by Sega) -> Puyo Puyo Sun Switch (the PS2 port/remake/whatever, which was only released in Japan) -> Panic! Zoom 909 (the SG-1000 version, which was only released in Japan) -> Buck Rogers: Planet of Zoom What should we do about these? Should we just drop the categories from the redirect pages? If so, I can also get rid of the redirect following code outright, and the Sonic games will continue to show up in the scan database as not having scans available. Or should we do something different entirely? Thanks. Also I should probably stop calling it the "scan system" and start calling it a "scan tracker"; it sounds less awkward but :/
I wouldn't bother categorising redirects and tracking them - it makes things slightly inaccurate, but it just adds to the workload for very little gain. Strictly speaking, unless you're listing every release under its localised name things are inaccurate anyway - for example, German scans of The Lion King don't exist - for maximum accuracy it would have to be listed as Der König der Löwen. You'd have to make five exceptions on that page alone, and since most games vary between regions, you'd have to go through almost every game page on Sega Retro. I might jump the gun and give Zoom 909 its own page, so that'll address that issue
Thanks; that settles that, and I'll just remove the redirect following. The Sonic part can wait. Now for the other side: the pages indexed in the database but not here. Mainly this consists of accessories and console pages. Should we start tracking these too? If so, should they be listed with the games or separate? Or will I need to add a flag that tells us not to track scans of those, or not have those use Scanbox at all? Thanks.
Accessories are worth doing, though it will be tricky to work out what constitutes a "complete" set of scans. Most boxes will have six sides that need scanning, but some may only need four or five, since there's not much consistency amongst third-party stuff. The other point to make is not all accessories will have scans - bundled controllers is the prime example Consoles are more awkward. tbh I'm not keen on how we're documenting all the different models/bundles right now - the solution I came up with a few months ago kind-of works... but I'm not convinced shoving everything on one page is the best plan, particularly if we start getting detailed info on contents and release dates and prices. It will almost certainly confuse the tracker too in its current state, and we're also not very consistent right now.
I'm not entirely sure either, but the scenario is this: In order for a scanbox to be listed, the console= value in the template has to match the category name exactly, and right now the categories are hardcoded to be Cateogryconsole)_games. I could alter things to show everything on one page and use color coding to separate accessories from games, but I would have to make things more complicated to merge multiple lists (I have a potential idea that I could get to later). Or we could put the accessories and games on separate pages, say "Mega Drive games" and "Mega Drive accessories", etc. The issue then is that the console= wouldn't necessarily match, and I would need extra information if I wanted to make the front page faster, because now I have no way of differentiating an accessory from a game. Of course, I could use string comparisons on the table... I'll probably map out something tomorrow morning, but I know it will involve no longer automatically trying to discern the list of consoles through a filter and just specifying the list of consoles manually. Until now I enjoyed the luxury of the tracker automatically producing the list of consoles to track, but albums already give us special-casing and it's going to get worse as I add more things (see above). In that case, I would have my configuration file have a pair (category name, scanbox= field) and I could take the latter approach... I'll map it out tomorrow
Okay, let's see: let's say I manually state which categories to watch using pairs of the form Code (Text): { "category name": "scanbox= text" } For the front page, I would need to do the following Read all scanboxes Read all page names for all categories For each scanbox, determine the values and add them to the stats for the category it belongs to Produce the full report For individual report pages, I would need to do the following Read all scanboxes for the given console= value Read all page names from the category For each scanbox, determine the values and add them to the stats Produce the full report My issue is whether or not this would require me to duplicate code, and what data structures I would need to use to make this sortable (for the report page). However, this looks attractive, and if I use a map[category string]ScanSet I think I can avoid the need to use separate code outright (just pass in all the category names or just the one category name as a list...). I'll just do that, if everybody thinks it'll do. The only cost is that this list needs to be edited if we add or drop categories, but since it will be in a configuration file, this won't be a hassle. In that case, here's what the category list will be: Spoiler Code (Text): 32X accessories (?? - I doubt any exist) 32X games 32X systems Albums Amiga games Amstrad CPC games Apple II games Atari 2600 games Atari 5200 games Atari 8-bit family games Atari ST games Atomiswave games Aurora games BBC Micro games Beena accessories Beena games Beena systems Chihiro games Colecovision games Commodore 64 games Copera games (?? - should this just be merged with Pico?) DOS games Dragon 32 games Dreamcast accessories Dreamcast games Dreamcast systems Europa-R games FM Towns games Fujitsu FM-7 games G80 games Game Boy Advance games Game Boy Color games Game Gear accessories Game Gear games Game Gear systems Game.com games GameCube games Hikaru games Intellivision games LCD games Laserdisc hardware games Lindbergh games MSX games Mac games Master System accessories Master System games Master System systems Mega CD accessories Mega CD games Mega CD systems Mega CD 32X accessories (?? - I doubt any exist) Mega CD 32X games Mega CD 32X systems (?? - I doubt any such distributions exist) Mega Drive accessories Mega Drive games Mega Drive systems Mega LD accessories Mega LD games Mega LD systems (?? - assuming the Mega LD was sold separately) Mega Play games Mega-Tech games N-Gage games NAOMI games NAOMI 2 games NEC PC-6001 games NEC PC-8001 games NEC PC-8801 games NEC PC-9801 games NES games Neo Geo Pocket Color games Nintendo 3DS games Nintendo DS games PC accessories (Sega Logistics Service-manufactured controllers) PC games PC Engine CD-ROM² games Palm OS games Pico accessories (?? - were there any?) Pico games Pico systems PlayStation games PlayStation 2 accessories (Sega Logistics Service-manufactured controllers) PlayStation 2 games PlayStation 3 games PlayStation 3 systems (?? - various Yakuza bundles in Japan) PlayStation Portable games PlayStation Vita games Pocket PC games RingEdge games RingEdge 2 games RingWide games SC-3000 accessories SC-3000 games SC-3000 systems SG-1000 accessories SG-1000 games SG-1000 systems Saturn accessories Saturn games Saturn systems Sega Titan Video games Sharp MZ games Sharp X1 games Sharp X68000 games System 24 games System E games TI-99/4A games TRS-80 games TRS-80 CoCo games Triforce games TurboGrafx-16 games VFD games (?? - actually do we need this or should we just have LCD? or at all?) VIC-20 games Wii games Wii U games WonderSwan games Xbox games Xbox 360 games ZX Spectrum games Accessories without indiviudal scans can be dealt with individually; I have Template:NoScans for that... This is also a proposal for the system variations pages: put them all in a category like Category:Mega Drive systems, put the system variation name in the region= field (yes this is clumsier but avoids needing to have multiple entries or one category), and then it would be covered too. I can also edit this list to cover tings like our video scans... This leaves a number of scanboxes (such as the ones for the Football Manager titles) that are marked PC/Mac; not sure how to handle those apart from splitting nad having duplicates :/ Any other ideas? Also should the two games for PC Engine Super CD-ROM² games (Gain Ground SX and Bonanza Bros.) have the scanbox say (and thus be listed as) PC Engine CD-ROM² games?
Yes, this is all leading up to the manual column. I want to get this polished and fast before breaking it more =P
Well I went ahead and made the switch on the scan server, and everything's MUCH faster now The front page takes between 1 and 3 seconds to load, whereas it used to take between 4 and 6. The only cost was that I had to split the Mega Anser and Miracle Piano pages into one for the accessory and one for the software because the system can't deal with a page being in both (and that would require a fair amount of change). I also changed it to use one database connection per page request rather than one globally, so there shouldn't be issues should the MySQL server drop and come back. I'll be polishing the system off either all day tomorrow or over the next few days (depending on how long it takes) and at the end I'll add tracking manual scans to the mix. Do you want to go with my proposal for dealing with consoles themselves? Is there anything I am missing from these categories? Did any games or categories disappear? Thanks.
answers - there aren't any 32X/CD 32X accessories. Or any Mega CD 32X consoles in boxes because that would make too much sense - copera should be merged with Pico - Pico accessories do exist also might I suggest different tables for accessories/consoles for the sake of readibility
Thanks. It turns out all the Copera games are already listed under Pico, so that settles that. As for splitting by type, I'll see what I can do after settling everything else.
The scan tracker is back up, and now tracks manuals too! Once again, feel free to report any bugs or make any suggestions (suggestions on how to not make it horizontally scroll while not line wrapping the console list would be nice). Also covers gaming books, since Black Squirrel recently did a bunch of them. The front page may take a few seconds to load. I'm not really sure anymore why it's not running as fast as it did two months ago, but whatever; at this point I imagine the data load will eventually make it not really matter :/ (maybe?...)