don't click here

Things to mirror

Discussion in 'General Sega Discussion' started by Black Squirrel, Mar 6, 2020.

  1. RyogaMasaki

    RyogaMasaki

    0xffffffff Oldbie
    At the very least, that confirms beyond a shadow of a doubt that that the 32X example code (which we've already had from other sources for a long time) came entirely from Sega. Obviously some of the headers are clearly marked as written by Sega, but one in particular, maincpu.inc, which is related to the Mega CD (my particular area of dev interest), was particularly intriguing to me and made me wonder if the example code we had was partially from a third party source. That file has this comment:

    What he's referring to here is use of the Main CPU side Boot ROM library, which was officially undocumented yet was used in a number of production Japanese games. The fact that these calls were present in only Japanese games* always made me think that maybe the Boot Library documentation was never translated to English and was only made available to Japanese devs. Indeed, this source file, which is now definitely from a Sega of America source, indicates that the dev had no knowledge of this area of RAM being usable and was working from an example in a tech bulletin (which is, indeed, the only English translated references to using the Boot Library code in games and which discusses it in such a way that it was common knowledge).

    I know SOJ and SOA didn't always have the best relationship, but this is fascinating that even internal SOA didn't have the full story on what was available in the Mega CD hardware.

    Aside from all that, on the JUN16_95 disc, in nov01_94.dts/32x/samples/gnusmple/md/include, a number of the headers are new to me. They don't offer any deep new insight, but it's always great to get official, internal source code, especially ones with hardware defines.

    * Actually, in that twitter thread I posted, the Blastem author recently pointed out that a couple of Psygnosis games also use some Bootlib code, which is intriguing. It doesn't quite destroy the "Japan devs only" theory, as there are *far* more Japanese games that make Bootlib calls than those couple of Psygnosis games (perhaps they had a close contact at SOJ, perhaps they had some example code from SOJ they copied from), but it does thicken the plot.

    (Edit for clarity)
     
    Last edited: Oct 10, 2022
    • Informative Informative x 2
    • List
  2. JaxTH

    JaxTH

    Pudding Deity Oldbie
    10,360
    588
    93
    Los Angeles
    Jack shit.
    It's linking it's source as Assembler Games, so it needs to be mirrored ASAP.

    I'm sure it's somehow completely SOA's fault. :V
     
  3. Overlord

    Overlord

    Now playable in Smash Bros Ultimate Moderator
    19,233
    969
    93
    Long-term happiness
    It still boggles my mind just how much the two seemed to hate each other in that you ended up with effectively self-sabotage of the company. Both SoJ and SoA are guilty of this.
     
  4. Pirate Dragon

    Pirate Dragon

    Member
    539
    109
    43
    Psygnosis were pretty involved in Japan, they developed their first CD-ROM games for FM-Towns, some of which were ported to Mega CD. It's plausible that they were working with Japanese documentation.

    Edit: A good article about that.
     
    Last edited: Oct 11, 2022
    • Informative Informative x 2
    • List
  5. Ted909

    Ted909

    the future kick your ass Member
    480
    292
    63
    Without wanting to risk another pointless blame game, I do think Sega's top management in Japan warring with themselves (Shoichiro Irimajiri's bad blood with Hayao Nakayama, the power struggle surrounding the Dreamcast, etc) rather than America is the stronger argument to be made, alongside the latter's own questionable decisions.

    I'm sure there were some misgivings about SOA on their part, particularly surrounding the failure of the 32X, but not to the extent that Kalinske and co. make out in retrospect with their whole "jealously" narrative.
     
  6. Asagoth

    Asagoth

    Behold! The mighty, the flawless, salted cod eater Member
    494
    93
    28
    Portugal
    wiki stuff... and a beer... or two... or more...
    Done... ;)
     
  7. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,584
    2,481
    93
    Northumberland, UK
    steamboat wiki
    hoo boy
    https://archive.org/details/@revengeofthehubz?&sort=-publicdate

    Here's the story: the Video Game History Foundation scanned a truckload of magazines earlier in the year, and I didn't have time to go looking for uploads. Instead, magazines have been trickling through Retromags and I've just been mirroring those versions. Now the non-Retromags-edited versions on archive.org. And they're better, but they're not.

    We probably want these versions, because as I've said in the past, it's better to document "what's actually there" than "what we think should be there". Retromags cleans up pages which makes them nice and pretty, but not necessarily indicitive of what the magazine actually looked like. However, no scanner is perfect, and magazines deteriorate with age. I also don't want to be some kind of arbiter or preservation because a) I don't know the best methods of preserving documents in digital form and b) 99% of the time I just want to be able to read them.

    aka I don't know if it's worth replacing the PDFs already on Retro CDN with these 600dpi versions, least because it's a bit of a faff replacing files that large. But someone else can make that call.


    This also leads me to the other factor - OCR, or optical character recognition. In addition to showing the image, you can get a PDF to store text data, and tools exist that will try to convert the former into the latter (i.e. you can "highlight" areas of text in a scan and copy-paste). Those archive.org PDFs have been given this treatment, most of our PDFs haven't, because I only started thinking about this a year ago and never put a plan together.

    It doesn't matter that much but it can help Google search results. idk needs more thought.
     
  8. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,584
    2,481
    93
    Northumberland, UK
    steamboat wiki
    Wow this became an issue far quicker than I expected:

    GameWEEK

    Literally half the GameWEEK scans in that set have been put together "incorrectly". Basically, supplements are in the middle of the PDF, which breaks the page numbering scheme. We typically treat supplements as separate material because they can be huge and annoying. But my current methods of splitting and then recompiling the PDF would lose all the OCR data, which seems... avoidable.

    Tesseract is the OCR engine of choice (or at least the one archive.org uses), and I found a suitable tool, OCRmyPDF to add the data back in. So now I have a process of converting a bunch of JPEGs to PDFs, with OCR data without losing image quality. Hoo. Ray.


    But can I suggest, mystery developers - the process of getting these tools is pointlessly complicated. There isn't just a simple Windows command line exe, you need to download Python, Ghostscript and Tesseract separately and make sure all the environment variables line up. Good luck teaching this to your average office worker; you know, the people most likely to benefit from such tools.
     
  9. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,584
    2,481
    93
    Northumberland, UK
    steamboat wiki
    Incorrection

    https://segaretro.org/File:GameWEEK_US_Supplement_2000-10-16.pdf
    (the software is struggling to render thumbnails when the source image is so big)

    Sega ran the ad through GameWEEK, which I guess makes sense since that was an industry business/retail publication.
     
  10. kitsunebi

    kitsunebi

    Member
    205
    85
    28
    Japan
    Anybody know who "Nemesisretrocdn" is? They've been uploading lots of stuff to RetroCDN (mostly Nintendo-exclusive stuff, for god only knows why reasons), and I just noticed they uploaded a file of mine. I've basically got just one request regarding my files: don't redistribute them in an altered form (I specifically ask for them not to be redistributed as a PDF in every listing I make). And wouldn't you know it, this file has been uploaded to RetroCDN as a PDF. And not just any PDF, but the crappy PDF auto-generated by the Internet Archive's software. So rather than the 60 MB CBR (which I have no problem with being redistributed), they have uploaded a super-compressed 5 MB PDF file. Sigh. Guys, I've already got all of my files mirrored in multiple places, so they're safe - there's really no need to "mirror" them to RetroCDN. Especially if people are going to upload garbage versions of the files. (And what does a Sega/NEC site need Famicom-exclusive strategy guides for anyway?)
     
  11. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,584
    2,481
    93
    Northumberland, UK
    steamboat wiki
    Yes I noticed some of that, but haven't had the patience to clean it up - point me in the direction of your least favourite uploads
     
  12. kitsunebi

    kitsunebi

    Member
    205
    85
    28
    Japan
    Right now, the only scan of mine they've uploaded is a short supplement from Family Computer Magazine for Super Chinese 2. It's an older scan of mine from when I used to upload to Retromags, but I know it was pulled from my Internet Archive account because the size of the file they uploaded is the same as the auto-generated super-compressed PDF that Archive creates when a CBR is uploaded. I'm less worried about this particular supplement than I am that Nemesisretrocdn will strip-mine my account and upload garbage PDFs of everything that isn't already up at RetroCDN (especially since they seem entirely unconcerned with whether the files have any relevance there in the first place.) Unfortunately, I don't see any way to contact RetroCDN uploaders, so I'm not sure what I can do to stop it.
     
  13. Gestalt

    Gestalt

    Sphinx in Chains Member
    @kitsunebi Sorry, I'm also guilty of this. It won't happen again, promised.
     
  14. Asagoth

    Asagoth

    Behold! The mighty, the flawless, salted cod eater Member
    494
    93
    28
    Portugal
    wiki stuff... and a beer... or two... or more...
    We really don't need Nintendo-exclusive stuff... and "garbage PDFs"... but we really have no other choice than to use the PDF format for reasons explained here...
     
  15. kitsunebi

    kitsunebi

    Member
    205
    85
    28
    Japan
    While I disagree with a few of the things said on that page, I understand that the Wiki is technically constructed in such a way that makes using PDFs necessary. Which is why I don't upload my files to RetroCDN. Of course, I've literally only ever scanned one Sega-exclusive magazine anyway, and RetroCDN already has a (different) copy of that one. So I don't think most of my scans belong there regardless.

    As for multi-format mags like Famitsu, I don't care if relevant pages are extracted and turned into a PDF for use in the wiki, but it does irk me when someone uploads the entire mag as a PDF, since I specifically ask that the files not be distributed that way. As I said before, my mags are already mirrored in multiple locations, so uploading the mag to RetroCDN "so that it isn't lost" is unnecessary. And if the reason PDFs are necessary on Retro is indeed so that individual relevant pages can be viewed in the wiki software, then only pages useful to the wiki really need uploading anyway, right?

    I and almost all other scanners have our own reasons for preferring the CBR format which I won't get into here, and in addition to those reasons, some of my scans will simply not display correctly as a PDF if directly converted, since they were not edited with that software in mind. It's for these reasons that I ask not to have complete copies of my scans redistributed in PDF format (redistributing the original CBR files is fine). I realize I can't actually enforce that, this being the Internet and all. All I can do is ask nicely and hope my wishes are respected. I reached out here simply because I know of no way to ask whoever "nemesisretrocdn" is, but thought that perhaps someone else might (or maybe they use the forums under a different name.)

    All of that aside, though...ANYONE uploading stuff to RetroCDN that was found on the Internet Archive needs to be aware that most magazine scans are originally released in CBR/CBZ format. When a CBR is uploaded to Archive, it auto-generates a VERY compressed PDF file (what I referred to as a "garbage PDF", which I believe is what's used for their preview viewing software on their site so that it loads quickly. But that file should NEVER be downloaded. If you plan to upload a PDF to Retro, you need to download the CBR and convert it to PDF using the lossless conversion software that's been recommended here before (I forget the name). The only time downloading a PDF from the Internet Archive is appropriate is if there ISN'T a CBR file, meaning that the original file that was uploaded was already a PDF.
     
    • Informative Informative x 1
    • List
  16. Pirate Dragon

    Pirate Dragon

    Member
    539
    109
    43
    I don't think that I have the required permissions to delete those from Retro CDN, but if I did, I would use the following rule, clearly stated on the main page of Retro CDN;

    So yeah, Nintendo specific stuff such as manuals and guides shouldn't generally be uploaded, but if for some reason it's required for a source for a wiki article then that's allowed. If non-obvious wiki related stuff is being uploaded and not being used as a source for an article, then it's clearly out of scope for Retro CDN and should be deleted.

    As for only including specific pages from multi-format magazines which are in the scope of Retro CDN ... other format specific sites have done that before, and it's massively irritating if you're researching something and the specific page that you need is missing because it doesn't contain anything about the format that the original uploader was interested in ... we don't want to be like that for people researching about other formats. Also, whilst we may not cover certain companies, we may, for example want to compare release dates, prices etc between multi-format games. I think that including the whole magazine is just a basic courtesy for other researchers who may be researching something that you never even considered was important.

    Also agreed, if downloading off of IA, download the original files and make a PDF out of that.
     
  17. JaxTH

    JaxTH

    Pudding Deity Oldbie
    10,360
    588
    93
    Los Angeles
    Jack shit.
    This.

    In the early 2000s single page scans were everywhere and ALSO watermarked by whatever website it was for and the Sonic community was lamenting it then because it doesn't help anybody.

    Find something of interest in your research and learn of some scans? Too bad, because the person who did the scans was only interested in the Amiga and only scanned and watermarked the three pages relative to them while the magazine issue itself has been discontinued for 10 years or something like that and no one you knew had it in the first place.

    You never know what you or other people may need that will benefit you as well somewhere down the line.
     
  18. Black Squirrel

    Black Squirrel

    no reverse gear Wiki Sysop
    8,584
    2,481
    93
    Northumberland, UK
    steamboat wiki
    There's a widespread set of Sonic the Comic issues where only the "useful" pages were included, and they're appalling. However, because it looks "complete" and most of the readers are only there for the stories, nobody has scanned a replacement set. Which is bad, because the scans aren't great, and the bits that were scrapped, such as ads, and reprints, could be very useful to us.

    And yes it's curtosy:

    [​IMG]

    I know for example that while Sonic features on the cover of this issue of i-D, there's not a lot of content inside the magazine that's of use to us. But it's going to be useful to someone - it's a snapshot of popular culture circa 1993 which will help explain why history is the way it is.
     
  19. kitsunebi

    kitsunebi

    Member
    205
    85
    28
    Japan
    Indeed, allowing incomplete magazines to be downloaded would be a travesty. But as I already had stated that my one and only request (included in every listing on my Internet Archive page) was that my scans not be redistributed as PDFs, I was suggesting (not entirely seriously in all honesty) the possibility that relevant pages could be used for the wiki if someone felt they were really needed. I was not suggesting that they be made available to download. Whether pages or the entire magazine, it's the downloadability (redistribution) of the altered format of the file which I've specifically asked not to occur. I suppose it could be another limitation of the wiki software that it is impossible for it to utilize PDF images without also making that PDF publicly available to download. If that's the case, all I can do is refer back to my original request and hope it's respected.

    That's also why I have all of my magazines available in multiple places, including the Internet Archive, which is probably the highest-profile site for any potential researcher to use. They can either download the entire mag or browse the pages online.

    Ironically, the initial problem that prompted me to post here as well as the specific case cited above with the Sonic comics and book both illustrate why I don't care for my scans to be redistributed as PDFs. Once the scans are made available to download in an altered form (such as the compressed PDF auto-generated at Archive or the incomplete Sonic comics), many people will download those without realizing that they aren't getting "the real deal." And Black Squirrel is exactly right - most people won't even notice or care. I put in countless hours and expense creating something I can be proud of and share it freely with the world, my only request being that it be left in its original format when re-shared. But all it takes is one person uploading a compromised version somewhere else, and from that point on the compromised version will be downloaded and possibly further re-shared elsewhere down the line to the point where the compromised version could conceivably be more prominently available than the original.

    Anyway, I'll stop there on the subject. I'm sure to most people I sound like a diva trying to be too controlling, and I don't expect everyone to agree with or even understand why I would want my scans left intact in their intended format when being passed around to be downloaded. Someone (Black Squirrel?) has already removed the specific scan that prompted my post, so they have my thanks. I guess there's no way to appeal to the person who uploaded it or to monitor what gets uploaded to RetroCDN in the future, so all I can do is keep one eye open going forward, I suppose.
     
  20. JaxTH

    JaxTH

    Pudding Deity Oldbie
    10,360
    588
    93
    Los Angeles
    Jack shit.
    They only way would be to sign up to the CDN and go to their talk page. Whether they'll respond is a different matter.
     
    • Informative Informative x 1
    • List