ROM Properties Page shell extension

Discussion in 'Technical Discussion' started by GerbilSoft, Jul 20, 2016.

  1. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    So I'm getting close to preparing a release build. Here's a short list of what I want to finish up before releasing rom-properties-0.8 (Beta 1): (mostly copied from the GitHub issues page)
    • Use a modified version of pngcheck to verify that downloaded PNG images are valid. While there aren't any known PNG exploits at the moment, historically libpng (and other image format libraries) have had several remote execution exploits. (The TIFF format was notorious for this, as most PSP hackers know.) pngcheck verifies each chunk by itself without using libpng, so if any chunk is invalid, the image will be discarded.
    • Implement a system-specific backend class for rp_image. Currently, rp_image works as a "system-independent" image class. It stores the format type and raw data of the image, plus a palette for 256-color images. It then has to be copied over to a system-specific image class (QImage, HBITMAP), which adds overhead. By using a system-specific backend, I can have the image data stored in a QImage or GDI+ bitmap directly without any extra conversion steps. (GDI+ is needed for proper alpha-transparency on Windows.)
    • Fix an issue on Windows XP where thumbnail downloading causes Explorer to freeze.
    • Add a system logo to the properties page. Currently, there's no actual indication as to what system the ROM is for. Adding a logo (which may be region-specific depending on implementation) will help, plus the game icon (for ROMs that support it) and internal title(s).
    • Wii: Identify the system update version. This requires decrypting the update partition and looking for the system menu WAD file. (The decryption keys will not be included in the main distribution, but I'll post a configuration file with them later.)
    • Use std::unique_ptr instead of std::auto_ptr.
    • Identify the specific system type for MD ROMs, e.g. 32X, Pico, MCD. Also decode SMD-format ROMs for MD only.
    • Use manual "activation contexts" in the Windows build when compiling with gcc/MinGW-w64.
    I have a few other TODOs not currently published, and I'll probably add issues for those tonight.

    Systems and images currently supported:
    • Sega Mega Drive
    • Nintendo Game Boy (Color)
    • Nintendo Game Boy Advance
    • Nintendo DS(i) (DS game icon is displayed)
    • Nintendo GameCube (Disc scan is downloaded from GameTDB)
    • Nintendo Wii (Disc scan is downloaded from GameTDB)
    Does anyone else have any other suggestions for "must-have" functionality in the first release?
     
  2. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Today's status update:
    • Wii decryption is almost done. Just need to have it go through the FST for the Update Partition and determine which System Menu version is present.
    • I have some preliminary code for localized system names. For Mega Drive ROMs, if a ROM is single-region, then it will display "Sega Genesis" for US or "Sega Mega Drive" elsewhere. If it's multi-region, then it will check the system locale to see what the closest match is. Currently, I'm assuming that any country in North America uses "Sega Genesis" branding, countries listed on Mega Drive consoles in Asia with JP branding use JP, and everywhere else uses PAL/EU branding. (With an exception for South Korea, which has Samsung branding due to various restrictions in the 80s and 90s.) This will be extended to logos later on, which will be slightly more complicated due to Brazil having its own distinct Mega Drive branding.
    • Somewhat hackish support for alpha transparent thumbnails on Windows XP. IExtractImage doesn't fully support it, so I'm now using COLOR_WINDOW as the background color. IThumbnailProvider (Vista+) does fully support alpha transparency, and I plan on implementing it before the first release.
    • More unit tests for image loading and AES decryption to make sure nothing breaks with future updates.
    I set up a tracker bug for the first beta release (0.8) here: https://github.com/GerbilSoft/rom-properties/issues/16
     
  3. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    All that work on decryption, and here's what I have to show for it. :v:

    [​IMG]

    It now parses the update partition and determines the System Update version. It also shows the partition sizes now.
     
  4. Overlord

    Overlord

    ...did we just switch realities? Moderator
    17,538
    57
    28
    Berkshire, England
    Learning Cymraeg
    Looks like it's coming along nicely.
     
  5. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    I'm looking into decoding the "age ratings" section of Wii disc images. Wiibrew unfortunately doesn't list what each value corresponds to, though I suspect it's an age value. (Mapping the age to ESRB and CERO might be tricky.)

    If anyone has Wii disc images available (e.g. from an HDD used for USB Loader GX), can you dump the following ranges from your disc images and post them here in a code block? (including the game's name, region, and rating on the box)
    • 1:1 dump ("ISO"): 0x4E000 - 0x4E01F
    • WBFS: 0x24E000 - 0x24E01F

    If you're using Linux or have a Cygwin environment installed, you can run one of these commands:
    • dd if=wii-disc.1-to-1.iso bs=1 skip=319488 count=32 iflag=binary oflag=binary | hexdump -C
    • dd if=wii-disc.wbfs bs=1 skip=2416640 count=32 iflag=binary oflag=binary | hexdump -C
     
  6. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Today's update:
    • Added support for the little-used GameCube/Wii CISO sparse image format. It's significantly simpler than WBFS, but no one uses it. :v: (The PSP CISO format is completely different, as it uses zlib compression. I'll probably add support for that and Dolphin GCZ later.)
    • Started an FST test suite. For this, I'll need to extract original FSTs from all partitions on GameCube and Wii discs. WiiXplorer can do the extraction, but one at a time and using a bit of a clunky interface. I should be able to hack on a quick and dirty UI that automatically dumps the FSTs in response to pressing a button.
    With regards to PSP: I've never actually used a PSP, so I don't know if UMDs have a game icon like DS/3DS games or what. I do know that the discs use ISO-9660, and I wrote a simple ISO-9660 parser years ago (in VB6 :v:), so that shouldn't be a problem.
     
  7. Overlord

    Overlord

    ...did we just switch realities? Moderator
    17,538
    57
    28
    Berkshire, England
    Learning Cymraeg
    PSP games have an icon (I think these can be animated, from memory), a wallpaper (possibly optional?) and a music file, as well as some basic display text (seemingly optional). Here's the one from Wipeout Pure:

    [​IMG]
     
  8. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Today's status update:
    • I got a bit sidetracked with implementing CISO support in Nintendont. I'm also reworking some other parts of the menu to improve some things, including color-coding the disc image format in the game list.
    • I re-ripped all of my GameCube discs in order to convert them to CISO format, so now I have a pretty good collection of GameCube FSTs. Need to rip the FSTs from all of my Wii discs next.
    • A few of my GCN and Wii discs don't have disc scans in GameTDB. I uploaded D87E01 yesterday (GameCube Preview Disc), and will be getting to a few others today, including Sonic Heroes JPN (if I can figure out where I put the box).
     
  9. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    I finally started the FST test suite, and I found the directory parser was completely broken. I was interpreting the "next offset" field for directories as "index of the last entry in the directory". It's actually "index of the next entry *after* the directory". This ended up resulting in the parser showing directories containing lots of duplicated files.

    This has been fixed, and I now have a test to ensure that no subdirectory has files that have the same name. Currently testing this on 40 GameCube FSTs.
     
  10. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    So I got a little bit distracted from finishing up the last few blockers and I was getting annoyed that my GameCube save files were all showing unknown file icons.

    [​IMG]

    The ROM Properties extension now supports GameCube save files in GCI, GCS, and SAV formats. (Full memory card dumps aren't supported because they contain multiple files, though I may consider adding something like the folder previews that show multiple files on a single thumbnail later.)

    EDIT: You might notice that the bottom two lines of the Eternal Darkness icon are missing. Turns out I derped a loop unroll optimization that broke CI8 GameCube icons on Windows. It's fixed in git now.
     
  11. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    More distraction. :v: I implemented banner and icon display in the property sheet window:

    [​IMG]

    The icon is animated if the file has an animated icon (e.g. GameCube save files and Nintendo DSi ROM images). This isn't normally doable in the file browser because none of them support animated icons AFAIK, but it's doable in the property sheet. Currently only supported on the KDE build; will add Windows support later today.
     
  12. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Finally getting close to a release. I fixed the last major blocker (installing the shell handlers on Windows even if the user has a custom file association), so here's what's left:
    • Finish TextFuncsTest. I've occasionally hit issues where some changes to the text encoding functions resulted in massive breakage on certain configurations. A comprehensive test suite for every function will make sure this gets caught by failing in 'make test'.
    • Set up CPack to handle .deb packaging on Linux. I'm going to use one .deb per frontend (that is, one for KDE4, one for KDE5) rather than lump all of them in a single .deb package.
    • Write a script to combine the Win32 and Win64 builds into one .zip archive. CMake unfortunately doesn't support building for multiple architectures at once on anything but Mac OS X, so I'll need to build Win32 and Win64 separately, then combine the two. I already wrote some custom batch files to register and unregister the DLLs using regsvr32. On 32-bit systems, it only registers rom-properties-i386.dll; on 64-bit systems, it also registers rom-properties-amd64.dll. Generally, users aren't going to be using standalone 32-bit Windows Explorer on 64-bit systems, but they might use it as part of the standard file dialogs in a 32-bit application.
    I also added some infrastructure to use gcov/lcov in gcc builds to graph code coverage. I'll add this to the travis-ci build later. (Slightly annoyingly, the code I'm using will fail if any of the tests fail, but then again, the tests shouldn't fail, so that's a bit of an incentive to fix the tests.)
     
  13. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    v0.8 (Beta 1) is now available! Check the original post for download links.
     
  14. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Today's hax:

    [​IMG]

    Support for Dreamcast VMS icons. VMI and DCI files aren't supported yet, nor are ICONDATA_VMS files. (VMI will require some extra work because it has to load the icon from the VMS file.)
     
  15. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Finally figured out a derpy bug that was causing some thumbnails to appear slightly offset to the top-left on Windows.

    Windows 7's IThumbnailProvider interface will not resize thumbnails, so for small icons I use nearest-neighbor resizing. It turns out GDI+'s resize algorithm uses a weird setup: by default, the center of a pixel is (0, 0). This is supposedly "faster" than keeping the center of the pixel at (0.5, 0.5), but it results in weird shenanigans for nearest-neighbor resizing.

    The solution:
    Code (Text):
    1.  
    2. Gdiplus::Graphics g(/*blah*/);
    3. g.SetPixelOffsetMode(Gdiplus::PixelOffsetModeHalf);
    4.  
    This sets the correct pixel center. According to MSDN, PixelOffsetModeHalf (aka PixelOffsetModeHighQuality) results in a higher-quality image at the cost of being slightly slower. I'll take "correct rendering for nearest-neighbor resizing" over "it's faster but why is it shifted", thanks.

    EDIT: Strangely, the Windows headers claim PixelOffsetModeHalf "offsets" the pixel by (-0.5, -0.5) for "fast anti-alias perf". This contradicts MSDN. :specialed:
     
  16. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    And now for something completely different:

    [​IMG]

    amiibo NFC dump parsing and thumbnails, courtesy of amiibo.life.

    Special thanks to CheatFreak since he suggested amiibo parsing and provided some useful information and data files.
     
  17. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    Quick update: I'm planning on releasing the next beta version (0.9) around Christmas. One major thing I want to have implemented by then that I haven't started yet is support for XFCE. This requires writing a property page plugin as well as an XFCE-compatible thumbnailer. (The thumbnailing interface on XFCE is the same as GNOME, so I should be able to support both with one project later on.)

    Some other stuff that I've added recently:
    • Support for Wii RVT-R disc images. This doesn't do much, since the update partition doesn't have a full system menu, but it does at least identify the partitions as using the Debug key instead of Retail or Korean.
    • Added more amiibo entries and fixed a few that were incorrect.
    • Fixed a race condition in pngcheck that caused crashes on Windows 10 in some cases.
    Things I also plan to have implemented for 0.9:
    • amiibo: Show owner and name. Requires the amiibo decryption key.
    • Wii: Display the opening.bnr game name.
    • Windows: Automatically pad icons so they're square, since Windows doesn't really like non-square icons. (It ends up being resized to be square, resulting in the wrong aspect ratio.)
    Other stuff I've been working on that may or may not end up in 0.9:
    • Nintendo 3DS support. Currently handles .smdh, .3dsx, and unencrypted .cia files that have a META section. No support for .3ds/.cci or encrypted .cia yet. (Support for encrypted .3ds/.cci and .cia might not happen unless the protected boot ROM is dumped.)
    • Wii save files. Requires the SD common key, IV, and MD5 blanker. Contains a banner and icon similar to GameCube save files, but at a higher resolution. (Banner is 192x64; icon is 48x48.)
     
  18. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    A quick update, since I haven't posted about it in over a month:
    • I've added property page functionality for Nautilus 3.x (GNOME/Unity) and Thunar (XFCE).
    • Thumbnailing is also supported on Nautilus.
    • Thumbnailing has not been implemented yet for Thunar, and I probably won't have it ready for 0.9. (It requires writing a D-Bus service, which is painful.)
    • I have a work in progress (not committed yet) for "fallbacks" on Windows. This will allow rom-properties to be registered for files like .bin and .iso, which would otherwise cause conflicts. Basically, if rom-properties is called for a .iso file, but that file isn't supported (or it is supported but not for thumbnailing), the request will be forwarded to the previously-registered handler.
    • DSi age ratings are now displayed, and ratings for ESRB, CERO, and AGCB are decoded into the corresponding letter values.
    I've received a suggestion about making a GitHub pages site for rom-properties, but I kinda suck at web design. If anyone would like to contribute, please let me know on IRC.
     
  19. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    v0.9-beta2 is out now. Check the OP for download links.
     
  20. GerbilSoft

    GerbilSoft

    RickRotate'd. Administrator
    2,889
    20
    18
    USA
    rom-properties
    So here's what I've been working on since v0.9-beta2:
    • Parser for Windows/DOS EXEs, DLLs, and other executable files. This includes NE, LE, LX, and PE formats. For NE and PE, the header flags are shown, and if a version resource is present, the version resource is displayed in a subtab. Not yet implemented is icon thumbnailing (for non-Windows platforms only) and manifest parsing (for PE).
    • The Nintendo DS, GameCube, and Wii parsers now support downloading cover scans. The user configuration program for determining which type of scan to use has not been started yet, but it will be included for v1.0.
    • Parser for Wii U disc images. (*.wud) Currently only shows the basic product information, including game ID, region, and system version; everything else is encrypted using a per-disc key. GameTDB downloads of disc and cover scans are available.
    • Support for multiple sizes of external images. This will allow e.g. downloading the "default" image for small thumbnails, and then downloading "large" images if the thumbnail size is increased. Supported for Nintendo DS and Wii U; support for internal images (e.g. Windows executables) will be added later. An option to control downloading of large images will be added to the user configuration program, since that might not be wanted on low-bandwidth connections.
    Some other stuff in progress:
    • Windows: Property tab for actual CD-ROM drives, e.g. if you insert a Wii DVD-R. Icon handling probably won't be added initially, since I don't want to conflict with the default CD-ROM icon handler.
    • Nintendo 3DS parser. Currently only supports standalone .smdh and .3dsx files, and I think unencrypted .cia files that have a META section. Will be adding support for encrypted .cci/.3ds for GameTDB image downloading and partition table viewing; for decrypted .cci/.3ds and .cia, it will parse the icon as well.
    EDIT: I'm still looking for feature suggestions and/or code contributions. If you have any ideas, feel free to reply.