What I found was quite strange to say the least.
In Sonic Jam's files, I found that I could view certain files correctly in Tile Layer Pro and assemble them into perfect sprites. These were mostly the sprites for Sonic and Co, along with some special stage artwork, and the Pause Menu graphics. Some files also contained a few compressed looking files, but they were somewhat viewable. Despite having strange file extensions, they can either be opened in TLP and formatted to Genesis, or they can be re-named as BIN files and opened in TLP normally.
As for Sonic 3D, I found some interesting pictures that I have no way of opening. Although I cannot open them, I have a feeling that at least two of them are unused files from the PC version that are also unused here. They are the two pictures of Sonic with Knuckles and Miles that were theorized to be connected to the Special Stage on the Wiki. Another interesting thing is that the special graphics that were added for this version of the game are in their own folder that's separate from their respective level folder. (Things like the new bridges and the weather effects.)
And for Sonic R, all I was able to uncover was a strange naming system for all the levels. They are NOT called what they are normally called in game. (Resort Island is called BEACH for instance.) Rayman used a similar naming system for its levels, though it was discovered by the public as early on as the release of Rayman Designer, which called the levels by their internal filings rather than established names in universe.
Now for proof, all I can post are some pictures at this time. I don't want to put up the files unless I'm told otherwise that I won't be banned for doing so, however I will post a text file showing my research of Sonic Jam so that you can get somewhat of a look as to what I've found.
A rip of Sonic's Sonic 1 art from Sonic Jam, which is located in a file called "DATA.SN1". All that's on here is the graphics for Sonic. They are also unassembled, though Sonic seems to have all of his unused frames intact.
Also located here are the dust effects for when he skids or spindashes.
Another screenshot from TLP showing the Sonic 1 sprite being assembled in the tile viewer.
A screenshot of TLP showing Sonic Jam's in-game pause menu. Strangely, certain parts are either compressed or omitted from the pause menu graphics, such as the top half of the CONTINUE text.
A screenshot showing Sonic R's strange internal naming. I'm unsure if the PC version names them the same way, so if anyone could tell me, that'd be great.
As for the Text file, it's still being updated as I check the files in my HEX editor. If you still want to see the incomplete file, you can find that here:
Sonic Jam Documentation v0.1
I do apologize if someone else has also found this and posted it before me, I couldn't find any topics pertaining to this using the Search feature. I hope this allows others to learn more about how Sonic Jam works, and (hopefully) to spark more interest in the Sega Saturn.