I'm not a formally trained expert in game engine design, I'm just a hobbyist for now, but I sure as anything have more than just a little experience with source code, compilers, libraries, and graphics, and programming C++.
I can help with the planning.
Level Geometry
To me, the most complicated thing, yet also the most important thing to take on is the geometry, collision, and basic rendering first.
By basic rendering I mean being able to render polygons with correct normals, and being able to render UV mapped stuff, the 3D format could be anything I guess, .dae, .3ds, .obj
And then collision, to the point where the player can at least walk across any polygon surface normal in any orientation, then the real Sonic momentum physics can come!
And I'd say the 3D Sonic physics is the absolute most critical thing about making a 3D Sonic engine; I really want to see a fully working Sonic physics system with rolling and running on walls and everything implemented in
C++.
Basically, once we take care of that, we've gotten the majority of the complex math out of the way and it's easy Object Orient programming to add in the actual game elements, no?
-------
Editor Integration
I assume the engine does not intend to be a complete content creation solution and level geometry and characters are handled with external editors.
Personally, I always thought a fully built-in editor for things like the actual level would be the easiest thing for the level designer(but the most involved thing for the coder).
Hear me out, if we use our own custom map format, then things like the camera, cutscene scripting, water, object placement and other things that are naturally in a level don't have to be disruptively separate from the main editor. Streamlining this stuff would ease the workflow for the level designer(s). Organizing file directories for a level would be almost completely handled for you and possibly packed in a neat little zip file.
Why such integration would be useful is because: say you could just draw a 3D area right in the editor, and in that drawn area whenever the player touches it, it executes a script. And that script could be edited right there in the built in menu and saved with the level itself. And then you can instantly test your level without having to wait or rebuild or anything.
Imagine that you can play a cutscene you made in game, and you see that one of the camera angles is a little bit off, you press the edit button in the engine, tweak that camera right there, switch back to game mode, then be right on your merry way without having to go re-edit some script in another editor and relaunch the game.
Content Organization
The actual level format could be as simple as a zip file with sub folders for the geometry format(.obj? .3ds? .dae?), the textures for the level(and/or references to a main set of textures that comes on a default install), the script files including things like the level title, map author, camera sequences etc. and the extra volumes data such as water, bottomless pit areas, etc. stored in .txt or in binary format.
Have the engine automatically arrange these things for you when you save a level.
Some people may prefer editing and handling every script file and .objectposition.txt file themselves, but doing this seems rather fragmented and not as efficient, but that's just my opinion. Still, using a zip based format would let people who DO like to edit everything themselves have that kind of workflow, and for faster dev times, make the engine also support loading content directly from uncompressed folders too!
This opinion comes from the way I see people have been editing with UDK(SonicGDK), or even while hacking Sonic Generations, I know working with Sonic Generations is far more difficult than having an actual IDE, but than again, it's not THAT different from the way modern game engine development works. So many different editors and systems and scripts, with both UDK and Sonic Generations you have to work separately on the actual level geometry file and the collision file for the level!
I'm not saying make a completely do-everything editor, but I mean at-least make the collision data and geometry somewhat unified if possible! (Automatically generated collision organized into Octrees? And then that collision data can be saved into the neat .zip file for the level?)
But the packaging is really step 3.
Step 1. Is to be able to load and render some kind of standard, flexible geometry format (pretty simple, no?)
Step 2. Be able to get good
3D SONIC PHYSICS with it (Using original math definitions for vector/matrix math, let's not use boost or Bullet or something for the
core vector physics/math please!!! No engine bloat!)
3D Sonic physics in plain C/C++, that's one thing I've ALWAYS DREAMED OF, someone make it happen, I shall try to make it happen but my math skills are no where near as good as many of the members here! Once Sonic physics has any form of decent start to it (I.e. walk along any normal with any orientation, some basic gravity), then the rest is of creating the engine should be rather easy in comparison.
Step 3. Decide on a good packaging system for content.
Step 4. Graphics: Gen, I really like graphics, the technical side too, exactly how much GLSL goodness could you deliver? Deferred rendering? Realtime Shadows? Depth of Field? Global Illumination?!*dreams*
Step 5. Add the Sonic stuff like rings title cards, level transitions, the easy stuff