I love this idea! I honestly adore the concept of Mecha Knuckles, and I wish we had seen more of him.
It's weird because most villains either turn good or end up inferior to Eggman. Hypothetically speaking, a 1st class swimming teacher whose goal it is to teach the world's bluest non-swimmer how to swim. His career drove him to delusional beliefs and he just won't take no for an answer. He would say things like How dare you sully the beauty of swimming? or I will not rest until you get out of your comfort zone, because Sonic doesn't want to take swimming lessons. What if they brought back the Echidna tribe & Mecha Knux?
Or how about if the rival mad scientist to Eggman is like I said, a biochemist? Think of them as an E-rated version of the various villains from the Resident Evil series.
I imagine some sort of supercomputer created by Robotnik that turns on its creator and starts bioenginering evil creatures much like organic badniks, and starts to corrupt and assimilate the world and its inhabitants. A la RayForce or Run Sabers. We would see rampaging bioweapons, assimilated Robotnik and a more serious threat than ever before.
I think it would be good to have more villains who: A. Aren't affiliated with Eggman and don't team up with him. B. Don't become Sonic's allies by the end of the game. C. Aren't mindless monsters, destroyed/defeated by the end of the game. Another thing to consider is whether they would be a singular side-villain, or if they'd have minions under their command. Those minions would have to be enemies Sonic can casually smash through, so humans or animals would be a bad idea, and robots would be too repetitive. A leader figure for all the ghost/Boo enemies we've seen through the years is one possibility - someone higher up the food chain than King Boom Boo. For a solo villain, the core series hasn't really had a teenage delinquent before. The Sonic Riders series had Jet, but he ended up as more of a friendly rival.
Step 1 to creating a good Sonic villain: Create a good Sonic Story. The reason we haven't had any good villains recently is because the games' stories have been half baked. From SA1 to 2006 the cutscenes have averages like 90-100 minutes. After 2006 the average length has been like 40 minutes. There just isn't enough time or heart in the games to make you care about the villain. There's a reason people love Shadow. It's not necessarily his character, it's that the story was long enough (and good enough) to give him a character arch. Even though 2006 was universally panned, Mephiles was generally liked as a bad guy. Compare that to Zavok or *insert random final boss monster from recent games* They're already forgotten.