The key is the aforementioned emerald. Tails, as the original guardian of the Master Emerald (as proven by his life monitor in the zone), single-handedly caused the zone to be axed from the final game. By eating the thing he was most responsible for.
Tails ate the Master Emerald. Is the Master Emerald in Hidden Palace Zone? Yes. Is Hidden Palace Zone in Sonic 2? No. That is the only explanation, first being posed back in the early 2000s by forum-poster "l0st". It also explains why Tails hasn't been in the spotlight since Sonic 2, and why Knuckles never liked him.
You see, Tails eventually digested the emerald and was forced to give up his position as the guardian. The emerald was given to Knuckles, who used it to keep the Floating Island in the air. Knuckles was not happy with this, and has hated Tails ever since. Sonic has never liked him as much, instead making many new friends with each adventure and pushing his once best friend further and further away.
But why did Tails eat it? Here's an explanation posed back in 2002 by "HedgesMFG":
"Tails heard rumours of the Chaos Emeralds from Sonic's first battle, and he thought they might taste really good... so he decided to follow Sonic hoping he'd come across emeralds and Tails would then eat and digest them and become über-1337 Tails and rule the world with a glowing fist and an emerald-filled stomach!"But will Tails ever do this again? Eating the Master Emerald was bad enough, after all:
"But beware... one day when Sonic and Knuckles LEAST expect it, Tails will dive upon a Chaos Emerald and consume it before their very eyes! BEWARE I TELL YOU!"Certainly a frightening prospect. The theory has been put forward to Sega of America and Sonic 2 developer Mark Cerny in the past, both times with no response. They are clearly hiding the truth - the key to the entire back-story of the Sonic games up until the present day.
is the Master Emerald, and yes, Tails did eat it. That is also why Hidden Palace Zone in Sonic and Knuckles is so drastically different - once the Master Emerald was gone, its appearance changed dramatically. When it was brought back, along with the other Chaos Emeralds, it formed the zone that we now see.
Maybe one day Sega will accept the truth and admit it. We all know why Hidden Palace Zone was axed, and why there are so many new characters with each game. Sure, it was Tails' fault, but we must forgive him. After all, the Master Emerald does look ever so tasty.