It didn't used to be like that. Until the 10spd era things were more or less compatible between mtb and road, and inbetween bikes, like tourers or beaters, mixed. And earlier than that, before the modern 7/8 spd era, everything was designed and intended to be mixed and matched, on high level bikes even, because components were just components. The idea of a group SET is a modern one.
An integrated and comprehensive system is simply superior for most people, over freedom. Freedom is not actually an important idea. People want what they want and they're told what they want and that's what they want it has to be easy and it has to be that which what it is. It's not about choice but they want to choose it.