rpg

I personally think it's three things.

1. D&D is fucking old. Like, people who grew up on OD&D are dead now. It's had a lot of time to accrue clooge and gunk as the genre of D&D shifted from 1st Person Wargame to Dungeon Crawler to Epic Fantasy to Heroic Fantasy to whatever 5e is, and this means a lot of different monsters to serve these different genres being carried forward into future editions because Gen X gets weird when the things they grew up with get changed or disappear.
2. D&D has always fundamentally been about going out into the world, seeing weird things, and killing them. There's only so many times you can kill zombies before you get bored of zombies. Also, a number of monsters have gimmicks that get old fast, like rust monsters eating your weapons as you hit them and gelatinous oozes being nearly invisible. The only counter to that is more monsters.
3. Different monsters serve different purposes, even when they fill the same niche. For example, goblins and kobolds both are tiny little mooks that attack in large numbers and are inherently evil so we don't have to feel bad about slaughtering entire families of them, but one is more likely to ambush you with ranged fire and Explosives while the other is likely to set up traps for you to wander into. Very different styles of play. Orcs and hobgoblins are both basically people, but while orcs allow you to hold up a mirror to the party by being essentially normal people but ugly, hobgoblins lead tactically complex multi-pronged attacks and are make very good scheming bad guys. Also, not every monster belongs everywhere. Imagine finding a stone golem in the middle of the feywild, or a treant in a dungeon. Attempting to make a monster for literally every situation is how AD&D ended up with the Monstrous Compendium's fifteen volumes and appendices, so in reality the Monster Manual is an exercise in restraint.