rpg

A) because the fantasy setting lends itself to actually having monsters.

B) the often slower-paced gaming style can make a better use of a diverse "cast", as you have the time for Monster analysis, e.g. elemental resistance, magic resistance, etc.

C) it didn't plop out of the writers pen all at once. It's a vast "ecosystem" which has grown for decades, with a myriad of writers adding to the environment, as compared to a single writer who'd have to keep track of all of those monsters in one go. Compare it to let's say Egyptian or Greek mythology, same story.

I personally think it's three things.

1. D&D is fucking old. Like, people who grew up on OD&D are dead now. It's had a lot of time to accrue clooge and gunk as the genre of D&D shifted from 1st Person Wargame to Dungeon Crawler to Epic Fantasy to Heroic Fantasy to whatever 5e is, and this means a lot of different monsters to serve these different genres being carried forward into future editions because Gen X gets weird when the things they grew up with get changed or disappear.
2. D&D has always fundamentally been about going out into the world, seeing weird things, and killing them. There's only so many times you can kill zombies before you get bored of zombies. Also, a number of monsters have gimmicks that get old fast, like rust monsters eating your weapons as you hit them and gelatinous oozes being nearly invisible. The only counter to that is more monsters.
3. Different monsters serve different purposes, even when they fill the same niche. For example, goblins and kobolds both are tiny little mooks that attack in large numbers and are inherently evil so we don't have to feel bad about slaughtering entire families of them, but one is more likely to ambush you with ranged fire and Explosives while the other is likely to set up traps for you to wander into. Very different styles of play. Orcs and hobgoblins are both basically people, but while orcs allow you to hold up a mirror to the party by being essentially normal people but ugly, hobgoblins lead tactically complex multi-pronged attacks and are make very good scheming bad guys. Also, not every monster belongs everywhere. Imagine finding a stone golem in the middle of the feywild, or a treant in a dungeon. Attempting to make a monster for literally every situation is how AD&D ended up with the Monstrous Compendium's fifteen volumes and appendices, so in reality the Monster Manual is an exercise in restraint.

So I have been DMing a DnD campaign from level 3 to level 15 at this point. At this point my players have killed ~900 "monsters" at in around 40 different adventures. This doesn't even count the numbers that were in an adventure they didn't kill because they ran away, avoided or were NPCs with monster stats. They have killed at least 10 in all categories except plants, fey and celestials so having lots of categories helps as well.

So we need a lot of different options to keep everything fresh and interesting for the players and myself. Some of these are nuanced adventures with complex interactions and moral dilemmas while some are basic hack and slash. So I need a lot of different monsters to use that fit different flavors, use different mechanics and cover different difficulties. I have created some from scratch but it helps to have templates to start with. I won't use everything in the monster manual but I have used a lot of it so far.

I think it helps to think of each adventure as its own little novel. So in one we are exploring the culture and cruelty of the Yuan-ti in a vaguely Mesoamerican inspired setting while the next they are exploring Bridgerton inspired high culture and dance that was infiltrated by both a rowdy fey and cosmic horror beyond the stars. It helps to have lots of inspirations to draw on since every adventure can be different.

Monsters are more interesting than animals and humans?

Are you asking why there are so many kinds of monsters, or why monsters appear so frequently in the campaigns you've played in?

Because it's easier to throw the latest Monster of the Week at players than it is to craft NPCs and relationships such that there is compelling drama. Combat is easy. Drama is hard.

While the D&D Fight monsters ad nauseam is more an exception rather than the norm in RPG, it still has a couple of asset.

Monsters are easy bad guys, there is not much moral consideration in fighting giant spyder or "children eating satanic Goblins" while if you fight against other humans who live in the forest that the Duke want to cut-downt to get wood for his navy it make everything more nuanced and complex. It's great in regular RPG where you can get tons of interesting moral dilemna out of it, but if you're into dungeon crawling monsters are removing all of that, and allow you to focus on the manage ressources to fight monsters apect of D&D.

if you're in zero to hero fantasy, you need opponent within that scale. so start with Goblins, then orcs, and finally dragon, and had a whole list in the way

A you mentionned, it's pretty specific to D&D and in many RPG you don't use much monster, or come with something unspeakable and too powerful for the PC to fight against

More stuff to kill in the kill stuff game. If you look at games that emphasize more narrative, there will be fewer pages dedicated to just monsters, like The One Ring 2e. The downside is having less varied combat and less freedom on crafting your own stuff

First, the new monster manual is atrocious... they people that worked on it should all be fired.

They made the weakest and lamest challenges, then threw in a lot of bad mechanics and obviously "bugged" creatures.

It draws from a lot of fantasy mythologies and needs a variety of monsters to populate encounters of all levels with lots of variety, plus intelligent species often have cultures which require more subtypes.