Even taking you at your word, just building more houses wouldn't solve the problem unless the other existing issues are solved first. There are already more than enough houses, several times more unoccupied houses than there are homeless people in fact. If you just make it easier to build more, those new houses will just end up in the same situation as the existing lot: bought up by corporate groups as investments, held ransom by landlords, and generally NOT made available to consumers who want to buy a home.
So yeah. You're gonna see some pushback if you're only making that second argument, all that will do is make the investor class richer without solving any problems.