So I did a bit more research, it seems that there are tons of loan programs under the USDA umbrella. Lots having to do with farming, but also lots having to do with "rural development".
These loans fall under something called the SFH Direct Loan Program (for nonfarm tracts). Which you can read more about here and see the forms here.
There are multiple tiers of these loans and they are all dependent on income and ability to pay. They work almost inverse of a typical mortgage loan where the only thing that matters is your history with on time payments and having at least an average credit score. They start at 3.25% interest for "Low Income" individuals and families (determined by a chart that's bucketed by county). Seems to be between $45k and $65k on average for "Low Income" and the "Very Low Income" group is anyone below the $20k poverty line.
All these income buckets are determined by number of applicants with deductions disabled or child dependants (about $10k taken off your net income per dependent). You can also deduct medical expenses. So the name of the game is trying to figure out how to structure who in your family will be on the loan to get the lowest mortgage rate (1% with the grants for very low income applicants, but these need to be repaid if the house is ever sold).
All these loans are given out directly by the government meaning you don't owe a mortgage to a bank, but to the USDA itself. You have no down payment if you are accepted and are allowed to purchase any "quality" home that's at or below the loan limit for your area (seems to be around $300k plus or minus in most places).
So if you're really desperate and live outside a major metropolitan area, this might actually be an option to break out of the rental hellscape or if you have a bad mortgage and think you'd qualify this might be a way to refinance directly through the federal government.
I'm by no means offering you any financial advice, and an definitely not a financial advisor, but this program seems to genuinely have some merit and honestly I think it could work as a greatly expanded solution to the housing crisis, directly government issued mortgages with downwardly adjustable rates, long terms, and income based grants.
I was just summarizing for the people who are too lazy to go read anything and will just stop here