What exactly does gender achieve in a language? Is English missing out on any nuance?

Sort of. Grammatical gender and the interplay with grammatical case (the "role" of a noun in a sentence) allows some extra meaning to be packed in. For example, German has 3 genders and 4 cases leading to 12 different contexts for nouns to be in. Many of those have their own conjugation patterns, and separate words for the articles "a/the".
That can, theoretically, allow meaning of the type "whose what did what to whom" to be obvious or pieced together in a sentence, whereas translating it into English you might need to spell it out, lose it, or rely on context.

In practice, a lot of that sort of information is often redundant or clear from context anyway, and only matters if you're being clever or succinct. My German is shit, so I will not try to provide examples.

It's also worth pointing out that it's a naturally occurring feature, likely arisen by accident.

Is it literally thinking about nouns as male or female, or is it just a weird name for the concept?

It is mostly just a weird name. Some of it makes sense along (social) gender lines, much of it makes no sense at all. This thread is full of good examples of counterintuitive noun genders in all kinds of languages.

Who decides gender when a new noun is made? What about borrowed words from other languages?

The speakers of the language, collectively, usually with some disagreement, trial and error. Borrowing depends: a gendered noun borrowed into a non-gendered language would just slip in there. In the reverse case, people would just arrive at some gender for it arbitrarily or based on similar words, what gender any "parts" of the term might be if translated, or whatever other method. There's no correct answer.

Do you sound stupid if you speak French without using it, or are you just a language hipster?

Quite likely. There's no "without it" in gendered languages, it is a more or less fundamental part of the noun and the language, like how certain nouns and verbs are just different in English. Dropping random grammar and syntax from English would just be "doing it wrong", ranging from cute foreign accent quirks to Ralph Wiggum's cave-dwelling ancestor.

Of course, fucking up is unavoidable when learning languages, and most people will give you a lot of leeway due to being foreign. Maybe not everywhere in France, though...