memes

I mean yes it's a bit under-nuanced to describe any language as "easy" or "hard". The single biggest influence is whether you're already familiar with a similar language. English is going to be much easier if you already know German; Japanese will be much easier if you already know Okinawan. And as you say, written and spoke language can be quite different.

That said, I don't think it is the case that all of the different factors trade off against one another perfectly. I would expect them to trade off against one another to an extent though, because I would imagine there are forces which cause overly complex languages to become simpler, and more simple languages to become more complex. (One aspect of complexity comes through redundancy, such as requiring agreement between inflections of words when the inflection only conveys information already imparted from the rest of the sentence. But extra redundancy can aid in understanding because the listener generally doesn't hear everything perfectly)

But yeah, some languages just have incredibly complicated and picky grammar, whilst others have relatively simple grammar. As an English speaker, Japanese grammar has lots of unfamiliar features but could still be simpler than Finnish, which also has lots of unfamiliar grammar but which is very complex.