think of the positive version of the sentence and it becomes more obvious it should be is:
"Either of us is going"
Or, to further stress the point, turn it into a question:
"Is either of us going?"
The subject is not us, it's "either of us", which is singular
Accepted behavior among geologists