Funnily enough there's actually sexual dimorphism in human vision. While studies about exactly how different the structure of the eye is and how it might differ from an assumed objective commonalities is still not conclusive it's thought Female phenotypic people supposedly do tend to have higher color differentiation on average and are better at recognition of static objects while Male phenotypic people have better night vision and ability to interpret objects in motion.

So a guy holding up two paint swatches they can't tell the difference in and being confused is basically a cross cultural phenomenon.