Most things in our regular day-to-day lives would leave residue, even water (maybe deionized wouldn't?) just a matter of how much and how noticeable. About the soap, well if you smell it on your hands after wash, then it's still there and you will rub it onto stuff. At least it's just soap tho...
I don't know if you can reasonably remove all contaminants off your skin, unless you plan to like glove up or destroy the skin by washing waayyyyy too often. But you know, good hygiene will keep things you touch nicer for longer so keep at it.
I'd second swimming or in general doing stuff in water. The difficult part will be to find a place to do that which can accommodate your particular needs, i.e. if you don't wanna actually swim, does the pool have area for that, is it not full of kids (not just kids) who won't pay any attention to you and potentially hurt you while playing