Science

There's already been issues like this in using copper sulfate (which is inorganic in terms of molecular structure and persistent as an element like you mention, but labeled as organic in some jurisdictions in terms of agricultural treatments and food marketing) as a fungicide. It's a very short lived fungicide for leaf borne fungal diseases, meaning it must be present on the leaf to prevent infection (it is easily washed away by rain or errant irrigation) and applied repeatedly, but is long lived in the soil meaning it can build up and kill off mycorrhizae and other beneficial soil fungus causing longer term drops in yield.

So, yeah, your worry is valid. And, that's just one example.