I've not tried this myself, but how about mounting the volume using sshfs?
https://simplytim.io/mounting-a-sftp-ssh-share-as-a-volume-in-docker-compose/
This is of course assuming that you've got ssh access to the VPS.
I'm currently mounting volumes using cifs and NFS, but I don't think I'd be too keen on exposing those to the internet at large.