THANK YOU ALL!
It was a problem with my docker compose file! I didn't list the needed devices from the jellyfin documentation. I thought the Container was detecting the gpu but it wasn't. Docker exec nvidia-smi is your friend!
Edit: so now it doesnt kick me out saying the playback failed but its just a black screen with 4k media
Edit 2: my bad forgot to enable some transcoding settings in jellyfin lol