You may want to update your post to mention which version of qBittorrent 4.6.0 you're on.
e.g. are you using the build with Libtorrent 1.2.x or Libtorrent 2.0.x?
Libtorrent 2.0.x does tend to use more memory during runtime (especially when you have many actively uploading/downloading torrents) but it's fine overall, the OS / kernel knows to redistribute memory away from qBittorrent to other applications as needed. In other words if nothing is crashing then you should be okay.
That said I've mainly tinkered with Libtorrent 2.0.x clients on Windows (Deluge & qBittorrent) so there might be something I'm missing specific to Linux or Docker. qBittorrent with "Physical memory (RAM) usage limit" set to its max will basically let Libtorrent use as much memory as it likes.. it is lower priority memory so in theory as long as everything else in Windows is working the other applications can still request memory & run properly. Funny enough with Deluge I don't think it even has a RAM usage limit setting so Deluge with Libtorrent 2.0.x will happily use the max memory available to it.
From the /c/piracy rules