Games

Rendering a 3D scene is much more intensive and complicated than a simple scaler. The scaler isn't advanced at all, it's actually very simple. And it can't be compared with running a large model locally. These are expert systems, not large models. They are very good at one thing and can do only that thing.

Like I said the cost is fixed, so if the scaler can handle 1080p at 120fps to upscale to 2K, then it can always handle that. It doesn't matter how complex or simple the image is, it will always use the same amount of power. It reads the image, does the calculation and outputs the resulting image.

Rendering a 3D scene is much much more complex and power intensive. The amount of power highly depends on the complexity of the scene and there is a lot more involved. It needs the gpu, cpu, memory and even sometimes storage, plus all the bandwidth and latency in between.

Upscaling isn't like that, it's a lot more simple. So if the hardware is there, like the AI cores on a gpu or the dedicated upscaler chip, it will always work. And since that hardware will normally not be heavily used, the rest of the components are still available for the game. A dedicated scaler is the most efficient, but the cores on the gpu aren't bad either. That's why something like DLSS doesn't just work on any hardware, it needs specialized components. And different generations and parts have different limitations.

Say your system can render a game at 1080p at a good solid 120fps. But you have a 2K monitor, so you want the game to run at 2K. This requires a lot more from the system, so the computer struggles to run the game at 60 fps and has annoying dips in demanding parts. With upscaling you run the game at 1080p at 120fps and the upscaler takes that image stream and converts it into 2K at a smooth 120fps. Now the scaler may not get all the details right, like running native 2K and it may make some small mistakes. But our eyes are pretty bad and if we're playing games our brains aren't looking for those details, but are instead focused on gameplay. So the output is probably pretty good and unless you were to compare it with 2K native side by side, probably you won't even notice the difference. So it's a way of having that excellent performance, without shelling out a 1000 bucks for better hardware.

There are limitations of course. Not all games conform to what the scaler is good at. It usually does well with realistic scenes, but can struggle with more abstract stuff. It can get annoying halos and weird artifacts. There are also limitations to what bandwidth it can push, so for example not all gpus can do 4K at a high framerate. If the game uses the AI cores as well for other stuff, that can become an issue. If the difference in resolution is too much, that becomes very noticeable and unplayable. Often there's also the option to use previous frames to generate intermediate frames, to boost the framerate with little cost. In my experience this doesn't work well and just makes the game feel like it's ghosting and smearing.

But when used properly, it can give a nice boost basically for free. I have even seen it used where the game could be run at a lower quality at the native resolution and high framerate, but looked better at a lower resolution with a higher quality setting and then upscaled. The extra effects outweighed the small loss of fidelity.