It's absolutely possible, though. MPV has it. It definitely takes longer than going forward, and sometimes I have to press the "back one frame" shortcut 2-5 times per frame. But, it does exist.

A simple explanation: Compressed video is typically stored in such a way that you have say, one full frame every 5 frames or so. The in-between frames are just what changed from the previous frame. (Actually, smart compression is adaptive, changing how many full frames it needs depending on what the content is.)

So going forward a frame is easy. The current view is stored in the frame buffer, and you just add the changes to the next frame.

Ah, but how to go back? There are (at least) 3 possibilities.

1. Reverse the process. "Subtract" the last frame data. But this may leave nothing behind, and so probably isn't viable.
2. Double buffer. You have to keep the previous frame around in a second buffer. This adds a new copy operation every frame, and another buffer in memory, but performance for that probably isn't too bad. But you eat that performance all the time just so you can rewind one frame. And to rewind more than just 1 frame, you actually need many past frame's buffered! So something more like a stack of old frames would need to be kept around (like a stack of photos) and you can riffle back through them anytime. We'd throw away old frames past the last checkpoint, but ultimately this is just an increasingly expensive way to do it. This would need to be optional I think.
3. Last option is the classic comp sci tradeoff: if option 2 used a lot of memory, option 3 uses a lot of calculation. Each time you go back a frame, we find the previous checkpoint frame, and then apply each change frame up to where you rewound to. This takes a moment to calculate each time, but unless you're QTE-ing on the frame back button it's probably fast enough!

The best solution probably mixes 2 and 3. Maybe a double or triple frame buffer, with the option to calculate new back-frames as further needed?

Anyway, that was fun to think about!