It's less the result of a sensible system of units (like how 1 L of water ideally weighs 1 kg), and more fortunate happenstance in this case.

The formula for hydrostatic pressure* is:

∆P= ρ·g·∆h

where ∆P is the difference in pressure across the difference in height ∆h, ρ is the density of the liquid (~1000 kg/m³ for water, slightly more for sea water), and g is the acceleration due to gravity.

So the reason it works out nicely is because g is a little bit less than a nice factor of ten (9.8 m/s²), and the density of sea water is a little bit more than a nice factor of ten (typically 1025 kg/m³), and 1 atm also happens to be almost a nice factor of ten (101,325 Pa). That's why the difference between the approximation and the actual* is less than a percent.

*This assumes a constant density of the liquid, which for water is reasonable, however different depths can have different salinities and temperatures in layers which change the density by less than a percent. Additionally, this assumes a constant acceleration due to gravity. At depth, the acceleration due to gravity can be higher, but this also has an effect that amounts to less than a percent even at the deepest point in the ocean.