When you say the first element of a matrix, first implies one and not zero. This is how linear algebra was invented (on paper, by a human mathematician), taught, and passed down to fellow humans.

Starting indexes at zero stem from the lineage of C programming and binary nature of computer. For example,

Computer memory addresses have 2^N cells addressed by N bits. Now if we start counting at 1, 2^N cells would need N+1 address lines. The extra-bit is needed to access exactly 1 address. (1000 in the above case.). Another way to solve it would be to leave the last address inaccessible, and use N address lines.

This is why, math and physics people who learn linear algebra and matrix calculus learn to index at 1 (on a piece of paper) while computer science programmers index at 0.