the real question is why numbers are used at all. it's morning, afternoon, evening, or night.
we don't call the beginning of summer A and the middle of summer A+1, the end of summer A+2. fall would become B, B+2, or B+3.
in the beginning of the use of clocks, they had only one hand, the hour hand.
some big muckety muck probably decided, when he said 3:12, he expected his appointment to be there precisely at that appointed time, so the big shot would know how long to keep his underling waiting.