In your calculation, you are ignoring the fact that there is not only byte-by-byte access.
When reading, in extreme cases you can read the entire EEPROM area with a single READ command (see description in the data sheet, section 6.5).
When writing, you can write a page of 512 bytes in one go, i.e. up to 512 consecutive bytes. You only address the first byte of the page and can then let the rest follow in the same write cycle, so you do not need to wait for a complete write cycle each time.
Incidentally, although the 5ms are specified as the maximum time for writing, it does not make sense to simply wait 5ms, but is much better to query the bit WIP in the status register (data sheet, section 6.3).
Regards
/Peter