I have a small head cannon that part of the reason why Neptune can't tell anyone apart is because he's not wearing his glasses.
Like sure he's bad at remembering names, but I like to think it's worse because he can't see anyone's face.