I think the trick is to make an effort to cover as many possibilities as can be dealt with by a reasonable effort (definition of “reasonable” varies significantly by context) when setting up something which you expect the general public to interact with. Not so much assuming that any given person has some disability you can’t see, but that any large group of people will have at least a few.

Interactions with a specific person are another matter entirely, as you point out. There, I think the best you can do is roll with it if someone tells you that they’re unable to do something without subjecting them to interrogation or scepticism