It's a split second observation. Judgement isn't always negative.
The people you ignore are still there and you are aware that they are, just like you're likely aware of at least their general appearance from even a split second glance.
Most people won't care and will move on because as you said, caring about everyone else's tiny little differences would be taxing but they still judge you based on those observations.
If you interact with them however it will suddenly matter.
Essentially what I'm saying is that it's not something people turn on and off. It's always there but the information is just discarded if it's not considered useful.