The N170 is a brain electrical potential proposed to index the earliest time of categorical perception of faces in occipitotemporal visual areas implicated in face cognition, being more negative for faces than nonface objects between 120 and 200 ms. The interstimulus variance (ISV) account instead explained N170 face-specificity as an artifact due to objects varying more visually than faces. Ganis, Smith, and Schendan (2012) tested this account directly, finding that N170 face-specificity remains even when ISV is eliminated. Here, N170 peak amplitude and face-specificity is quantified for individual stimuli and participants, revealing that the right hemisphere N170 is especially sensitive to stimulus variability. Further, ISV contributes 0 to 37% to N170 face-specificity. These findings provide evidence for optimizing face processing science. The paradigm can apply to any research in which ISV may be uncontrolled (e.g., category comparisons).