The highly influential theory of “Motivated System 2 Reasoning” argues that analytical, deliberative (“System 2”) reasoning is hijacked by identity when considering ideologically charged issues—leading people who are more likely to engage in such reasoning to be more polarized, rather than more accurate. Here, we fail to replicate the key empirical support for this theory across five contentious issues, using a large gold-standard nationally representative probability sample of Americans. While participants were more accurate in evaluating a contingency table when the outcome aligned with their politics (even when controlling for prior beliefs), we find that participants with higher numeracy were more accurate in evaluating the contingency table, regardless of whether or not the table’s outcome aligned with their politics. These findings call for a reconsideration of the effect of identity on analytical reasoning.