> That’s exactly my point: by that definition any incorrect answer can be explained by “overfitting to training data”.

Not really, getting 94381294*123=... wrong, but close within the actual answer, cannot be overfitting since it wasn't in the training data.