Ask a model to analyse a dataset and it writes code, the code runs, real numbers come out, and a paragraph explains what they mean. Three independent things had to be right. Only one of them tells you when it was not.
Three places to be wrong
The code can be wrong. If it crashes you find out immediately, which is the benign case. The dangerous case is code that runs cleanly and computes something other than what you asked.
The statistics can be wrong. The code faithfully executes a procedure whose assumptions the data violates, or which answers a different question from the one you have. Nothing errors; the number is simply not evidence for what you think.
The interpretation can be wrong. This is where the model is on its home turf and at its most dangerous, because generating a fluent explanation of a result is exactly what it is good at, and it will do so with equal confidence whether the result supports the explanation or not.
