Chapter 18 Sensitivity

About why we shouldn’t rely only on sampling variation to indicate what we know about the effect size. We should look at a variety of models and model architectures, proxies for the measured quantities (since a variable is not necessarily what we want it to be) to get a sense of how much variation there is among models of equal plausibility.

“The Statistical Confidence Game” – why do we focus on doing the same thing over and over?

Should get similar results when using different proxies for the effect, e.g. in the SAT data use expenditures, but also teachers’ salaries and class size, perhaps building and administrative expenses, ….

Motivated by Michael Lavine’s essay the The American Scientist. “Frequentist, Bayes, or Other?”" Michael Lavine https://doi.org/10.1080/00031305.2018.1459317

Also see Steven Ziliak’s article in the same issue, from which these quotes are taken:

G-7 Minimize “Real Error” with the 3 R’s: Represent, Replicate, Reproduce

A test of significance on a single set of data is nearly valueless. Fisher’s p, Student’s t, and other tests should only be used when there is actual repetition of the experiment. “One and done” is scientism, not scientific. Random error is not equal to real error, and is usually smaller and less important than the sum of nonrandom errors. Measurement error, confounding, specification error, and bias of the auspices, are frequently larger in all the testing sciences, agronomy to medicine. Guinnessometrics minimizes real error by repeating trials on stratified and balanced yet independent experimental units, controlling as much as possible for local fixed effects.

G-6 Economize With “Less Is More”: Small Samples of Independent Experiments

Small-sample analysis and distribution theory has an economic origin and foundation: changing inputs to the beer on the large scale (for Guinness, enormous global scale) is risky, with more than money at stake. But smaller samples, as Gosset showed in decades of barley and hops experimentation, does not mean “less than”, and Big Data is in any case not the solution for many problems.