Chapter 25 Appendix: Data by design

25.1 Approaches to collecting data

https://en.wikipedia.org/wiki/Nonprobability_sampling

Focus on observational data.

1. What population are the results to be applied to? The sampling frame is your representation of this.
   • census
   • patients, purchasers, college grads, …
     • can introduce spurious correlations (Give example in simulation chapter with a collider)
2. Sampling
   • simple random sampling
   • convenience sampling (not good!) – contrast it with simple random sampling when the sampling frame is a census
   • cluster sampling
   • case-control
   • longitudinal versus cross-sectional
     • prospective cohort
     • retrospective cohort
     • survival bias
3. Experiment and orthogonality: you are part of the system!

Then introduce experiment as a technique for dealing with covariates and pro

1. Common collection practices and their consequences
   0. Census (example: employment discrimination and hypothesis testing)
   1. Simple random sample
   2. Cluster sampling
   3. Case/control study (always retrospective. Refer to example about the HBC high-blood pressure study in classification error)
   4. Cohort (prospective?)
   5. Retrospective cohort study
   6. Experiment (refer to fun examples from Kahneman’s Thinking Fast and Slow)
   7. Pathetic reaching out
      • telephone polling
      • convenience samples
2. Measuring covariates
3. Proxies for latent variables.

Additional precautions against confounding/sampling-bias

1. Random selection within strata. Why? Provides a kind of certificate believable by others, that your sampling wasn’t biased.

2. Assignment. Cut off any possible relationship X <- C. You can do this by instituting U -> X C -> Y and possibly X -> Y. But, you can fool yourself (“this patient isn’t healthy enough to be put on the experiment drug” or “this is a lost cause, so no risk in trying the new drug”). So random assignment is a good way to go.

Still, keep a record of covariates.

Intent-to-treat and instrumental variables approaches.

SEE MATERIALS IN 033-samples.Rmd

To avoid unnecessary and confusing abstraction, I’ll start with a specific setting: predicting whether a person will develop diabetes. Ideally, you should have a precise definition of “develop diabetes”: what constitutes diabetes, the time horizon of the prediction, and so on. There are many ways in which appropriate data can be collected. The different ways are called study designs.

One possible study design is a prospective cohort study. A cohort is a group of subjects with some features in common and others that differ. In a cohort study, a group of subjects are identified (“assembling” the cohort), say adults in the US with no previous signs of diabetes, and relevant observations made of existing conditions which will play the role of explanatory variables, e.g. age, sex, weight, diet, exercise, … whatever you think might be relevant. In a prospective study, the group is then followed forward over time and the eventual outcome – developing diabetes in our example – is recorded. To use the data for prediction for a new subject, compare the conditions for the new subject (age, sex, weight, etc.) to the set of observations in the cohort subjects. Pick out the members of the cohort whose original observations best resemble those of the new subject. Let’s call this the matching subset of the cohort. Then tabulate the eventual outcomes of the matching subset. This tabulation constitutes the prediction for the new subject.

It’s helpful to distinguish a prospective cohort study from a retrospective cohort study. In a retrospective study, the cohort subjects are not followed forward over time but backward. That is, the outcome is already known at the time the cohort is assembled. The values of the explanatory variables are extracted from historical records, say, the subjects’ medical records.

THIS IS FOR DETECTION, NOT PREDICTION. Still another design is a case/control study. If the outcome being studied is rare, a cohort might have to be very large in order to include people with the condition to be predicted. To illustrate, suppose we believe that drinking large amounts of sugary beverages (e.g. “Big Gulps”) is a factor that can be used to predict the onset of diabetes. For instance, according the the US Centers for Disease Control and Prevention (“National Diabetes Statistics Report, 2017” 2018), in a randomly selected group of 1000 US adults without signs of diabetes, approximately 7 will develop diabetes in the next year.

Possible introduction or enrichment for need to look at multiple variables. Hill-1937a-WA-I.pdf under “Definition of Statistics” and the following “Planning” section.

25.2 Generalization

A nice set of examples of selection bias is in Hill-1937a-WA-II.pdf.

25.3 Create model representation of your system.

A model is a representation for a purpose. You want something you can easily manipulate because you are going to be doing experiments on the model to understand/interpret the real-world system better.

25.4 Evaluate technical performance.

Feedback loop with (4)

25.5 Interpret and communicate.

Apply loss functions.

Express risk sensibly, attribute risk (causality) responsibly.

Express your uncertainty/ Standardize your results (adjustment) to help decision-makers see contrasts that are meaningful. (Example: Mexico and US death rates.)

Be attentive to false discovery.

Don’t be afraid to frame things in terms of causation, but only do so if you have handled the possibility of confounding in a responsible way.

Not p-values: - ASA editorial March 2019 - Nature article March 2019

25.6 On a second reading

After reading the methods, go back and re-read this.

Other points …

25.7 Sensitivity

About why we shouldn’t rely only on sampling variation to indicate what we know about the effect size. We should look at a variety of models and model architectures, proxies for the measured quantities (since a variable is not necessarily what we want it to be) to get a sense of how much variation there is among models of equal plausibility.

“The Statistical Confidence Game” – why do we focus on doing the same thing over and over?

Should get similar results when using different proxies for the effect, e.g. in the SAT data use expenditures, but also teachers’ salaries and class size, perhaps building and administrative expenses, ….

Motivated by Michael Lavine’s essay the The American Scientist. “Frequentist, Bayes, or Other?”" Michael Lavine https://doi.org/10.1080/00031305.2018.1459317

Also see Steven Ziliak’s article in the same issue, from which these quotes are taken:

G-7 Minimize “Real Error” with the 3 R’s: Represent, Replicate, Reproduce

A test of significance on a single set of data is nearly valueless. Fisher’s p, Student’s t, and other tests should only be used when there is actual repetition of the experiment. “One and done” is scientism, not scientific. Random error is not equal to real error, and is usually smaller and less important than the sum of nonrandom errors. Measurement error, confounding, specification error, and bias of the auspices, are frequently larger in all the testing sciences, agronomy to medicine. Guinnessometrics minimizes real error by repeating trials on stratified and balanced yet independent experimental units, controlling as much as possible for local fixed effects.

G-6 Economize With “Less Is More”: Small Samples of Independent Experiments

Small-sample analysis and distribution theory has an economic origin and foundation: changing inputs to the beer on the large scale (for Guinness, enormous global scale) is risky, with more than money at stake. But smaller samples, as Gosset showed in decades of barley and hops experimentation, does not mean “less than”, and Big Data is in any case not the solution for many problems.