The course proceeded in the usual way:
 onegroup designs
 twogroup designs
 manygroup designs
 factorial designs
 OLS regression
 multiple regression
 mixed factors and continuous covariates (ANCOVA)
The key to the course was the concept of a model. I spent quite a bit of time on a distribution and an observation from it. I also talked about the randomvariable notation like \(Y_i\) for each of \(i\) observations in a data set. Then, all inference in the class was model comparison. No hypotheses, only models that instantiate theoretically interesting positions. The models always have to be written down before analysis.
So, for the onegroup design, we carried two models. Students had to master the following notation:
$$ \mbox{Null Model}: \quad Y_i \sim \mbox{Normal}(0,\sigma^2) $$
$$ \mbox{Effect Model}: \quad Y_i \sim \mbox{Normal}(\mu,\sigma^2) $$
I then asked them how the model accounted for each observation. For example, if the data were the observations (2,1,0,1), we would make the following tables:
The next step is to calculate \(SS_E\), \(R^2\), and BIC, with \(R^2\) serving as a measure of effect size and BIC as a relative model comparison statistic. Of course, I taught the formulas and meanings of these three quantities. BIC, for example, was taught in terms of error and penalty, e.g., \( \mbox{BIC}=n\log(SS_E/n)+k\log(n)\). Continuing, the following modelcomparison table was produced:
NullModel Table
Data  Account  Error  Squared Error 

2  0  2  4 
1  0  1  1 
0  0  0  0 
1  0  1  1 
EffectsModel Table
Data  Account  Error  Squared Error 

2  .5  1.5  2.25 
1  .5  .5  .25 
0  .5  .5  .25 
1  .5  1.5  2.25 
The next step is to calculate \(SS_E\), \(R^2\), and BIC, with \(R^2\) serving as a measure of effect size and BIC as a relative model comparison statistic. Of course, I taught the formulas and meanings of these three quantities. BIC, for example, was taught in terms of error and penalty, e.g., \( \mbox{BIC}=n\log(SS_E/n)+k\log(n)\). Continuing, the following modelcomparison table was produced:
Model  Parameters  SSE  R^2  BIC 

Null

0

6

0

1.62 
Effect

1

5

.167

2.27 
Here we the effect was actually pretty big in \(R^2\), but we do not have the resolution to prefer the effects model over the null model given the small sample size. To help understand what \(R^2\) means, I provide a list of accountedvariances for various phenomena, say how much variance foes smoking account for in lungcancer rates.
Students were taught how to calculate all the values by calculator in small data sets and by spreadsheet in large data sets.
Interpretation
For interpretation, students were taught that:
a. their inference was only as good as their models
b. no model was true or false, all were just helpful abstractions
c. their primary goal was model comparison rather than testing
d. they need not make decisions, just assess evidence judiciously
e. they should consider both model comparison (BIC) and effect size \(R^2\) in assessment
Extension
The above modelcomparisonthrougherror approach extends gracefully to all linear model applications. In this sense, once the mechanics and interpretations are mastered in the onesample case, the extensions to more complex models, including multiple regression and multifactor ANOVA are straightforward. By getting the mechanics out early, we can focus on the models and how they account for phenomena. Contrast this to the usual case where you may teach one set of mechanics for the ttest, another for Ftests, and a third for regression.
Results
The quarter went well. The students mastered the material effectively. I had fun. They had fun. And I never lost any sleep at night wondering why I was teaching what I was teaching.
3 comments:
This is really interesting! Is there a way to get the lecture notes of your class?
JR, I was going to ask the same, but then saw that professor Rouder posted a link the files on his Twitter account.
https://twitter.com/JeffRouder/status/1109885236387545089
Files:
https://t.co/fLSRURYjjy
Thank you for making these materials available, professor Rouder.
Post a Comment