Chapter 12

Chapter 12, page 1 Math 445

Chapter 12: Strategies for Variable Selection

Chapter 12 introduces criteria for model selection and comparison and discusses some standard methods of choosing models. Model selection depends on the objectives of the study. Ramsey and Schafer identify three possible objectives (pp. 345-6) that will influence how you select a model or models: 1. Adjusting for a large set of explanatory variables. We want to examine the effect of a particular variable or variables after adjusting for the effect of other variables which we know may affect the response. 2. Fishing for explanation. Which variables are important in explaining the response? 3. Prediction. All that is desired is a model that predicts the response well from the values of the explanatory variables. Interpretation of the model is not a goal. Before considering some criteria by which a “best” model might be selected among a class of models, let’s review model development. Model Development Steps • Variable Selection: identification of response variable(s) and all candidate explanatory variables. This is done in the planning stages of a study. Note that a general rule of thumb is that you need 5 to 10 times as many observations as there are explanatory variables in order to do a good job of model selection and fitting. •

Model Formation: fitting and comparing models based on some selection criteria to determine one or more candidate models.

•

Model Diagnostics: checking for problems in the model and/or its assumptions. 1. Residual Analysis - identifying outliers, missing variables, model lack of fit, and violation of assumptions. 2. Influence Statistics - identifying influential observations, or those which have a great effect on the form of the model.

Example: Suppose we are studying differences in abundance of bird species in 3 forest habitats. The habitats represent various levels of prescribed burns. The experiment itself consists of counting the number of birds of each species type heard from a station within 100 meters in a 10-minute period. Many stations were used in the study for replication. What is the response variable? Explanatory variables? Habitat type, Neighboring habitat type, elevation, slope, aspect, visibility, etc. Suppose we have 8 candidate explanatory variables X 1 , … , X 8 . How many possible first-order models are there?

With 20 variables, there are 1,048,575 models, and these are only the first-order models. Clearly fitting all possible models is not a feasible prospect.

Chapter 12, page 2 Criteria for selecting models 1. R2 : R2 cannot decrease when variables are added so the model maximizing R2 is the one with all the variables. Maximizing R2 is equivalent to minimizing SSE. R2 is an appropriate way to compare models with the same number of explanatory variables (as long as the response variable is the same). Be aware that measures like R2 based on correlations are sensitive to outliers. 2. MSE = SSE/(n-p): MSE can increase when variables are added to the model so minimizing MSE is a reasonable procedure. However, minimizing MSE is equivalent to maximizing adjusted R2 (discussed below) and tends to overfit (include too many variables). 3. Adjusted R2 : This statistic adjusts R2 by including a penalty for the number of parameters in the model. This statistic is closely related to both R2 and MSE, as shown below. Adjusted R2 = Total mean square - Residual mean square MST − MSE MSE ( p − 1)(1 − R 2 ) = = 1− = R2 − Total mean square MST MST n− p where p is the number of coefficients (including the intercept) in the model. The third expression shows that maximizing adjusted R2 is equivalent to minimizing MSE since MST is fixed (it’s simply the variance of the response variable). •

Adjusted R2 tends to select models with too many variables (overfitting). This can be seen from the fact that adjusted R2 will increase when a variable is added if the F statistic for comparing the two models is greater than 1. This is a very generous criterion as this corresponds to a significance level of around .5.

4. Mallows’ Cp: The Cp statistic assumes that the full model with all variables fits. Then Cp is computed for a reduced model as

(σˆ

)

2 − σˆ full σˆ 2 ( ) = n − p + 2p − n 2 2 σˆ full σˆ full where p is the number of coefficients (including the intercept) in the reduced model.

Cp = p + (n − p )

2

•

Note that σˆ 2 is simply MSE (mean square error or mean square residual) for a model.

•

Models with small values of Cp are considered better and, ideally, we look for the smallest model with a Cp of around p or smaller. Some statistics programs will compute Cp for a large set of models and plot Cp versus p, as in Display 12.9 on p. 357. Unfortunately, SPSS does not compute Cp automatically.

•

CP assumes that the full model fits and satisfies all the regression model assumptions. Outliers, unexplained nonlinearity, nonconstant variance, may seriously affect the performance of Cp as a model selection tool.

•

Chapter 12, page 3 Mallow's Cp is closely related to AIC. AIC has come to be preferred by many statisticians in recent years.

5. Akaike's Information Criterion (AIC): The AIC statistic for a model is given by: ⎛ SSE ⎞ AIC = n ln⎜ ⎟ + 2p ⎝ n ⎠

where SSE = the error SS for the model under consideration and ln is natural log. •

The model with the smallest AIC value is considered best.

•

The term 2p is the penalty for the number of parameters in the model.

•

Ripley: “AIC has been criticized in asymptotic studies and simulation studies for tending to over-fit, that is, choose a model at least as large as the true model. That is a virtue, not a deficiency: this is a prediction-based criterion, not an explanation based one.'' BIC (below) is a criterion based on “explanation” approach and places a bigger penalty on the number of parameters.

•

AIC can only be used to compare models. It is not an absolute measure of fit of the model like R2 is. The model with the smallest AIC among those you examined may fit the data best, but this does not mean it's a good model. Therefore, selecting which models to consider (which variables, transformations, form of the model) and making sure the models satisfy the regression model assumptions is very important.

•

Since AIC is not an absolute measure of fit, many authors suggest reporting ∆AIC, the difference between the AIC of each model and the AIC of the best fitting model. A further suggestion is to consider all models with ∆AIC less than about 2 as having essentially equal support.

•

Neither AIC nor Cp nor R2 nor adjusted R2 can be used to compare models with different response variables.

•

AIC is based on the assumption that the models satisfy the regression model assumptions and can be greatly affected by outliers.

6. Bayesian Information Criterion (BIC). BIC is similar to AIC but the penalty on the number of parameters is pln(n) where ln is the natural log. That is,

⎛ SSE ⎞ BIC = n ln⎜ ⎟ + p ln(n) ⎝ n ⎠ BIC is motivated by a Bayesian approach to model selection and is said not to tend to overfit like AIC. Therefore, it may be better for model selection for “explanation.” The purpose of having the penalty depend on the sample size n is to reduce the likelihood that small and relatively unimportant parameters are included (which is more likely with large n).

Chapter 12, page 4 7. PRESS Statistic (not in text): another prediction-based model selection statistic is the PRESS statistic. It is calculated as follows: Remove the ith observation and fit the model with the remaining n-1 observations. Then use this model to calculate a predicted value for the left-out observation; call this predicted value Yi* . Compute Yi − Yi* , the difference between the observed response and the predicted response from the model without the ith observation in it. Repeat this process for each data value. The PRESS statistic is then defined as:

∑ (Yi − Yi* ) n

PRESS =

2

i =1

•

The model with the smallest PRESS statistic is considered “best.”

•

Leaving one item out at a time is known as n-fold cross-validation or leave-one-out crossvalidation..

•

The Yi − Yi* are called “deleted” residuals in SPSS. So the PRESS statistic can be computed in SPSS by saving the deleted residuals, creating a new variable which is the square of the deleted residuals, then computing the sum of this new variable using Analyze…Descriptive Statistics…Descriptives and choosing Sum under Options.

•

PRESS is similar to SSE, but is based on the deleted residuals rather than the raw residuals. Unlike SSE, it’s possible for PRESS to increase when variables are added to the model.

The PRESS statistic is an example of the general idea of using crossvalidation to assess the predictive power of models. A model will generally predict the data it's based on better than new data and bigger models will necessarily do a better job of predicting the data they’re based on than smaller models: SSE always decreases as more terms are added to the model. A less biased way of assessing the predictive power of a model is to use the following general idea: fit a model using a subset of the data, then validate the model using the remainder of the data. This is called crossvalidation (abbreviated CV). In k-fold CV, the data are randomly split into k approximately equal-sized subsets. Each subset is left out in turn and the model based on the remaining subsets is used to predict for the left-out subset. The PRESS statistic is based on n-fold CV, that is, only one observation at a time is left out. Simulations have suggested that smaller values of k may work better; 10-fold CV has become a standard method of cross-validation. Cross-validation is most useful as a way to compare models rather than as an absolute measure of how good the predictions will be. This is because the model used for prediction of each subset is different than the model based on all the data that will actually be used to predict future observations. Each of the models being compared should use the same splits of the data. It’s also best to repeat the 10-fold CV several times and average the results.

Chapter 12, page 5 Example Ozone data without case 17. n = 110 cases. Dependent variable is log10(ozone). All possible models with main effects and two-way interactions Model W + T + S + W:T + W:S + T:S

p 7

SSE 21.534

R2 0.695

MSE 0.209

AIC -165.39

BIC -146.49

PRESS 25.62

W + T + S + W:T + W:S W + T + S + W:T + T:S W + T + S + W:S + T:S W + T + S + W:T W + T + S + W:S W + T + S + T:S W+T+S W + T + W:T W+T W + S + W:S W+S T + S + T:S T+S W T S Constant

6 6 6

22.152 21.537 21.867 22.182 22.726 21.897 23.069 26.372 26.995 36.121 36.410 27.029 28.038 44.985 31.908 57.974 70.695

0.687 0.695 0.691

0.213 0.207 0.210

-164.28 -167.38 -165.70

-148.08 -151.17 -149.50

0.679 0.690 0.674 0.627 0.618 0.489 0.485 0.618 0.603 0.364 0.549 0.180 0.000

0.216 0.209 0.218 0.249 0.252 0.341 0.340 0.255 0.262 0.417 0.295 0.537 0.649

-163.47 -167.56 -163.82 -149.10 -148.53 -114.50 -115.62 -146.39 -144.36 -94.36 -132.14 -66.45 -46.63

-149.96 -154.05 -153.02 -138.30 -140.43 -103.69 -107.52 -135.59 -136.26 -88.95 -126.74 -61.05 -43.93

25.56 24.51 25.44 24.55 25.63 24.54 25.20 28.54 28.78 39.39 38.70 29.22 29.68 46.84 32.98 60.15 72.00

5 5 4 4 3 4 3 4 3 2 2 2 1

All possible models with main effects and quadratic terms Model W + T + S + W^2 + T^2 + S^2

p 7

SSE 20.175

R2 0.715

MSE 0.196

AIC -172.56

BIC -153.66

PRESS 23.57

W + T + S + W^2 + T^2 W + T + S + W^2 + S^2 W + T + S + T^2 + S^2 W + T + S + W^2 W + T + S + T^2 W + T + S + S^2 W + T + W^2 + T^2 W + T + W^2 W + T + T^2 W + S + W^2 + S^2 W + S + W^2 W + S + S^2 T + S + T^2 + S^2 T + S + T^2 T + S + S^2 W + W^2 T + T^2 S + S^2

6 6 6

20.754 20.875 21.270 21.393 21.818 22.614 24.924 25.390 25.998 29.996 32.958 33.350 25.466 26.418 27.207 41.263 30.579 49.093

0.706 0.705 0.699

0.200 0.201 0.205

-171.45 -170.81 -168.75

-155.25 -154.61 -152.55

0.691 0.680 0.647 0.641 0.632 0.576 0.534 0.528 0.640 0.626 0.615 0.416 0.567 0.306

0.208 0.215 0.237 0.240 0.245 0.286 0.311 0.315 0.243 0.249 0.257 0.386 0.286 0.459

-167.95 -164.01 -153.31 -153.27 -150.67 -132.94 -124.58 -123.28 -150.95 -148.91 -145.67 -101.86 -134.82 -82.74

-154.45 -150.51 -139.81 -142.47 -139.87 -119.43 -113.78 -112.47 -137.44 -138.11 -134.87 -93.76 -126.72 -74.64

23.79 23.51 24.15 23.65 24.36 25.12 28.19 27.68 28.33 32.79 35.31 36.12 28.14 28.58 29.39 43.98 32.32 51.72

5 5 5 4 4 5 4 4 5 4 4 3 3 3

Chapter 12, page 6 Approaches to choosing a model There are a number of possible approaches to model selection using the measures above to compare and select models: •

Choose several models a priori that make scientific sense. Use criteria above (like AIC and BIC) to compare models.

•

Examine all possible models involving the variables, including interactions and/or quadratic terms or both (this is what was done with Ozone data). Generally feasible only up to 3 or 4 variables.

•

Examine all main effects models only (there are 2k-1 possible models where k is the number of variables). Consider interactions or other higher order terms only after the main effects have been selected.

•

If the number of variables is large, select a subset of the variables first, perhaps based on the correlation of each of the variables individually with the response and/or eliminating redundant variables (ones which are highly correlated with another variable). Then proceed with one of the above approaches.

•

If the number of variables is large, use stepwise regression to select possible models. Stepwise regression does not require examination of all models.

Some authors do not believe in stepwise methods and other procedures that search for “good-fitting” models because they are essentially searching through many tens or hundreds of possible models, whether they make any scientific sense or not, and picking the “best” ones. The more models you consider, the higher the likelihood you will select the “wrong” one. Therefore, they believe, you should select a few models a priori that you will compare. Others argue that there is no “right” model and that if the goal is prediction, it does not matter if the model makes physical sense. In that case, crossvalidation (discussed above) might be an important tool.

Stepwise regression Stepwise regression methods attempt to find models minimizing or maximizing some criterion without examining every possible model. Stepwise methods are not guaranteed to find the best model (in terms of the criterion selected), but simply try to find the best models using a one-step at a time approach. The three most common types of subset selection methods employed are outlined below. The criterion used in these descriptions is the F statistic for comparing two nested models, but stepwise methods can also use the associated P-value, or AIC or BIC as a criterion. The latter two are now generally preferred to the F statistic or P-value. SPSS, however, only does stepwise regression with the F statistic or P-value. The three types of stepwise methods are: Forward Selection 1. Start with the model with only the constant.

Chapter 12, page 7 2. Consider all models which consist of the current model plus one more term. For each term not in the model, calculate its “F-to-enter” (the extra sum-of-squares F statistic). Identify the variable with the largest F-to-enter. Higher order terms (interactions, quadratic terms) are eligible for entry only if all lower order terms involved in them are already in the model. For example, do not consider the interaction AxB for entry unless both A and B individually are already in the model. 3. If the largest F-to-enter is greater than 4 (or some other user-specified number), add this variable to get a new current model and return to step 2. If the largest F-to-enter is less than the user-specified number, stop. The criterion could also be the P-value for the F-test, in which case a term is added only if its P-value is less than the user-specified cutoff (usually somewhere between .05 to .20). If a variable is a categorical variable with more than 2 levels, we add all the indicator variables for this variable at once. Note that once a variable has been entered it cannot be removed, even if its coefficient becomes statistically nonsignificant with the addition of other variables, which is possible. Backward Elimination 1. Start with the model with all of the candidate variables and any higher order terms which might be important. 2. Calculate the F-to-remove for each variable in the current model (the extra-sum-of-squares test statistic). Identify the variable with the smallest F-to-remove. A lower order term is eligible for removal only if all higher order terms involving that variable have already been removed. For example, the variable A is not eligible for removal if AxB is still in the model. 3. If the smallest F-to-remove is 4 (or some other user-specified number) or less, then remove that variable to get a new current model and return to step 2. If the smallest F-to-remove is greater than the user-specified number, stop. •

Again, the criterion for removal could be the P-value (remove a variable only if its P-value is greater than the cutoff).

•

Backward elimination is preferred to forward selection by many users because it does not eliminate a term unless there is good reason to (forward selection, on the other hand, does not include a term unless there is convincing evidence to include it).

Stepwise Selection This method is a hybrid of the previous two, involving both forward selection and backward elimination. 1. Start with the model with only the constant. 2. Do one step of forward selection. 3. Do one step of backward elimination. 4. Repeat steps 2 and 3 until no changes occur during one cycle of steps 2 and 3. The F-to-enter must be greater than the F-to-remove; otherwise, you could have a never-ending cycle of a variable being entered, then eliminated. If a P-value cutoff is used, then the P for entry must be smaller than the P for removal.

Chapter 12, page 8 Forward selection in SAT data (Case study 12.1) using P of .05 or less to enter. Preliminary analysis presented in text suggested that log of percent taking exam (log(takers)) should be used in place of takers. Coefficientsa

Model 1 2

3

(Constant) Log10(takers) (Constant) Log10(takers) expend (Constant) Log10(takers) expend years

Unstandardized Coefficients B Std. Error 1112.248 12.275 -135.896 9.476 1060.351 15.539 -148.061 8.459 2.900 .646 851.315 87.022 -143.383 8.272 2.698 .620 12.833 5.265

Standardized Coefficients Beta -.900 -.981 .252 -.950 .234 .127

t 90.611 -14.340 68.239 -17.504 4.488 9.783 -17.333 4.350 2.438

Sig. .000 .000 .000 .000 .000 .000 .000 .000 .019

a. Dependent Variable: sat

Excluded Variablesd

Model 1

2

3

income years public expend rank income years public rank income public rank

Beta In .078a .157a .048a .252a .221a -.057b .127b -.014b .101b -.051c .056c .369c

t .997 2.592 .755 4.488 1.028 -.783 2.438 -.254 .546 -.726 .938 1.939

Sig. .324 .013 .454 .000 .309 .438 .019 .801 .588 .472 .353 .059

Partial Correlation .144 .354 .109 .548 .148 -.115 .338 -.037 .080 -.108 .138 .278

a. Predictors in the Model: (Constant), Log10(takers) b. Predictors in the Model: (Constant), Log10(takers), expend c. Predictors in the Model: (Constant), Log10(takers), expend, years d. Dependent Variable: sat

Collinearity Statistics Tolerance .648 .960 .980 .897 .086 .533 .943 .916 .084 .532 .727 .067

Chapter 12, page 9 These three stepwise methods will not necessarily lead to the same model. In addition, changes in the F or P-to-enter and F or P-to-remove can result in more or fewer variables in the final model. The SPSS stepwise regression procedure has some disadvantages. SPSS has no way of knowing that some variables may be higher order terms that involve lower order terms. Therefore, it cannot enforce the restriction that higher order terms cannot be added before the corresponding lower order terms have been added, nor that lower order terms cannot be eliminated until all higher order terms involving them have been eliminated (that is why I used the SAT data and not the Ozone data with higher order terms in this example). SPSS also cannot treat the set of indicator variables corresponding to a categorical variable as one set of variables that should all be added or eliminated at once. However, SPSS does allow you to define blocks of explanatory variables which can be treated differently in stepwise regression. Therefore, for the ozone data, where I wanted to look at adding two-way interactions and quadratic terms, I defined Block 1 to be Wind, MaxTemp and SolarRad and Block 2 to be all the two way interactions and quadratic terms. I also defined the “Method” for Block 1 to be “Enter”, which means these variables will be in the starting model and cannot be eliminated. I also defined the “Method” for Block 2 to be “Stepwise”, which means these variables can be added or eliminated. The P-to-enter and P-to-remove were the default values of .05 and .10, respectively. will be in the starting model and cannot be eliminated. Ozone data, case #17 deleted: stepwise regression; Wind, MaxTemp and SolarRad fored to be in the model. Coefficientsa

Model 1

2

Unstandardized Coefficients B Std. Error .114 .226 -.030 .006 .019 .002 .001 .000 .518 .260 -.096 .024 .018 .002 .001 .000 .003 .001

(Constant) Wind speed (mph) Maximum temperature (F) Solar radiation (langleys) (Constant) Wind speed (mph) Maximum temperature (F) Solar radiation (langleys) Wind^2

Standardized Coefficients Beta -.308 .519 .245 -.980 .489 .247 .676

t .504 -4.779 7.830 4.248 1.992 -4.040 7.534 4.429 2.868

a. Dependent Variable: Log10(Ozone) Excluded Variables

Model 1

2

Wind^2 MaxTemp^2 SolarRad^2 WindTemp WindSolar TempSolar MaxTemp^2 SolarRad^2 WindTemp WindSolar TempSolar

Beta In .676 1.929 -.359 -.776 -.256 1.198 1.431 -.384 -.021 -.117 .933

t 2.868 2.454 -1.453 -2.049 -1.258 2.371 1.789 -1.606 -.038 -.572 1.846

Sig. .005 .016 .149 .043 .211 .020 .076 .111 .969 .568 .068

Partial Correlation .270 .233 -.140 -.196 -.122 .225 .173 -.156 -.004 -.056 .178

Collinearity Statistics Tolerance .052 .005 .050 .021 .074 .012 .004 .050 .010 .069 .011

Sig. .615 .000 .000 .000 .049 .000 .000 .000 .005

Chapter 12, page 10 One significant problem with using the F statistic or P-value is that the addition and elimination of variables is not based on a criterion for comparing models – the final model is not necessarily “optimal” in any sense. Why not add or eliminate variables based on one of the measures considered in the first part of this handout, such as AIC or BIC? The stepAIC function in the MASS library of S-Plus does stepwise regression using AIC (or BIC) as the criterion. In forward selection, it looks for the single variable which reduces AIC the most; if no variable reduces AIC, then it stops. In backward elimination, the goal is the same: find the variable whose elimination reduces AIC the most. If no variable reduces AIC when it's eliminated, then stop. In stepwise using both directions, find the addition or deletion which reduces AIC the most. Using AIC has the additional appeal of not having to set arbitrary criteria for entering and removing variables. The stepAIC function also handles categorical variables and interactions properly: an interaction cannot be added unless all the variables involved in the interaction have been added; similarly, a variable cannot be eliminated unless all higher order interactions involving that variable have been eliminated. Unfortunately, stepAIC does not handle quadratic terms properly. > m0 <- lm(sat~1,data=case1201) > summary(m0) Call: lm(formula = sat ~ 1, data = case1201) Residuals: Min 1Q Median 3Q Max -158.4 -59.45 19.55 50.55 139.6 Coefficients: Value Std. Error (Intercept) 948.4490 10.2140

t value Pr(>|t|) 92.8574 0.0000

Residual standard error: 71.5 on 48 degrees of freedom Multiple R-Squared: 2.465e-029 F-statistic: Inf on 0 and 48 degrees of freedom, the p-value is NA > stepAIC(m0,~log(takers) + income + years + public + expend + rank) Start: AIC= 419.42 sat ~ 1 Df Sum of Sq RSS AIC + log(takers) 1 199006.8593 46369.26 339.7760 + rank 1 190296.7388 55079.38 348.2108 + income 1 102026.4049 143349.72 395.0799 + years 1 26338.2438 219037.88 415.8538 <none> NA NA 245376.12 419.4176 + public 1 1231.7335 244144.39 421.1710 + expend 1 385.5838 244990.54 421.3406 Step: AIC= 339.78 sat ~ log(takers) Df + expend 1 + years 1 <none> NA + rank 1 + income 1 + public 1

Sum of Sq 20523.4615 6363.5198 NA 871.1345 785.0507 448.9059

RSS 25845.80 40005.74 46369.26 45498.13 45584.21 45920.36

AIC 313.1361 334.5429 339.7760 340.8467 340.9393 341.2993

Chapter 12, page 11 - log(takers)

1 199006.8593 245376.12 419.4176

Step: AIC= 313.14 sat ~ log(takers) + expend Df Sum of Sq RSS + years 1 1248.184463 24597.62 + rank 1 1053.599508 24792.20 <none> NA NA 25845.80 + income 1 53.329409 25792.47 + public 1 1.292761 25844.51 - expend 1 20523.461462 46369.26 - log(takers) 1 219144.737003 244990.54

AIC 312.7106 313.0967 313.1361 315.0349 315.1336 339.7760 421.3406

Step: AIC= 312.71 sat ~ log(takers) + expend + years Df Sum of Sq RSS + rank 1 2675.51301 21922.10 <none> NA NA 24597.62 - years 1 1248.18446 25845.80 + public 1 287.82166 24309.80 + income 1 19.19044 24578.43 - expend 1 15408.12616 40005.74 - log(takers) 1 190946.97826 215544.60

AIC 309.0681 312.7106 313.1361 314.1339 314.6724 334.5429 417.0660

Step: AIC= 309.07 sat ~ log(takers) + expend + years + rank Df Sum of Sq RSS AIC <none> NA NA 21922.10 309.0681 + income 1 505.3684 21416.74 309.9253 + public 1 185.0259 21737.08 310.6528 - rank 1 2675.5130 24597.62 312.7106 - years 1 2870.0980 24792.20 313.0967 - log(takers) 1 5094.3405 27016.44 317.3067 - expend 1 13619.6111 35541.72 330.7455 Call: lm(formula = sat ~ log(takers) + expend + years + rank, data = case1201) Coefficients: (Intercept) log(takers) expend years rank 399.1147 -38.1005 3.995661 13.14731 4.400277 Degrees of freedom: 49 total; 44 residual Residual standard error: 22.32106

Stepwise regression starting with the main effects model and allowing all two-way interactions. > stepAIC(mfull,list(upper=~.^2,lower=~1)) Start: AIC= 311.88 sat ~ log(takers) + income + years + public + expend + rank

+ years:public + log(takers):public + income:public + income:years - public + public:rank

Df 1 1 1 1 1 1

Sum of Sq 5027.807692 3617.792915 1977.822427 1804.755461 19.997447 1452.863422

RSS 16368.93 17778.95 19418.92 19591.98 21416.74 19943.87

AIC 300.7547 304.8035 309.1269 309.5617 309.9253 310.4340

Chapter 12, page 12 - income 1 340.339906 + log(takers):years 1 1197.663996 + log(takers):income 1 1194.412626 + income:rank 1 1046.006240 <none> NA NA + years:rank 1 485.951497 + log(takers):expend 1 447.951860 + expend:rank 1 323.487437 + years:expend 1 93.688852 + public:expend 1 51.522079 + log(takers):rank 1 44.248267 + income:expend 1 9.445369 - log(takers) 1 2150.004922 - years 1 2531.615348 - rank 1 2679.046601 - expend 1 10964.372896

21737.08 20199.07 20202.33 20350.73 21396.74 20910.79 20948.79 21073.25 21303.05 21345.22 21352.49 21387.29 23546.74 23928.35 24075.78 32361.11

310.6528 311.0570 311.0649 311.4235 311.8795 312.7538 312.8428 313.1330 313.6645 313.7614 313.7781 313.8579 314.5712 315.3590 315.6599 330.1517

Step: AIC= 300.75 sat ~ log(takers) + income + years + public + expend + rank + years:public Df Sum of Sq - income 1 193.844212 + log(takers):public 1 923.331155 + income:rank 1 869.194138 <none> NA NA + public:rank 1 587.095100 + expend:rank 1 513.555766 + log(takers):expend 1 496.074306 + log(takers):income 1 417.822552 + income:public 1 119.187306 + log(takers):rank 1 96.896741 + income:expend 1 16.336369 + income:years 1 10.664688 + log(takers):years 1 9.199796 + public:expend 1 4.688396 + years:rank 1 4.080195 + years:expend 1 3.618119 - log(takers) 1 2319.536747 - rank 1 2533.477921 - years:public 1 5027.807692 - expend 1 13670.486641

RSS 16562.77 15445.60 15499.74 16368.93 15781.84 15855.37 15872.86 15951.11 16249.74 16272.03 16352.59 16358.27 16359.73 16364.24 16364.85 16365.31 18688.47 18902.41 21396.74 30039.42

AIC 299.3315 299.9097 300.0811 300.7547 300.9649 301.1927 301.2467 301.4877 302.3966 302.4638 302.7058 302.7227 302.7271 302.7406 302.7425 302.7439 305.2482 305.8060 311.8795 328.5038

Step: AIC= 299.33 sat ~ log(takers) + years + public + expend + rank + years:public Df Sum of Sq RSS AIC + log(takers):public 1 7.036022e+002 15859.17 299.2045 <none> NA NA 16562.77 299.3315 + expend:rank 1 6.439627e+002 15918.81 299.3884 + log(takers):expend 1 6.224671e+002 15940.31 299.4545 + public:rank 1 4.726451e+002 16090.13 299.9129 + income 1 1.938442e+002 16368.93 300.7547 + public:expend 1 3.375877e+000 16559.40 301.3216 + log(takers):rank 1 1.935137e+000 16560.84 301.3258 + years:expend 1 1.528711e+000 16561.25 301.3270 + years:rank 1 8.679866e-001 16561.91 301.3290 + log(takers):years 1 5.202697e-002 16562.72 301.3314 - rank 1 2.456165e+003 19018.94 304.1071 - log(takers) 1 2.985168e+003 19547.94 305.4514 - years:public 1 5.174303e+003 21737.08 310.6528 - expend 1 1.615704e+004 32719.81 330.6919

Chapter 12, page 13 Step: AIC= 299.2 sat ~ log(takers) + years + public + expend + rank + years:public + log(takers):public Df Sum of Sq RSS AIC <none> NA NA 15859.17 299.2045 + expend:rank 1 602.5956096 15256.58 299.3063 - log(takers):public 1 703.6021875 16562.77 299.3315 + log(takers):expend 1 549.9128359 15309.26 299.4752 + income 1 413.5731794 15445.60 299.9097 + years:rank 1 141.9104795 15717.26 300.7640 + log(takers):years 1 102.4165565 15756.76 300.8870 + public:rank 1 54.7708444 15804.40 301.0350 + public:expend 1 39.7984090 15819.37 301.0813 + log(takers):rank 1 6.6716882 15852.50 301.1839 + years:expend 1 0.8878288 15858.28 301.2017 - years:public 1 2725.3253513 18584.50 304.9749 - rank 1 3086.8696076 18946.04 305.9190 - expend 1 12860.9171063 28720.09 326.3031 Call: lm(formula = sat ~ log(takers) + years + public + expend + rank + years:public + log(takers):public, data = case1201) Coefficients: (Intercept) log(takers) years public expend rank years:public 2590.556 19.42852 -134.2278 -26.43972 4.347684 5.991911 1.661026 log(takers):public -0.5848999 Degrees of freedom: 49 total; 41 residual Residual standard error: 19.66746

Stepwise using BIC > stepAIC(mfull,list(upper=~.^2,lower=~1),k=log(49)) Start: AIC= 325.12 sat ~ log(takers) + income + years + public + expend + rank Df + years:public 1 + log(takers):public 1 - public 1 - income 1 + income:public 1 + income:years 1 <none> NA + public:rank 1 - log(takers) 1 + log(takers):years 1 + log(takers):income 1 + income:rank 1 - years 1 - rank 1 + years:rank 1 + log(takers):expend 1 + expend:rank 1 + years:expend 1 + public:expend 1 + log(takers):rank 1

Sum of Sq 5027.807692 3617.792915 19.997447 340.339906 1977.822427 1804.755461 NA 1452.863422 2150.004922 1197.663996 1194.412626 1046.006240 2531.615348 2679.046601 485.951497 447.951860 323.487437 93.688852 51.522079 44.248267

RSS 16368.93 17778.95 21416.74 21737.08 19418.92 19591.98 21396.74 19943.87 23546.74 20199.07 20202.33 20350.73 23928.35 24075.78 20910.79 20948.79 21073.25 21303.05 21345.22 21352.49

AIC 315.8892 319.9381 321.2762 322.0037 324.2615 324.6963 325.1222 325.5686 325.9221 326.1916 326.1995 326.5581 326.7099 327.0109 327.8884 327.9773 328.2676 328.7990 328.8959 328.9126

Chapter 12, page 14 + income:expend - expend

1 9.445369 21387.29 328.9924 1 10964.372896 32361.11 341.5027

Step: AIC= 315.89 sat ~ log(takers) + income + years + public + expend + rank + years:public Df Sum of Sq - income 1 193.844212 <none> NA NA + log(takers):public 1 923.331155 + income:rank 1 869.194138 + public:rank 1 587.095100 + expend:rank 1 513.555766 + log(takers):expend 1 496.074306 - log(takers) 1 2319.536747 + log(takers):income 1 417.822552 - rank 1 2533.477921 + income:public 1 119.187306 + log(takers):rank 1 96.896741 + income:expend 1 16.336369 + income:years 1 10.664688 + log(takers):years 1 9.199796 + public:expend 1 4.688396 + years:rank 1 4.080195 + years:expend 1 3.618119 - years:public 1 5027.807692 - expend 1 13670.486641

RSS 16562.77 16368.93 15445.60 15499.74 15781.84 15855.37 15872.86 18688.47 15951.11 18902.41 16249.74 16272.03 16352.59 16358.27 16359.73 16364.24 16364.85 16365.31 21396.74 30039.42

AIC 312.5743 315.8892 316.9361 317.1075 317.9913 318.2191 318.2731 318.4910 318.5141 319.0487 319.4230 319.4901 319.7321 319.7491 319.7535 319.7670 319.7688 319.7702 325.1222 341.7466

Step: AIC= 312.57 sat ~ log(takers) + years + public + expend + rank + years:public Df Sum of Sq RSS AIC <none> NA NA 16562.77 312.5743 + log(takers):public 1 7.036022e+002 15859.17 314.3390 + expend:rank 1 6.439627e+002 15918.81 314.5230 + log(takers):expend 1 6.224671e+002 15940.31 314.5891 + public:rank 1 4.726451e+002 16090.13 315.0475 - rank 1 2.456165e+003 19018.94 315.4581 + income 1 1.938442e+002 16368.93 315.8892 + public:expend 1 3.375877e+000 16559.40 316.4561 + log(takers):rank 1 1.935137e+000 16560.84 316.4604 + years:expend 1 1.528711e+000 16561.25 316.4616 + years:rank 1 8.679866e-001 16561.91 316.4635 + log(takers):years 1 5.202697e-002 16562.72 316.4659 - log(takers) 1 2.985168e+003 19547.94 316.8024 - years:public 1 5.174303e+003 21737.08 322.0037 - expend 1 1.615704e+004 32719.81 342.0428 Call: lm(formula = sat ~ log(takers) + years + public + expend + rank + years:public, data = case1201) Coefficients: (Intercept) log(takers) years public expend rank years:public 3274.012 -34.05226 -164.8157 -33.8661 4.651103 5.040749 2.042115 Degrees of freedom: 49 total; 42 residual Residual standard error: 19.85829

Chapter 12, page 15 > m1 <- lm(log(ozone)~wind+temp+solar,data=Ozone) > summary(m1) Call: lm(formula = log(ozone) ~ wind + temp + solar, data = Ozone) Residuals: Min 1Q Median 3Q Max -1.0203 -0.31515 -0.0093072 0.32296 1.1222 Coefficients: Value Std. Error t value Pr(>|t|) (Intercept) 0.26236 0.52033 0.50423 0.61515 wind -0.06931 0.01450 -4.77854 0.00001 temp 0.04445 0.00568 7.82953 0.00000 solar 0.00219 0.00052 4.24768 0.00005 Residual standard error: 0.46651 on 106 degrees of freedom Multiple R-Squared: 0.67369 F-statistic: 72.947 on 3 and 106 degrees of freedom, the p-value is 0

Stepwise regression using AIC: start with the main effects model and allow all two-way interactions and quadratic terms; “lower” specifies the lowest allowable model, which is the main effects model. > stepAIC(m1,list(upper=~.^2+wind^2+temp^2+solar^2,lower=m1)) Start: AIC= -163.82 log(ozone) ~ wind + temp + solar Df Sum of Sq + I(wind^2) 1 1.67592921 + I(temp^2) 1 1.25107360 + temp:solar 1 1.17208023 + wind:temp 1 0.88700820 + I(solar^2) 1 0.45453682 <none> NA NA + wind:solar 1 0.34252408

RSS 21.392844 21.817700 21.896693 22.181765 22.614236 23.068773 22.726249

AIC -170.11663 -167.95347 -167.55592 -166.13308 -164.00908 -163.82005 -163.46557

Step: AIC= -170.12 log(ozone) ~ wind + temp + solar + I(wind^2) Df Sum of Sq RSS AIC + temp:solar 1 0.67869427353 20.714150 -171.66297 + I(temp^2) 1 0.63901036417 20.753834 -171.45243 + I(solar^2) 1 0.51800644492 20.874838 -170.81295 <none> NA NA 21.392844 -170.11663 + wind:solar 1 0.06713886979 21.325705 -168.46239 + wind:temp 1 0.00030265311 21.392541 -168.11818 - I(wind^2) 1 1.67592920662 23.068773 -163.82005 Step: AIC= -171.66 log(ozone) ~ wind + temp + solar + I(wind^2) + temp:solar Df Sum of Sq RSS AIC + I(solar^2) 1 0.7474978978 19.966652 -173.70586 <none> NA NA 20.714150 -171.66297 + I(temp^2) 1 0.2793246140 20.434825 -171.15638 - temp:solar 1 0.6786942735 21.392844 -170.11663 + wind:temp 1 0.0536327944 20.660517 -169.94815 + wind:solar 1 0.0015866564 20.712563 -169.67139 - I(wind^2) 1 1.1825432544 21.896693 -167.55592 Step:

AIC= -173.71

Chapter 12, page 16 log(ozone) ~ wind + temp + solar + I(wind^2) + I(solar^2) + temp:solar Df Sum of Sq RSS AIC <none> NA NA 19.966652 -173.70586 + I(temp^2) 1 0.2687418912 19.697910 -173.19646 + wind:temp 1 0.0981394822 19.868512 -172.24786 + wind:solar 1 0.0051540289 19.961498 -171.73426 - I(solar^2) 1 0.7474978978 20.714150 -171.66297 - temp:solar 1 0.9081857264 20.874838 -170.81295 - I(wind^2) 1 1.1811295810 21.147781 -169.38399 Call: lm(formula = log(ozone) ~ wind + temp + solar + I(wind^2) + I(solar^2) + temp: solar, data = Ozone) Coefficients: (Intercept) wind temp solar I(wind^2) 2.7000915 -0.19764083 0.016191722 -0.0024656831 0.0059294158 I(solar^2) temp:solar -0.000012334129 0.0001202964 Degrees of freedom: 110 total; 103 residual Residual standard error: 0.44028512

Stepwise using BIC (k is the multiplier on p; the default value is k=2); > stepAIC(m1,list(upper=~.^2+wind^2+temp^2+solar^2,lower=m1),k=log(110)) Start: AIC= -153.02 log(ozone) ~ wind + temp + solar Df Sum of Sq RSS AIC + I(wind^2) 1 1.67592921 21.392844 -156.61423 + I(temp^2) 1 1.25107360 21.817700 -154.45107 + temp:solar 1 1.17208023 21.896693 -154.05352 <none> NA NA 23.068773 -153.01813 + wind:temp 1 0.88700820 22.181765 -152.63068 + I(solar^2) 1 0.45453682 22.614236 -150.50668 + wind:solar 1 0.34252408 22.726249 -149.96317 Step: AIC= -156.61 log(ozone) ~ wind + temp + solar + I(wind^2) Df Sum of Sq RSS AIC <none> NA NA 21.392844 -156.61423 + temp:solar 1 0.67869427353 20.714150 -155.46008 + I(temp^2) 1 0.63901036417 20.753834 -155.24955 + I(solar^2) 1 0.51800644492 20.874838 -154.61006 - I(wind^2) 1 1.67592920662 23.068773 -153.01813 + wind:solar 1 0.06713886979 21.325705 -152.25951 + wind:temp 1 0.00030265311 21.392541 -151.91530 Call: lm(formula = log(ozone) ~ wind + temp + solar + I(wind^2), data = Ozone) Coefficients: (Intercept) wind temp solar I(wind^2) 1.1932358 -0.22081888 0.041915712 0.0022096915 0.0068982286 Degrees of freedom: 110 total; 105 residual Residual standard error: 0.45137719

Chapter 12, page 17 Bayesion posterior probabilities based on equal priors Model W + T + S + W:T + W:S + T:S W + T + S + W:T + W:S W + T + S + W:T + T:S W + T + S + W:S + T:S W + T + S + W:T W + T + S + W:S W + T + S + T:S W+T+S W + T + W:T W+T W + S + W:S W+S T + S + T:S T+S W T S Constant W + T + S + W^2 + T^2 + S^2 W + T + S + W^2 + T^2 W + T + S + W^2 + S^2 W + T + S + T^2 + S^2 W + T + S + W^2 W + T + S + T^2 W + T + S + S^2 W + T + W^2 + T^2 W + T + W^2 W + T + T^2 W + S + W^2 + S^2 W + S + W^2 W + S + S^2 T + S + T^2 + S^2 T + S + T^2 T + S + S^2 W + W^2 T + T^2 S + S^2

p 7 6 6 6 5 5 5 4 4 3 4 3 4 3 2 2 2 1 7 6 6 6 5 5 5 5 4 4 5 4 4 5 4 4 3 3 3

SSE 21.534 22.152 21.537 21.867 22.182 22.726 21.897 23.069 26.372 26.995 36.121 36.410 27.029 28.038 44.985 31.908 57.974 70.695 20.175 20.754 20.875 21.270 21.393 21.818 22.614 24.924 25.390 25.998 29.996 32.958 33.350 25.466 26.418 27.207 41.263 30.579 49.093

R2 0.695 0.687 0.695 0.691 0.686 0.679 0.690 0.674 0.627 0.618 0.489 0.485 0.618 0.603 0.364 0.549 0.180 0.000 0.715 0.706 0.705 0.699 0.697 0.691 0.680 0.647 0.641 0.632 0.576 0.534 0.528 0.640 0.626 0.615 0.416 0.567 0.306

MSE 0.209 0.213 0.207 0.210 0.211 0.216 0.209 0.218 0.249 0.252 0.341 0.340 0.255 0.262 0.417 0.295 0.537 0.649 0.196 0.200 0.201 0.205 0.204 0.208 0.215 0.237 0.240 0.245 0.286 0.311 0.315 0.243 0.249 0.257 0.386 0.286 0.459

AIC -165.39 -164.28 -167.38 -165.70 -166.13 -163.47 -167.56 -163.82 -149.10 -148.53 -114.50 -115.62 -146.39 -144.36 -94.36 -132.14 -66.45 -46.63 -172.56 -171.45 -170.81 -168.75 -170.12 -167.95 -164.01 -153.31 -153.27 -150.67 -132.94 -124.58 -123.28 -150.95 -148.91 -145.67 -101.86 -134.82 -82.74

BIC PRESS EXP(-BIC) Post. Prob -146.49 25.62 4.16676E+63 0.00002 -148.08 25.56 2.04328E+64 0.00012 -151.17 24.51 4.49052E+65 0.00254 -149.50 25.44 8.45328E+64 0.00048 -152.63 24.55 1.93360E+66 0.01093 -149.96 25.63 1.33906E+65 0.00076 -154.05 24.54 7.99954E+66 0.04522 -153.02 25.20 2.85589E+66 0.01614 -138.30 28.54 1.15592E+60 0.00000 -140.43 28.78 9.72689E+60 0.00000 -103.69 39.39 1.07645E+45 0.00000 -107.52 38.70 4.95841E+46 0.00000 -135.59 29.22 7.69111E+58 0.00000 -136.26 29.68 1.50302E+59 0.00000 -88.95 46.84 4.27065E+38 0.00000 -126.74 32.98 1.10276E+55 0.00000 -61.05 60.15 3.26346E+26 0.00000 -43.93 72.00 1.19828E+19 0.00000 -153.66 23.57 5.41614E+66 0.03062 -155.25 23.79 2.65594E+67 0.15014 -154.61 23.51 1.40046E+67 0.07917 -152.55 24.15 1.78494E+66 0.01009 -156.61 23.65 1.03481E+68 0.58499 -154.45 24.36 1.19339E+67 0.06746 -150.51 25.12 2.32093E+65 0.00131 -139.81 28.19 5.23253E+60 0.00000 -142.47 27.68 7.48057E+61 0.00000 -139.87 28.33 5.55609E+60 0.00000 -119.43 32.79 7.37547E+51 0.00000 -113.78 35.31 2.59434E+49 0.00000 -112.47 36.12 7.00004E+48 0.00000 -137.44 28.14 4.89140E+59 0.00000 -138.11 28.58 9.55897E+59 0.00000 -134.87 29.39 3.74366E+58 0.00000 -93.76 43.98 5.24144E+40 0.00000 -126.72 32.32 1.08093E+55 0.00000 -74.64 51.72 2.60459E+32 0.00000 1.76893E+68 1.00000

0.207

Example: Data were collected for each of the 50 states on the average SAT score and a number of other variables. The reason for collecting the other variables is to help explain the discrepancy between states' SAT averages. For example, many midwestern states (Montana included) have much higher SAT scores than other regions. A closer look reveals that

Chapter 12, page 18 this difference is due primarily to the fact that only the better students in these states actually take the SAT exam. Hence it is important to examine what factors affect the average SAT scores for each state. Some of the variables considered as ``explanatory'' variables were: \begin{enumerate} \item Percentage of eligible students who took the exam (TAKERS) \item Median income of families of test-takers (INCOME) \item Average number of years of study in social science, natural science, and humanities among the test-takers (YEARS) \item Percentage of test-takers in public schools (PUBLIC) \item State expenditures in hundreds of dollars per student (EXPEND) \item Median percentile ranking of test-takers within their schools (RANK). \end{enumerate} Before fitting any models, it is a good idea to examine the relationships between all pairs of variables. A scatterplot matrix and a correlation matrix are very useful. The variable TAKERS appears to have a nonlinear relationship with SAT score so we may want to consider a transformation of takers: log of TAKERS appears to work well. There also appear to be a couple of outliers; Alaska is a particularly extreme outlier on state expenditures (EXPEND). We can try For this data set, there are other possible objectives, besides finding good models for predicting SAT score. For example: \begin{quotation} {\em After accounting for the percentage of students who took the test (Log(TAKERS)) and the median class rank of the test-takers (RANK), which variables are important predictors of state SAT scores?} \end{quotation} \begin{quotation} {\em After accounting for the percentage of students who took the test (TAKERS) and the median class rank of the test-takers (RANK), which states performed best for the amount of money they spend?} \end{quotation} The first question might be examined by looking at partial correlations between SAT score and other variables after adjusting for TAKERS and RANK. Added variable plots and partial residual plots (available in S-Plus on the regression menu) allow us to look at this visually (these plots should be obtained by adding each variable separately to the model with TAKERS and RANK.

The second question could be answered in this way. First, fit the regression model involving the TAKERS and RANK variables. What do the resulting residuals tell us? The residuals are the difference in the observed SAT scores and those predicted by the variables TAKERS and RANK. A positive residual means the SAT score is higher than predicted and a negative residual means it is lower

Chapter 12, page 19 than predicted based on these 2 variables. The states could then be ranked based on these residuals. \end{document} \underline{Note:} Both AIC and BIC are available in S-Plus in the MASS library. The AIC of any fitted linear model can be obtained by the command \textbf{extractAIC(m)} and the BIC by \textbf{extractAIC(m,k=log(n))} where m is a fitted model and $n$ is the sample size. Stepwise regression using AIC or BIC is obtained from the \textbf{stepAIC} command which is illustrated on a separate handout. Example Ozone data without case 17. n = 110 cases. Dependent variable is log10(ozone).

Chapter 12, page 20 All possible models with main effects and two-way interactions Model W + T + S + W:T + W:S + T:S

p 7

SSE 21.534

R2 0.695

MSE 0.209

AIC -165.39

BIC -146.49

PRESS 25.62

W + T + S + W:T + W:S W + T + S + W:T + T:S W + T + S + W:S + T:S W + T + S + W:T W + T + S + W:S W + T + S + T:S W+T+S W + T + W:T W+T W + S + W:S W+S T + S + T:S T+S W T S Constant

6 6 6 5 5 5 4 4 3 4 3 4 3 2 2 2 1

22.152 21.537 21.867 22.182 22.726 21.897 23.069 26.372 26.995 36.121 36.410 27.029 28.038 44.985 31.908 57.974 70.695

0.687 0.695 0.691 0.686 0.679 0.690 0.674 0.627 0.618 0.489 0.485 0.618 0.603 0.364 0.549 0.180 0.000

0.213 0.207 0.210 0.211 0.216 0.209 0.218 0.249 0.252 0.341 0.340 0.255 0.262 0.417 0.295 0.537 0.649

-164.28 -167.38 -165.70 -166.13 -163.47 -167.56 -163.82 -149.10 -148.53 -114.50 -115.62 -146.39 -144.36 -94.36 -132.14 -66.45 -46.63

-148.08 -151.17 -149.50 -152.63 -149.96 -154.05 -153.02 -138.30 -140.43 -103.69 -107.52 -135.59 -136.26 -88.95 -126.74 -61.05 -43.93

25.56 24.51 25.44 24.55 25.63 24.54 25.20 28.54 28.78 39.39 38.70 29.22 29.68 46.84 32.98 60.15 72.00

All possible models with main effects and quadratic terms Model W + T + S + W^2 + T^2 + S^2

p 7

SSE 20.175

R2 0.715

MSE 0.196

AIC -172.56

BIC -153.66

PRESS 23.57

W + T + S + W^2 + T^2 W + T + S + W^2 + S^2 W + T + S + T^2 + S^2 W + T + S + W^2 W + T + S + T^2 W + T + S + S^2 W + T + W^2 + T^2 W + T + W^2 W + T + T^2 W + S + W^2 + S^2 W + S + W^2 W + S + S^2 T + S + T^2 + S^2 T + S + T^2 T + S + S^2 W + W^2 T + T^2 S + S^2

6 6 6 5 5 5 5 4 4 5 4 4 5 4 4 3 3 3

20.754 20.875 21.270 21.393 21.818 22.614 24.924 25.390 25.998 29.996 32.958 33.350 25.466 26.418 27.207 41.263 30.579 49.093

0.706 0.705 0.699 0.697 0.691 0.680 0.647 0.641 0.632 0.576 0.534 0.528 0.640 0.626 0.615 0.416 0.567 0.306

0.200 0.201 0.205 0.204 0.208 0.215 0.237 0.240 0.245 0.286 0.311 0.315 0.243 0.249 0.257 0.386 0.286 0.459

-171.45 -170.81 -168.75 -170.12 -167.95 -164.01 -153.31 -153.27 -150.67 -132.94 -124.58 -123.28 -150.95 -148.91 -145.67 -101.86 -134.82 -82.74

-155.25 -154.61 -152.55 -156.61 -154.45 -150.51 -139.81 -142.47 -139.87 -119.43 -113.78 -112.47 -137.44 -138.11 -134.87 -93.76 -126.72 -74.64

23.79 23.51 24.15 23.65 24.36 25.12 28.19 27.68 28.33 32.79 35.31 36.12 28.14 28.58 29.39 43.98 32.32 51.72