Discussion Of Results (better Presentation)

Chapter 6 Discussion of the Results A. Descriptive Statistics of the Second Visit Data Variables Table 6.1. Descriptive Statistics of the actual second visit data of the simulated variables Mean

Std.Dev

Minimum

Maximum

N

TOTEX2

102389.8 129866.6

8926.000

3203978

4130

TOTIN2

134119.4 216934.9

9067.000

4357180

4130

The above figure shows the descriptive statistics of the second visit actual data. The mean and the standard deviation of the data for TOTIN2 are larger than TOTEX2 primarily because most of the values from TOTIN2 are larger and vary significantly than TOTEX2. The statistics above will be used later in comparing the accuracy and precision of the imputed data to the actual data. B. Formation of Imputation Classes Table 6.2 Chi-Square test of independence between the matching variable candidates and the simulated second visit variables Matching Variables PROV CODES1 CODEP1 Matching Variables PROV CODES1 CODEP1

CODIN1 Test Stat P-Value 151.78 < 0.0001 613.859 < 0.0001 358.436 < 0.0001 CODEX1 Test Stat P-Value 137.83 < 0.0001 687.342 < 0.0001 193.132 < 0.0001

Degrees of Freedom 9 6 9 Degrees of Freedom 9 6 9

Table 6.3. Other measures of association between the matching variable candidates and the simulated second visit variables

Phi-Coefficient Cramer's V Contingency Test Phi-Coefficient Cramer's V Contingency Test

CODIN1 PROVINCE CODES1 CODEP1 0.192 0.386 0.295 0.111 0.273 0.17 0.188 0.36 0.283 CODEX1 PROVINCE CODES1 CODEP1 0.183 0.408 0.216 0.105 0.288 0.125 0.18

0.378

0.211

Results in the chi-square test for independence in table 6.2 and 6.3 indicates that the all matching variable candidates namely, the provincial codes (PROV), recoded education status for the first visit simulated variable (CODES1), and the recoded total employed household members for the first visit simulated variable (CODEP1) have a very significant relationship with the simulated variables, the coded total expenditure (CODEX1) and the coded total income (CODIN1) for the first visit data.

While all of the candidate variables were significant, the other measures of association namely, the Phi-coefficient, Cramer's V and Contingency Test, showed that CODES1 has the highest association among all candidate variables. In all the tests of association, only CODES1 measured at least a minimum of twenty percent to be used in dividing the data into imputation classes. In real complex data, the association between variables is expected to be lower than the requirement implemented in this study due to the variability of the observations. The CODES1 variable was chosen to be the matching variable to divide the data to its respective imputation classes in this study.

C. Mean rate of the simulated data by nonresponse rate for each nonresponse variable Table 6.4 Mean rate of observations set to nonresponse Mean rate of observations set to nonresponse TOTEX2 TOTIN2 10% 99160.23 127799.1 20% 103069.7 136098.2 30% 106309.4 142131.6

Table 6.5 Mean rate of observations retained Mean rate of observations retained TOTEX2 TOTIN2 10% 102748.6 134821.7 20% 102219.8 133624.7 30% 100709.9 130685.6

The mean rates of the observations set to nonresponse and observations retained showed contrasting results. When the nonresponse rate gets larger for both sets, the mean rate of observations set to nonresponse increases. Conversely, the mean rate of observations set to nonresponse decreases when nonresponse rate increases. It’s a possibility that large values were set to nonresponse that increased the means of the data sets containing nonresponse for the varying rates of nonresponse. D. Tests for regression model validation

E. Comparison of Imputation Methods E.1.

Nonresponse Biases and Variances of the imputed data

Table 6. Nonresponse Bias and Variance of the population mean of the imputed data for the imputation methods under the simulated variables, 10% Nonresponse rate

Methods OMI HDI3 DRI3 SRI3

TOTIN2 Bias Variance -597.839 0 -717.517 804.3251 -1128.45 0 897.1147 167.8997

Methods OMI HDI3 DREG3 SREG3

TOTEX2 Bias Variance 640.6604 0 491.9122 408.44 -720.456 0 536.3193 48.09648

Table 6. Nonresponse Bias and Variance of the population mean of the imputed data for the imputation methods under the simulated variables, 20% nonresponse rate.

Methods OMI HDI3 DRI3 SRI3

TOTIN2 Bias Variance -2855.49 0 -3095.41 1778.006 -2211.82 0 -1815.39 470.0971

Methods OMI HDI3 DRI3 SRI3

TOTEX2 Bias Variance 499.4282 0 179.4154 913.0443 -1469.57 0 1080.115 123.4498

Table 6. Nonresponse Bias and Variance of the population mean of the imputed data for the imputation methods under the simulated variables, 30% nonresponse rate.

Methods OMI HDI3 DRI3 SRI3

TOTIN2 Bias Variance -6093.27 0 -6508.65 2547.337 -4137.78 0 356.5012 726.5035

Methods OMI HDI3 DRI3 SRI3

TOTEX2 Bias Variance -222.758 0 -606.37 1344.199 -2266.38 0 398.391 154.7445

The effect of the percentage of nonresponse observations in the data is clear. For both the three methods namely OMI, HDI3 and DRI3 procedure, there is a direct relationship between bias of the population mean estimates of the imputed data and the nonresponse rate. As the nonresponse rate increase, the biases of the three methods also increase. The overall results of the hot deck method are clearly better for the TOTEX2 variable and worse under the TOTIN2 variable than the OMI. However, the deviation between the results of the bias in varying rates for the OMI and HDI3 is relatively small compared to the deviation of the HDI3 and DRI3.

In the OMI method, the biases of the imputed data gradually increase in TOTEX2 and exactly the opposite happens in TOTIN2. This increase was also seen in HDI3 and OMI3. The three methods increase in biases was tremendous since every ten percent increase in the nonresponse rate, the biases increase in magnitude by more than one hundred percent. The biases of the two methods seemed to move from an overestimation to an underestimation of population mean estimates of the imputed data as the nonresponse rate increases. These movements are seen in both variables and in both the OMI and the HDI3 method. In DRI3, the underestimation of the population estimates in both TOTEX2 and TOTIN2 seems to worsen as the nonresponse rate increases. It doubled and tripled its nonresponse bias in second and highest nonresponse rate respectively. The resulting biases of the three methods could be caused by the following reasons. For the OMI method, it is already obvious that the reason for such over and under estimations of the biases is the substitution of one value to the nonresponse observation specifically the overall mean of the first visit data. The other effects this method is seen on the discussion of the results in the distribution of the imputed data. For the HDI3 method, the reasons of over and underestimation of the estimates could be due to the number of simulations (1000), the implementation of the imputation classes in which the criteria in selecting such is not comparable with the usual procedure, or even the mechanism in the selection of donors. The last reason is always seen in many related literature. The mechanism that was used in this study is the selection of donors

with replacement. There is a chance that nonresponse observations could be substituted by the same donor from the other nonresponse observation in that data set. The only method that produced reasonable estimates is the SRI3 method. The random residual added to the deterministic predicted observation made the difference. Clearly, there is no relationship between the nonresponse bias estimates of the population mean and the nonresponse rate. The biases fluctuate from one nonresponse rate to the other. This method provided the least bias in the highest nonresponse for both TOTEX2 and TOTIN2. While the other methods reached a four digit bias, the SRI3 generated a much lesser bias than the other three methods. In fact, there is this huge disparity in the third nonresponse rate wherein it only produced less than twenty percent of the bias produced by its deterministic counterpart. The variances of the two methods namely, OMI and DRI3 were both zero because only one simulation was made. Since the estimate of the population mean is constant, automatically the variance is zero. On the other hand, the two methods which generated one thousand simulated data sets were required to compute the variance. The simulated data sets produced a distribution which can be used in order to compare the moments to each method. The variances of the SRI3 proved to be much better than its model-free counterpart which is the HDI3. In all the methods and nonresponse rate, it is clearly seen that there is a huge disparity between the variances of the SRI3 and HDI3. Variances from the HDI3 is almost ten times larger compared to SRI3 which indicates that HDI3 method likely has the same problem with the overall mean procedure in making the variability of the each observation substituted larger.

E.2.

Distributions of the imputed data vs. actual data Table 6. Kolmogorov-Smirnov Goodness of Fit Test for both total expenditure (TOTEX2) and total income (TOTIN2) of the imputed data under 10% nonresponse rate Methods OMI HDI3 DRI3 SRI3

Variables TOTIN2 TOTEX2 0.00% 0.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00%

Table 6. Kolmogorov-Smirnov Goodness of Fit Test for both total expenditure (TOTEX2) and total income (TOTIN2) of the imputed data under 20% nonresponse rate Methods OMI HDI3 DRI3 SRI3

Variables TOTIN2 TOTEX2 0.00% 0.00% 100.00% 96.90% 100.00% 100.00% 100.00% 98.40%

Table 6. Kolmogorov-Smirnov Goodness of Fit Test for both total expenditure (TOTEX2) and total income (TOTIN2) of the imputed data under 30% nonresponse rate Methods OMI HDI3 DRI3 SRI3

Variables TOTIN2 TOTEX2 0.00% 0.00% 1.00% 0.00% 100.00% 100.00% 100.00% 100.00%

Results above showed that in all nonresponse rates and variables, the overall mean imputation failed to maintain the distribution of the actual data. This was expected primarily because in each nonresponse observation from all nonresponse data sets, the

nonresponse observations were replaced by a single value which is the overall mean of the first visit of the nonresponse variable. Results from other studies stated that the overall mean imputation is one of the worst among all imputation methods. It is remarked that even if it is a simple process, inaccurate results are obviously made. Cases that vary significantly to the imputed values were the primary cause for inaccuracy. Also, the use of only a single value to be imputed for the missing data distorts the distribution of the data. The distribution of the data becomes too peaked which makes this method unsuitable for many post-analysis. (Cheng, 1999) Contrary to the results in OMI, the DRI3 methods maintained the distribution of the actual data for all nonresponse variables and rates. Results from the other studies proved otherwise. According to this study, similar to mean imputation, the distribution becomes too peaked. This distortion of the distribution is caused by the imputation of the best prediction. (Nordholt, 1998) The researchers surmise a different procedure in the study of Nordholt that contradicted the results of this study.

Results in the hot deck imputation method maintained the distribution of the data in both variables for the first nonresponse rate. However, as the nonresponse rate got larger, the distribution of the data became distorted especially in the second visit total expenditure variable (TOTEX2). From a perfectly matched distribution in the lowest nonresponse rate, the number of maintained distributions of the method dramatically dipped to nil on the last nonresponse rate. The same goes for the other variable, TOTIN2. The only difference was it maintained the distribution of the actual data set up to the

second nonresponse rate. Only one out of the one thousand simulated data sets matched the distribution of the actual data.

In the related literature, hot deck imputation performs better than mean imputation. This method reduces bias while preserving the joint and marginal distributions. This is because the hot deck imputation method allows more variability in the imputation of missing values compared to the mean imputation. (Cheng, 1999)

However, another related literature showed its disadvantages and negative effects on the distributional properties of the imputed data. The hot deck imputation procedure may easily give rise to multiple uses of an observation in the donor record. This event occurs whenever a set of nonresponse records is replaced by a single value from the donor record. (Kalton, 1983) When this happens, there is a chance that the distribution of the data might be distorted.

Lastly, results from the stochastic regression imputation performed better than its model-free counterpart that is the hot-deck imputation method. Unlike in hot deck imputation, stochastic regression imputation maintained the same distribution for all imputed data sets for the first and third nonresponse rates. It also outperformed the former in the second nonresponse rate, TOTEX2 variable.

In earlier studies, the stochastic regression imputation performs better than any of the four methods used here. The random residual was added to the deterministic predicted

value to preserve the distribution of the data. However, even if the original deterministic imputed values were feasible, the stochastic counterpart need not be. After adding the residual to the deterministic imputation, unfeasible values could namely result. (Nordholt, 1998) Again, the authors’ procedure in producing estimates for the said imputation methods was not the same procedure used in this study.

E.3. Other measures of accuracy and precision of the four imputation methods for imputing for nonresponse observations under TOTEX2 and TOTIN2. TOTEX2, 10% NONRESPONSE RATE METHODS OMI HDI3 DRI3 SRI3

MD -6406.603632 4919.395646 -7204.560545 5363.466919

Rank 3 1 4 2

MAD 56929.613669 78071.611061 23839.817150 33683.480259

Rank 3 4 1 2

RMSD 108547.820041 79251.216593 57726.615582 70553.643550

Rank 4 3 1 2

In the table above, the mean deviations vary between the various imputation methods. The two imputation methods, namely the overall mean imputation (OMI) and the deterministic regression imputation (DRI3), generated negative mean deviations indicating that the imputed values underestimated the nonresponse values of TOTEX2. On the other hand, the hot deck imputation method using three imputation classes (HDI3) and the stochastic regression method using the same set of imputation classes of the former (SRI3) which used randomization generated positive mean deviations, indicating that on the average, imputed values overestimated the actual (deleted) values of TOTEX2. The average overestimation of HDI3 fared the best in generating unbiased estimates. On the other hand, DRI3 finished last in generating unbiased estimates. All of the imputation methods generated more biased estimates than the bias in estimating the populations mean ignoring the nonresponse values. The use of the hot deck method in the values of HDI3 and random residual in SRI3 showed better estimates.

The mean absolute deviation (MAD) and the root mean square deviations (RMSD) of the DRI3 procedure fare best in terms of these values, having the smallest measures on both of them. The table above shows that the two procedures namely the DRI3, and SRI3, which used a regression model, yielded the best estimates for both measures. Surprisingly, HDI3 fared the worst in MAD and third in RMSD while OMI fared third in MAD and last in RMSD.

TOTIN2, 10% NONRESPONSE RATE METHOD S OMI HDI3 DRI3 SRI3

MD

Rank

MAD

Rank

RMSD

Rank

5978.393462 -7175.254063 -11284.461504 9043.982223

1 2 4 3

77502.270650 105369.153855 32115.804981 51363.168122

3 4 1 2

167206.240181 242022.994540 77228.476946 106374.388796

3 4 1 2

In the figure above, the mean deviations vary from one method to the other. Unlike in TOTEX2 with the same nonresponse rate, the two methods HDI3 and DRI3 generated negative estimates for MD indicating that the imputed values underestimated the actual values. On the other hand, the two other methods namely OMI and SRI3 generated positive estimates for MD indicating that the imputed values overestimated the actual values. While in TOTEX1 the OMI fared the second to the worst in generating unbiased estimates, it generated the best unbiased estimate. The DRI3 procedure have the same result for TOTIN2 having the largest value in MD. Both procedures which used models in imputing missing values fared worst than the other imputation procedures. Almost same results were generated in TOTIN2 except that HDI3 fared worst in both measures. The mean absolute deviation (MAD) and the root mean square deviations (RMSD) of the DRI3 procedure fare best in terms of these values, having the smallest measures on both of them. The table above shows that the two procedures namely the DRI3, and SRI3, which used a regression model, yielded the

best estimates for both measures. Estimates from the DRI3 procedure for both measures were three times smaller than the estimates from the HDI3 procedure. TOTEX2, 20% NONRESPONSE RATE METHODS OMI HDI3 DRI3 SRI3

MD -2497.141162 897.178516 -7347.862975 5400.712813

Rank 2 1 4 3

MAD 59555.355146 78292.631254 23231.654338 33782.602655

Rank 3 4 1 2

RMSD 119193.320481 67149.156877 53180.024322 72487.392833

Rank 4 2 1 3

In the table above, results from the second nonresponse rate was different from the results of the lowest nonresponse rate. Only the DRI3 procedure did not change its ranking while the rest of the imputation procedure swapped positions. As in the first nonresponse rate, the OMI and DRI3 generated negative mean deviations while the HDI3 and SRI3 generated positive values. In the mean deviation, HDI3 fared the best by generating the smallest estimate. Both regression IMs fared the worst in generating unbiased estimates than the other methods. Again, the mean absolute deviation and the root mean square deviation for DRI3 yielded the best for both measures. It seems that HDI3 and OMI for both ten and twenty percent nonresponse rate for TOTEX2, generates the worst estimate for MAD and RMSD respectively.

Similar to the results from TOTIN2 and TOTEX2 with ten percent

nonresponse rate, estimate from the DRI3 is three times smaller than the estimate of HDI3 for MAD.

TOTIN2, 20% NONRESPONSE RATE METHOD S OMI HDI3 DRI3

MD

Rank

MAD

Rank

RMSD

Rank

14277.427361 -15477.092677 -11059.090609

3 4 2

87469.865623 111748.043527 35274.032863

3 4 1

244757.995335 297151.501265 114957.425614

3 4 1

SRI3

-9076.980826

1

57429.236572

2

148278.489381

2

A different result for this data set and nonresponse rate was generated. In all the methods used, only the OMI procedure generated positive results which imply that it overestimates the actual values. On the other hand, the three other methods generated negative values which imply that it underestimates the actual values. Far from the results generated above, HDI3 and OMI were the worst and the second to the worst IM in measuring the effectiveness of the method used. All methods generated larger biases than the bias of the population ignoring nonresponse observations. In all the tests, both regression imputation methods showed better results than the two other methods. Again, the HDI3 estimates for the MAD and RMSD were almost three times larger than DRI3.

TOTEX2, 30% NONRESPONSE RATE METHODS OMI HDI3 DRI3 SRI3

MD 742.526312 -2021.192269 -7554.607707 1328.061322

Rank 1 3 4 2

MAD 62388.111315 81395.790207 24082.880071 32449.488519

Rank 3 4 1 2

RMSD 151740.940804 71390.645266 59795.667757 72803.602152

Rank 4 2 1 3

In the results above, the OMI procedure fared the best in mean deviation. Similar to the results of the other nonresponse rates, the DRI3 finished last in generating unbiased results for nonresponse observations. OMI and SRI3 mean deviations were better than bias in estimating the population mean ignoring the nonresponse observations. Like the results in the first nonresponse rate of the TOTEX2 variable, both HDI3 and DRI3 generated negative values for mean deviation. It is interesting to note that the mean deviation for this nonresponse rate under TOTEX2 among all nonresponse rates for OMI was the smallest. It indicates that as the nonresponse rate increases, the mean deviation gets larger and larger under this variable. Similar to the previous nonresponse rates, the HDI3 and the OMI fared the worst in the mean absolute deviation and root mean square deviation categories respectively.

On the other hand, DRI3 topped all methods in terms of the measure of “closeness” in all nonresponse rates under TOTEX2. TOTIN2, 30% NONRESPONSE RATE METHOD S OMI HDI3 DRI3 SRI3

MD

Rank

MAD

Rank

RMSD

Rank

20310.908394 -21695.518140 -13792.599597 1188.309972

3 4 2 1

90396.258755 115087.128145 34537.359021 51886.726288

3 4 1 2

271775.351071 313814.916561 103253.122885 131429.611649

3 4 1 2

The rankings of the second nonresponse rate were exactly the same with the results above. HDI3 values for both measures of closeness were three times larger than best IM for those categories. The regression imputation again bested the other two methods in all criteria. As in the results from other nonresponse rates under this nonresponse variable, the HDI3 and DRI3 underestimated TOTIN2 and OMI and SRI3 overestimated TOTIN2. Among all the methods, SRI3 has the smallest MD and the only method which has a value lower than ten thousand. While in TOTEX2 wherein the mean deviation of OMI was the smallest, the mean deviation of the TOTIN2 in this nonresponse rate was the largest. Same observation from TOTEX2 under the largest nonresponse rate was applied here. However in TOTEX2, the mean deviation gets even better while here the mean deviation gets worse. To summarize all the results shown above, deterministic regression showed better results in almost all categories under both nonresponse rate and variables. The regression model validation results especially the coefficient of determination could be the key why the estimates of the imputed values were better than the other methods. Earlier studies indicated that the deterministic regression imputation can provide a relatively higher accuracy as compared to the other methods if the chosen factor is very highly correlated to the dependent variable. (Cheng, 1999) Overall mean imputation was the least method to provide good results in the accuracy and precision of the imputed data. Hot deck imputation method was just little

better than overall mean imputation and surprisingly much worse than deterministic regression though in other related literature, it’s the other way around. Earlier related studies stated that hot deck performs better than the two imputation methods namely, the overall mean imputation and the deterministic regression imputation. This is because hot deck imputation allows more variability in the imputation of missing values compared to the overall mean imputation wherein every missing value is replaced by a constant value. (Cheng, 1999) Results were same in the study by Ford that the results using hot deck with the considerations of subclasses produced a smaller amount of bias and also had a smaller relative square root of the mean square error (Ford, 1976). In an undergraduate thesis, the hot deck imputation method yielded better results than mean imputation method. (Salvino, 1993)