Quantitative Research Methodology

Pusat Pengajian Ilmu Pendidikan Kolokium Pelajar Pengajian Siswazah PPIP 2005 8.00 - 10.00 pagi. Input Session IV, 16 Disember 2005 (Jumaat) Quantitative Research Methodology by Mokhtar Ismail Introduction The purpose of doing research is to add a new knowledge to the existing body of knowledge in an area of interest. A research problem therefore should not be a tedious one. It should be carefully thought and polished before going to the field for data collection. To compose a good research problem is time consuming. Sometimes our first idea about an issue is not really a research problem as yet, until the idea is discussed and expanded. The best method of expanding the research problem based on the first idea is by searching through the journal articles on what other people have done in the area. It is worth spending more time in polishing up a research problem than going too early to start the research. If a research problem needs quantitative approach to address the research questions related to the problem, the researcher should be ready to equip him/herself with sufficient knowledge of quantitative research methodology pertinent to the research itself. For that matter not every body can do research. Only those who are patient, persevere, hardworking and critical minded can. One should be patient in trying to understand the statistical concepts that underlie the rationale of making generalization of the research results. One should be persevere in selecting the most valid instrument for the research. One should be hardworking in looking for very related models or theories that underlie the research premise and one should be critical minded in analyzing the research data. To begin with one should learn about basic descriptive and inferential statistics and types of research and make sense out of the linkages between the two. Besides, one should also learn about the sequence of reporting research report: Chapter one to five (or sometimes more than five) and the skills on how to carry out literature review. The focus of this paper is on making sense out of the linkages between statistics and types of research because this constitutes the major difference between quantitative and qualitative research approach. Types of Research Fitting a research problem to a specific type of research is quite a task. Since one has to be crystal clear on the relationships among the variables in the research problem before deciding on the types of research to be adopted. There are two major kinds of relationships, i.e., with or without cause-and-effect. A cause-and-effect relationship demands experimental, quasi-experimental, ex-post-facto method or probably a time

2 series design for longitudinal observations. A non cause-and-effect relationship requires a plain descriptive research describing about the pattern of relationships among the variables. Let us take a look at a brief research problem as an example in selecting a type of research: Teaching of Science and Mathematics in English creates a lot of problems among students as well as teachers. The question ‘Do students actually benefited from the new policy?’ really bothers every citizen. Based on the above research problem many types of research could be planned depending on the focus by a researcher. For example, let cause-and-effect relationship be the focus, i.e., the interest of every citizen would be ‘Did the new policy causes students performance to drop?’ Thus, the relevant types of research are, a pure experiment, a quasi-experiment, an ex-post-facto or a time-series design: Type 1: A pure experiment. This type enables us to manipulate an independent variable (iv) in order to see the effect on the dependent variable (dv). With regard to the above research problem, the iv is ‘implementation of new policy’ and the dv is students performance. Practically, it is not possible to manipulate the iv in the above example, therefore a true experiment is out of question. Type 2: A quasi experiment. The only difference between a quasi experiment and a true experiment is that, in quasi experiment there is no randomization of subjects between levels of the iv, for instance between control and experimental groups. With regard to the above research problem a quasi experiment is also out of question for similar reason as Type 1. Type 3: Ex-post-facto or causal-comparative. A causal relation could also be established by causal-comparative method although not as strong as the experimental method. The method is suitable when iv is already available in the research setting. As for the above research problem, this type could be the closest in order to search for a cause-and-effect relationship between the iv and the dv. The iv is already there in the research setting: ‘implementation of new policy’. Our job is to search for the levels of the iv namely subjects who are being exposed to the new policy and subjects who are not. For example: if the new policy has been carried out for last three years, the comparison in Mathematics ability could be made between end of year three pupils this year (who were taught in English) and beginning of year four pupils (who were taught in Malay). Type 4: Time series design. A cause-and-effect relationship could also be established using a time series design. A series of observations based on a defined duration between observations are recorded for a group of subjects before and after a treatment is given. If we find that the performance of the subjects are consistently higher after the treatment, than the effect has taken place and it is caused by the treatment. As for the above research problem, probably it is not possible to carry out this research design since the treatment

3 which is ‘the new policy’ has already taken place. Had we known that this policy was going to be implemented last three years we could have gathered time series data way before that time, in order to see the effect after the policy is being implemented. Now let us use the similar research problem and try to fit in the types of research which are non-cause-and-effect in nature or also known as descriptive research. Type 5: Survey research. If the focus is not so much on A causes B, but rather the description of a phenomena such as relationship among variables, survey research is appropriate. For example, the concern is about views of general public including parents about the new policy, i.e., how the new policy has affected achievement of their children in Mathematics. Type 6: Correlational research. A researcher has prior knowledge about correlations among variables in previous studies similar to the one in the above research problem. For instance, there is a high correlation between ability to speak English and number of English vocabularies of each pupil. Type 7: R&D type of research. The focus is on development of a prototype and a validation process to justify its usefulness. With regard to the above research problem, probably the interest is on developing a new teaching method and this method is going to be validated by means of an experiment. PTPM students like to do this type of research. They develop courseware and they validate its effectiveness by means of an experiment. Psychometric students too carry out this type research. They develop tests and validate them by presenting validity evidences of various kinds: content, predictive and construct. Type 8: Evaluation research. The focus is on evaluating an event by means of the above types of research and to make judgment about its usefulness. This type of research is probably not truly quantitative due to the elements of value judgment made by the researcher. With regard to the above research problem, the focus is on for example the effectiveness of teaching of Mathematics courseware currently used by teachers. There is a slight difference between Evaluation Research and Program Evaluation. The latter is for decision making purposes rather than adding new knowledge to the body of existing knowledge in the area of interest and is usually based on existing program evaluation models pioneered by Ralph Tyler. There are sometimes overlaps in terms of procedures between each research type: 1 through 8, and sometimes it is not easy to categorize them objectively according to each of the type. Nevertheless a researcher should have some idea on what is the best name to be assigned to his or her research type. If the above types do not fit a researcher’s focus, probably qualitative research approach is the answer. This approach requires totally different philosophy in terms of ‘social reality’. As for the above research problem the interest is on the process rather than the outcome. For example, the researcher is interested in the mechanics of pupil comprehension process when they listen in English and think in their native language.

4 How does learning take place under this condition? Is there hindrance along the learning process? These are issues most appropriately addressed by a qualitative research approach. Hypothesis Testing and Types of Statistical Analysis Generalization process from sample to population is the intention of a quantitative as opposed to a qualitative researcher. In research, only one sample of subjects is studied and based upon characteristics of that sample, generalization is made back to the population where the sample is formerly chosen. Normally, characteristic of the sample is represented by the sample mean. In quantitative research the big question is whether that sample mean represents the population mean? The question is quite tricky and should be clearly understood by every quantitative researcher. Data collection activity normally starts with a sampling procedure when a random sample is carefully chosen. The sampling has to be random because we have to make use of the rationale of the probability theory that underlies the distribution of the sample means since in research we only pick one sample and compute the mean. This mean is one the many means in the distribution of the sample means if we endlessly pick up a sample and compute the mean from a defined population. The distribution of the sample means, picked up endlessly, will actually form a normal distribution. This is known as the Central Limit Theorem. In addition, the standard deviation of the distribution of the sample means is known as standard error, which can be estimated by the standard deviation of the sample and the sample size, and the mean of the distribution of the sample means is of cause the population mean. The value of standard error of the mean has bearing on the representativeness of a sample mean in a population. If it is small the implication is that the sample means are close to population mean and vice versa. In research we expect the standard error to be small because what it means is that, the sample mean that we obtain from our study is close to the population mean. In other words, our sample is representative. The question is how close is the sample mean to the population mean in order for a researcher to decide that his or her sample mean represents the population mean? To decide on the closeness, statisticians have developed hypothesis testing procedure. To make the testing procedure meaningful they start with the concept of null hypothesis or empty hypothesis. What they want to say is that, if the sample mean is truly close to or far from the population mean, that should happen beyond reasonable doubt (we borrow a legal term). It statistics, we say it differently, beyond purely chances alone. The null hypothesis is always presented in the following form: There is no significant difference between A and B. In our above example, we could say that, there is no significant difference between sample mean and population mean (A and B can also be correlations or variances). If we can reject the null hypothesis beyond reasonable doubt, we are sure that our sample mean is close to or far from population mean, therefore our

5 sample mean represents the population mean. We actually do not know what really happens in the population. That is why we do research. The next question is how to measure the distance of the sample mean to the population mean so that we are able to reject the null hypothesis beyond reasonable doubt? To start with, we should learn about the concept of z-score, i.e. a kind of derived score or standardized score transformed from a raw score. In any frequency distribution of scores, we can always transform raw scores into derived scores. One of them is z-score. A zscore is defined as the difference of a raw score against its mean in standard deviation unit (a z-score is raw score minus mean divide by standard deviation). This is just a linear transformation between raw scores into the standardized scores. The frequency distribution of standardized score is not necessarily normal, but if it is normal it is known as a standard normal frequency distribution or we usually call it as normal distribution. The normal distribution has the well known characteristics namely: 68 percent of the cases (we call it cases because y axis represents frequency of the distribution, so the area under the curve actually represents the density of frequencies or cases) are between 1 standard deviation below and above the mean, 95 percent of the cases are between 2 standard deviation below and above the mean, and 99 percent of the cases are between 3 standard deviation below and above the mean. Let us go back to the original question of how to measure and later decide on the closeness between a sample mean to the population mean by applying the null hypothesis concept. After getting the sample, we compute sample mean and sample standard deviation. From the sample standard deviation and sample size, we compute standard error of the mean. The standard error tells us about the standard deviation of the frequency distribution of sample means, if we pick up samples endlessly and compute their means respectively (but remember in research we only randomly pick up one sample, and yet we can still estimate the standard error). The next job is to compute the difference between the sample mean and the population mean (or hypothesized mean) in terms of standard deviation unit using the z-score concept where the numerator is the difference between sample and population mean and denominator is standard error. This activity is known as computation of test statistic. In order to know whether we can reject the null hypothesis, we have to compare the test statistic value with the critical value. As for our example, we have to compare the z-value that we get with the z-value in the standard normal frequency distribution which has the characteristics as mentioned above. If the test statistics value falls in the critical region, i.e. more than two standard deviations below or above the hypothesized mean, normally the decision is to reject the null hypothesis. The rationale is: if the null hypothesis is true, it is highly unlikely that the test statistics value to appear (the probability is less than five percent), but it does appear, so the decision is to reject the mull hypothesis. Practically, the z-score concept in computing the test statistic and testing it, is applicable to other types of test statistic. The types of major test statistic are given on Appendix A. The most common test statistic is independent sample t test for testing difference of two means. t test is actually the extension of z-score concept, and the t test is meant for smaller samples as compared to z-test.

6

Conclusion The intention of this paper is to highlight the most crucial statistical aspect of quantitative research methodology in relation to the types of research. There are a lot more other aspects that have to be considered which are not discussed. Copyright by Mokhtar Ismail 2005

7

Appendix A TYPES OF STATISTICAL ANALYSIS

Number of dependent variable (dv) ________________________________________ one two or more (univariate) (bivariate no iv or dv) (multivariate) ___________________________ between subject within subject _____________ one iv two or more iv ____________ two more than groups two groups

_____________ two levels two or more levels

______________________ relationship: between subject: cannonical, manova, discriminant mancova

Correlated t-test

two or more way ANOVA

Repeated Measue ANOVA

Oneway ANOVA Independent Sample t-test bivariate correlation, multiple regression Notes Correlations: statistical and practical significance Counterparts: parametric and non-parametric Input layout for ANOVA: one or more way

8 ANCOVA: relationship between covariate and dv Analysis: choose the most efficient Pls check with sugino notes on test statistics. Pls check with sugino notes on test statistics. Identification of variables Example on quantitative data analysis Validity of an experiment (threats) Research Plan SPSS package Choice of research approach: quantitative versus qualitative Sampling Summary