Why Use Experiments

Why use experiments for program evaluation? David W. Nickerson [email protected] Jared P. Barton [email protected] Broad Overview and Explanation One of the challenges of managing fieldwork is evaluating the success of the program. Program implementers want to replicate successful programs, and also to learn from (and avoid repeating) unsuccessful programs. It is also valuable to understand how costly achieving a goal is, so organizations can plan cost-effectively in the future. Experiments can help answer both questions of success and of cost-effectiveness, but to do so the research protocols need to be designed and implemented prior to the fieldwork. When nonprofit organizations evaluate their operations, they often rely upon observational data passively collected during the course and at the end of their programs. For instance, suppose an organization selected precincts in which to mobilize voters for an upcoming election. The group selected precincts on the basis of the density of AsianAmerican population and the lack of secure apartment buildings to facilitate canvassing. A typical observational study would take careful notes on the activities in the selected precincts and then compare the rates of turnout of these particular precincts to the neighboring precincts. Suppose the group discovered that in its selected precincts turnout was 5 percentage points higher than in neighboring precincts -- even after "controlling for" age and past turnout. Can the group safely conclude that its program was a success? Not necessarily. There may be other reasons that these precincts had higher turnout that are completely unrelated to the mobilization efforts of the group. For instance, perhaps the requirement that the precincts have few secure apartment buildings pushed the campaign into slightly wealthier neighborhoods with more single-family housing. Since income is positively correlated with voter turnout, it is possible that the type of neighborhood selected is responsible for the 5 percentage point difference. Thus, the difference in turnout could be the result of unobserved factors and not the result of the mobilization campaign. The potential causes could be even more subtle and unseen. For instance, suppose the Asian-American neighborhoods populated by the campaign are gentrifying. The higher voting rate might be a result of the changing demographics of the community rather than the mobilization campaign. In short, it is impossible to know whether the boost in turnout was the result of the mobilization campaign. Moreover, the campaign managers do not know how much of the success can be attributed to each technology. Was it the door-to-door knocking? Posters? Telephone calls? How much did each technology contribute to the success of the campaign? Were the early efforts conducted in September as effective as the efforts in the week prior to the

election? Was it even worth working in the field a month prior to the election? These are questions that an observational study is unable to answer. In contrast, a properly designed experiment can provide answers to each of these questions. At its most basic, an experiment manipulates the application of campaign techniques using the following steps: 1) The list of targets are identified and listed -- these are the people the campaign would like to contact with the technique; 2) The people listed as targets are randomly divided into a treatment group, where people are contacted using the technique, and a control group that is not contacted by the campaign (note: the division need NOT be 50-50); 3) The campaign then goes about its business contacting the treatment group (and leaving the control group alone); 4) The outcome variable (e.g., registration or turnout or support for a ballot proposition) is then measured. Because the assignment to treatment and control groups is random, they should be composed identically. They should be the same age, the same education, the same vote history, and the same income on average. Due to randomization, the treatment and control group should also be identical with regards to matters that the campaign can't know such as interest in the election, personal connections to the campaign, or convenience of driving to the polls on Election Day. The only difference between the two groups should be who receives the treatment. We say “should be” because the similarity of the two groups due to randomness depends upon the size of the overall population; we need a sufficiently large group to use this property in assuming sameness. Thus, to measure the effectiveness of the campaign, the manager need only compare the rate of turnout in the treatment group to the control group. If one voted at 45% and the other 37%, then the campaign boosted turnout in the treatment group by 8%. Moreover, we can then calculate the expenses of the campaign and determine how much it cost to generate each "new" vote. For instance, say the campaign that boosted turnout by 8 percentage points spent $5 on each attempt to contact a member of the treatment group. Then, we could conclude that each new vote cost $62.50. The manager now has the ability to decide whether to employ the same strategy in the next election knowing roughly how cost effective it was in the past election. Requirements The requirements to run an experiment are not onerous: 1) An identifiable subject population. For instance, Asian-American identified on the VAN or streets in neighborhoods with a high density of Asian-Americans are clearly defined. People at a street fair or festival are not a well-defined group. 2) A treatment that can be randomized. Mail, phone, and doors are very easy to manipulate. A national TV campaign or web broadcasts would be difficult to study. 3) The ability to measure the outcome variable of interest. Registration and turnout are public records that can be gathered. Vote choice would probably require a post-election survey, as would attitudes about the electoral process.

4) The willingness to extract a control group and abide by the agreed upon protocol. Sadly, it is this last requirement that trips up most organizations, and causes the loss of many opportunities to learn from past field work. We recognize that your desire to learn from experiments needs to be balanced by the goals of your program. We will work with you through the series of trade-offs and compromises that will inevitably come up. Experiments that are difficult to implement (politically or logistically) are often difficult for organizations to execute properly and thus lose a learning opportunity for both the organization and the evaluators. Thus, our overriding goal when designing experiments to test your field operations is to tailor the study to fit your organization as closely as possible. What does this mean in terms of how you would go about conducting an experiment with us? The process can be broken into a series of steps (note: the steps below assume that you are using the VAN and conducting phone or door-to-door canvassing): 1) Tell us what you plan to do, how your organization is structured, the logistical details behind your field operation, and any political constraints imposed by coalition members, donors, or past history (e.g., we have to hit 1000 doors in Precinct 5). This information will help us tailor an experiment to your need. You should also tell us what questions you would like answered in this election cycle and concerns you have about the experimental process, so we can be sure to address these issues up front. 2) We will work with you to come up with an evaluation plan that imposes as few burdens on you as possible. 3) You email a list of the places or people that you are targeting and will be subjects in the experiment. If you are using the VAN, you simply request an export of your target list. 4) We randomly assign the experimental subjects to the treatment and control groups and return the assignments to you. If you are using the VAN, we will return a file with the VANID, a flag for the treatment group, and perhaps a message assignment as well. 5) Upload the file onto the VAN. 6) When pulling your walk or phone lists, in addition to the targeting (e.g., precinct 5, Asian, under 50), select only the treatment individuals (you just check a box). 7) If there is a messaging component to the experiment, make sure the walk or call sheet you formatted includes the randomly assigned message flag (e.g., “E” for early voting and “D” for Election Day) in addition to name and address/phone. 8) Print the walk or call sheets. 9) Perform the walking and calling. 10) Be sure to enter in the contact information (e.g., not home, left message, contacted). 11) The day after Election Day, send us: a) any information on how you targeted neighborhoods; b) any information on what canvassers/callers said; c) copies of printed materials like flyers, mailers, or pamphlets; d) all the contact information; e) reports of any problems or colorful anecdotes.

12) Once vote history is released, we will check registration and turnout for both the treatment and control groups to perform the analysis. 13) We will write up the results and send you a draft. 14) Send us your edits. Everything excepting the results is fair game. 15) We will return an official white paper. You can distribute the white paper as you see fit. 16) You tell us whether you would like to be identified in our use of the experiments or remain anonymous. The list looks daunting, but each step is straightforward and many of them are standard actions for any voter mobilization campaign. The steps for mail can be even easier, since we can mail the names and addresses directly to the mail house on your behalf (or format it for a mail merge as you require). For projects where the VAN is not an option like voter registration (or groups without a subscription to VAN), we can format your call and walk sheets for you so that all you have to do is press “print” and hand the sheets out to your canvassers. We can tailor an experiment to fit your program and organization. Why are you performing this service? We are academics who need to publish articles in academic journals for career advancement. Our consulting services are very expensive, but we our services are free if you allow us to use the data for academic publishing. You can say whether your group is identified or anonymous in any publication. The nature of the review process and publishing cycle means that any academic work will likely be published no earlier than 2011. Furthermore, academic articles aren’t read by many people. So our academic writing won’t be good advertising for your group, but it will secure you top notch consulting services for free.