Art Of Hosting Research Report Cma

EVALUATION OF THE ART OF HOSTING TRAININGS Final Report July 2008

Prepared for: Columbus Medical Association Foundation Prepared by: Darryl Lang, Ph.D. Deborah A. Crawford

EVALUATION OF THE ART OF HOSTING TRAININGS Final Report Table of Contents Executive Summary……………………………………………………………. 3 Background and Purpose……………………………………………………… 6 Methods……………………………………………………………………….…. 6 Detailed Survey Results…………….………………………………………….. 7 Participant characteristics…………………………………………....... 7 Training Experiences: General………………………………………… 8 Training Experiences: Post-training Events………………………….. 9 Training Experiences: Satisfaction…………………………………….10 Training Experiences: Feelings After Training………………………..11 Training Experiences: Immediate Outcomes…………………………12 Training Applications: As A Participant…………………………….… 14 Training Applications: Professional Setting……………. …………… 16 Training Applications: Community Setting……………….. ………….18 Training Applications: Personal Setting……………………………… 20 Personal Change……………………………………………………….. 21 Future Needs……………………………………………………………. 22 ListServ Awareness…………………………………………………….. 24 Additional Comments…………………………………………………... 25 Additional Analyses…………………………………………………….. 25 Conclusions………………………………………………………………………26 Appendix • Survey verbatim open-ended responses • Art of Hosting community survey

Draft 7/1/08

2

Executive Summary The Art of Hosting is a training and practice for persons wanting to lead and engage groups and teams in interactive ways for the purpose of serving real needs in communities and to find new solutions. The focus of this evaluation project is to better understand the impact of the Art of Hosting training in Ohio and how it is practiced, especially focusing on identifying the extent of participant changes and application of what was learned.

Method Preliminary telephone interviews were conducted to collect qualitative information from a representative group of Art of Hosting training participants. Using these results, a written survey was developed and mailed to 149 training participants that had attended one of the Art of Hosting trainings between March 2005 and October 2007. The final response rate was 62% (83/133), a very good return rate for this type of survey.

Participant Characteristics The majority of survey participants work for not-for-profit or education organizations, are 45 years of age or older, female and represent a variety of Art of Hosting training experiences at various locations in Ohio. The main reason survey participants attended the training is to have the opportunity to learn something new and nearly one-half of the survey participants did not know much about the Art of Hosting before attending the training.

Training Experiences Sixty-nine percent (69%) of survey participants give the overall training experience high ratings (6 or 7 on a scale of 1 to 7). Participants reported that they appreciated learning new approaches to leadership and working with others. While this is positive, ratings and comments show that there were either features of the training that were not as effective as they could have been or that some survey participants do not feel comfortable with some aspects of the Art of Hosting model or its techniques. Below are some examples of verbatim comments: “The facilitators led by example to show effectiveness. During the training, I was more aware of the diverse opinions in our office around issues that I had never considered.” “I think I gained more from the training personally. I could not immediately see how it would apply professionally. It was somewhat unclear how the training would affect our organization, practices…..” The Art of Hosting training appears to be most effective in promoting communication, personal connection, listening and participation. Fifty-one percent (51%) of survey participants gave high ratings (6 or 7) regarding their knowledge increase as a result of the training. Sixty percent (60%) of survey participants said they were ready to apply what they learned. Thirty-five percent (35%) reported high ratings on the training improving their effectiveness as a leader or staff member. Below are some examples of verbatim comments: “The training was valuable. It helped to enhance my communication and facilitation skills. It provided an opportunity to challenge my current thinking and comfort level.”” Draft 7/1/08

3

“I use the World Café concept the most because it automatically results in all people becoming active and interested participants in the discussion.” “I thought the training teaches leaders a better way to communicate….and it also provides innovative and inclusive ways to gather collective input and move projects forward.”

Training Applications Many survey participants applied the Art of Hosting concepts and techniques --- a very positive finding. They applied what they learned in different settings: 72% in their work organization, 40% in a community setting and 32% in a personal setting. Seventeen percent (17%) applied what they learned in all three settings. Survey participants find nearly all the Art of Hosting techniques useful in professional, community and/or personal settings. The most popular are: World Café, Circle, Check In/Check Out, Talking Stick, Open Space and Powerful Questions. Survey participants’ experiences in leading or hosting have been moderately effective in reaching desired meeting, discussion or dialogue goals. Below are examples of some verbatim comments: “The meeting participants seem to be growing into the Art of Hosting process. It is a work in progress, with people at various comfort levels actively participating in this new process.” “At a board meeting, I asked the group to Check In by sharing what their desire was to be part of the group. It was incredibly meaningful to share with each other and it created a common bond.” “Asking Powerful Questions brings focus and intentionality to meetings. “There were lots of ideas generated and the participation was good. Now I ask myself “what happens next?” and “who’s responsible?” Accountability is a major issue.” “I have found the conversation-based approach to a problem is difficult for those having never been through the Art of Hosting training to grasp.”

Personal Change The training had a positive impact on the way survey participants listen to others and ask the “right” questions. Nearly all survey participants (85%) said the Art of Hosting training changed them in a variety of ways, including looking at situations and individuals with more awareness and a deeper understanding. Below are some examples of verbatim comments: “It helped me look and listen more deeply and to zero in on details, then step back to assess the broader picture. It helped me to realize we cannot assume we know what others are thinking.” “I have become a better listener. I am not so quick to offer my opinion. I wait to hear what has been said and only respond if necessary. Silence is okay.”

Draft 7/1/08

4

Future Needs Nearly all the survey participants think that if more organizations and individuals applied the Art of Hosting concepts and techniques, goals and outcomes would be more likely achieved. This shows that most participants believe the Art of Hosting is valuable and shows promise. Over one-half (55%) of the survey participants think that some type of follow-up activity or get-together would increase their use of the Art of Hosting concepts and techniques. .

Conclusions Survey participant feedback shows a desire for a new way of approaching groups, leadership and problem solving, but the complexity and unique approach of the Art of Hosting may require additional supports and reinforcements for applying the techniques in the real world. Survey participants feel they need more training on specific techniques and on how to apply and adapt the techniques in their particular setting in order to use them effectively. Behavior changes slowly, and behavioral change requires consistent feedback to reinforce messages. Many Art of Hosting survey participants expressed the desire and need for additional training and reinforcements to increase their comfort level and effectiveness at using the techniques. In order to assess the impact of the Art of Hosting training in the future, the training and its desired outcomes need to be more well-defined and consistent. The trainings examined through this evaluation were very different in several ways. These included how trainees were selected, the length and level of intensity of the training, the written materials provided, the components and featured techniques, and the types and mix of participants.

Draft 7/1/08

5

Background and Purpose The Art of Hosting training and practice focuses on leading by engaging through interactive ways of working with groups and teams and with a purpose to serve the real needs in communities and to find new solutions. As stated on the Art of Hosting Web site: The Art of Hosting is a pattern and a practice that allows us to meet our humanity in ourselves and in each other - as opposed to trying to be machines meeting. The Art of Hosting training is an experience for deepening competency and confidence in hosting group processes - circle, world café and open space and other forms. Each of these processes generates connection and releases wisdom within groups of people. They foster synergy and provide ways for people to participate in intention, design, and outcomes/decisions/actions. The experience is hosted by a team of facilitators who are skilled/trained in at least one, if not all of these processes; and the experience is aimed at people who want to serve as conversational hosts in their work, community, and personal lives.1 The focus of this evaluation project is to better understand the impact of the Art of Hosting training in Ohio and how it is practiced, especially focusing on identifying participant changes and the application of what was learned and the resulting outcomes in various settings: professionally, in the community and personally. Another purpose is to identify ways to improve and expand the learning process.

Methods Two data collection methods were selected for the research: telephone interviews and a mail survey. Telephone interviews were conducted to collect qualitative information from a representative group of Art of Hosting training participants. The interviews were designed to better understand training participants’ opinions of the Art of Hosting and their experiences in applying what they learned. This information was used to identify critical issues and develop the survey questions. Telephone interviews were conducted with individuals who attended one of the Art of Hosting trainings offered between March 2005 and October 2007. Art of Hosting leaders selected 22 Art of Hosting training participants to represent different training experiences and various levels of program interests. Out of the 22 participants, 15 were interviewed between March 28 and April 4, 2008. Interviews took between 40-50 minutes to complete. A six-page survey was developed based on the interview results and mailed on May 23, 2008 to 149 training participants. These were all the individuals who had attended one of the Art of Hosting trainings between March 2005 and October 2007. Training participants were asked to return their completed surveys to Saperstein Associates, Inc, an independent marketing research firm in Columbus, Ohio.

Response Rate Of the 149 training participants who were mailed surveys, 9 addresses were incorrect due to changes in employment and 7 said they had not attended any of trainings. Eighty-three (83) completed surveys were returned. This resulted in a response rate of 62% (83/133). This is a very good return rate for this type of survey.

1

www.artofhosting.org, Home page

Draft 7/1/08

6

Detailed Survey Results Participant Characteristics The majority of survey participants work for not-for-profit or education organizations and educational institutions. 100% 90% 80% 70%

62%

60% 50% 40% 30%

23%

20% 10%

1%

2%

Health Care

For-Profit

4%

8%

0% Self-Employed

Government

Education

Not-For-Profit

Q30: What best describes your current place of employment? Number of respondents = 83

Most survey participants are 45 years of age or older. Seventy-two percent (72%) are female. 100% 90% 80% 70%

58%

60% 50% 40%

24%

30% 20% 10%

12%

6%

0%

Under 30

30 to 44

45 to 60

Over 60

Q31: What is your age? Number of respondents = 83

Draft 7/1/08

7

Nearly one-half of the survey participants didn’t know much about the Art of Hosting before attending the training. 100% 90% 80% 70% 60% 50%

48%

40% 30%

22%

29%

20% 10%

1%

0%

Almost Nothing

A Little

Some

A Lot

Q2: Prior to attending your first Art of Hosting training how much did you know about the Art of Hosting concepts and techniques? Number of respondents = 83

Training Experiences: General Survey participants represent a variety of Art of Hosting training experiences at various locations in Ohio. It should be noted that the Art of Hosting trainings were not necessarily similar in terms of trainers, attendee characteristics, length and the emphasis on Art of Hosting concepts and techniques. Phil/Jewell: Do you think all the trainings were well represented. Were we short on the Hocking Hills group? Do you have dates for Raymond and Deer Creek. Keep in mind survey participants could have checked more than one location. Training Location Shawnee State Park (April 2007) Josephinium (October 2007) Darby Dan Farms (October 2007) CMA/CMAF/COTS/PFC at City Parks (August 2006 & 2007) Raymond Memorial (date) Deer Creek (date) Hocking Hills State Park (March 2005) Our Optimal Health Core Hosting Group (April/May 2006) Other locations (e.g., Shambhala Institute, John Glenn School, Boulder, Colorado, etc.)

Number of Respondents 28 23 19 14 14 6 8 6 5

Note: Survey participants could have attended more than one training.

Draft 7/1/08

8

The main reason survey participants attended the training is to have the opportunity to learn something new. Few said they attended because of an organizational requirement. Number of Percent Respondents 56 67% 41 49% 39 47% 38 46% 37 45% 28 34% 18 22% 5 6%

Reason for Attending Training Opportunity to learn something new Invited to attend by training organizer Promoted/encouraged by my organization Like the Art of Hosting concept Refresh my leadership/communication skills Opportunity to interact with community leaders Required by my organization Other (e.g., fan of Phil Cass, to apply concepts to personal/professional life, etc.) Note: Survey participants could have given more than one reason.

Sixty-seven percent (67%) attended the training with others from their organization. One survey question asked if survey participants had attended any other Art of Hosting trainings not listed on the survey. Sixteen (16) or 20% said “Yes.” The following are the trainings they attended: Additional Training Cherry Valley Pegasus Conference in Boston Our Optimal Health German Village and Columbus State Shambhala Institute in 2004 and 2006 Metro park in southeast Columbus Public health in Nova Scotia Kellogg’s food conference With my agency and CMA in May 2008 In Hilliard, Ohio Follow-up training in May 2008 CMA and AFC in May 2008 OSU at Deer Park The main reason for attending these additional trainings was to “deepen” their understanding of the Art of Hosting, learn more about the Art of Hosting and “get more comfortable” with the Art of Hosting concepts and techniques.

Training Experiences: Post-training Events As the following chart shows, over one-half of the survey participants have participated in at least one posttraining Art of Hosting event or get-together. One-third has participated in three or more. The main reasons they participate in post-training events or get-togethers are to reconnect and/or interact with other training participants and to share experiences. Draft 7/1/08

9

100% 90% 80% 70% 60%

45%

50%

34%

40% 30% 20%

10%

11%

1

2

10% 0%

None

3 or More

Q9: How many post-training Art of Hosting events or get-togethers have you attended? Number of respondents = 80

Training Experiences: Satisfaction Survey participants were asked to rate their Art of Hosting training experience overall on a scale of 1 to 7, where 1 is “Poor” and 7 is “Excellent.” Sixty-nine percent (69%) gave high (6 or 7) ratings. 100% 90% 80%

69%

70% 60% 50% 40%

27%

30%

33%

36%

6

7 Excellent

20% 10%

0%

1%

1%

2%

1 Poor

2

3

4

0%

5

Q5a: Rate your Art of Hosting experience overall Number of respondents = 83

Survey participants were asked to explain their rating. Seventy-two (72) wrote in comments. The reason for high ratings (6 or 7) focused on learning new approaches to leadership and working with others, seeing how the Art of Hosting concepts and techniques can be applied and relating to the other training survey participants. Here are some examples of the verbatim comments: “My first experience was very enlightening. I liked the whole concept of the Art of Hosting and I continue to practice it with positive results.” “The training was valuable. It helped to enhance my communication and facilitation skills.” Draft 7/1/08

10

“The facilitators led by example to show effectiveness. During the training, I was more aware of the diverse opinions in our office around issues that I had never considered.” “I learned and enjoyed the interactions with community leaders, learning new techniques, and the ability to practice them.” Those who gave lower ratings (less than a 6) tended to feel that the Art of Hosting concepts and techniques were too vague, too “out there” or didn’t see how the Art of Hosting concepts and techniques could be applied to their own situations. Below are some examples of verbatim comments: “I think I gained more from the training personally. I could not immediately see how it would apply professionally. It was somewhat unclear how the training would affect our organization, practices…..” “I think [the World Café] is a great way to encourage participation. However, the concepts were made to be too abstract and elusive...” “It was a good experience overall; however, it lacked the training aspect…I need more application.” All the verbatim comments are in the Appendix.

Training Experiences: Feelings After Training Survey participants were asked to describe how they felt after their training experience. Nearly all (89%) said they felt as though they had a better understanding of the Art of Hosting concepts and techniques. Interestingly, many survey participants (60%) said they were read to apply what they learned and nearly one-half felt that they had a better understanding of other survey participants’ opinions and ideas. Number of Respondents 72

Feelings After Training Percent Better understanding of the Art of Hosting 89% concepts and techniques Ready to apply what I learned 49 60% Better understanding of other training 39 48% participants’ opinions and ideas Wanted to learn more abut the Art of Hosting 39 47% A changed person 25 31% Not exactly sure how I would apply what I 19 23% learned Part of the community 10 7% Note: Survey participants could have given more than one description.

Draft 7/1/08

11

Training Experiences: Immediate Outcomes As the four charts below show, one-half of the survey participants increased their knowledge about Art of Hosting concepts and techniques a lot. Fewer said they developed relationships, improved their communications and interactions with others, and improved their effectiveness as a leader or staff member a lot. 100% 90%

Knowledge Increase

80% 70%

51%

60% 50%

39%

40%

28%

30%

23%

20% 10%

0%

0%

1

2

5%

5%

3

4

0%

5

6

7 A Lot

Very Little

Q10-1: As a result of the first Art of Hosting training you attended, how much did your knowledge about the Art of Hosting concepts and techniques increase? Number of respondents = 82

. 100% 90% 80%

Relationships Developed

70% 60%

25%

50% 40%

24%

30% 20% 10%

30%

15% 4%

2%

1

2

9%

16%

0%

Very Little

3

4

5

6

7 A Lot

Q10-2: As a result of the first Art of Hosting training you attended, how much did your relationships with other training survey participants develop? Number of respondents = 82

Draft 7/1/08

12

100% 90%

Communications and Interactions Improve

80% 70% 60% 50%

30%

37%

40%

23%

21%

30% 20% 10%

1%

4%

7%

1

2

3

7%

0%

4

5

6

7

Very Little

A Lot

Q10-3: As a result of the first Art of Hosting training you attended, how much did your communications and interactions with others improve? Number of respondents = 82

100% 90% 80% 70%

Effectiveness as a Leader or Staff Member Improve 35%

60%

41%

50% 40%

28%

30%

15%

20% 10%

2%

2%

5%

1

2

3

7%

0%

4

5

6

Very Little

7 A Lot

Q10-4: As a result of the first Art of Hosting training you attended, how much did your effectiveness as a leader or staff member improve? Number of respondents = 82

Draft 7/1/08

13

Forty-one percent (41%) of the training survey participants are “prepared” or ‘very prepared” to apply what they learned professionally after the Art of Hosting training. Only three survey participants (4%) said they weren’t prepared. Not surprisingly, those who rated their training experience higher, tended to feel they were more prepared to apply what they learned after the training than those who rated their experience lower. 100% 90% 80% 70%

41%

56%

60% 50%

32%

40% 30% 20% 10%

9%

4%

0%

Not Prepared

Somewhat Prepared

Prepared

Very Prepared

Q11: After the first Art of Hosting training, how prepared were you to apply what you learned professionally? Number of respondents = 82

Training Applications: As A Participant Eighty-four percent (84%) of the survey participants said they had experienced the Art of Hosting concepts and techniques as a participant in meetings, discussions, conferences and/or dialogues. They experienced many of the Art of Hosting concepts and techniques. World Café and Circle were the two techniques they experienced the most: World Café

81%

Circle

79%

Check In and/or Check Out

67%

Talking Stick

63%

Open Space

61%

Powerful Questions

59%

Harvesting

56%

Appreciative Inquiry

29%

Chaordic Path

19%

Divergence, Emergence, Convergence

13% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Q13: What Art of Hosting concepts and techniques did you participate in? Number of respondents = 70

Draft 7/1/08

14

Survey participants thought these Art of Hosting concepts and techniques were effective in increasing the quality of the meetings, discussions, conferences and/or dialogues they participated in. 100% 90% 80% 70%

50%

60% 50% 40%

33%

36%

30% 20% 10%

14%

12% 0%

1%

1

2

4%

0%

Not Effective

3

4

5

6

7

Very Effective

Q14a: In general, how effective were these Art of Hosting concepts and techniques in increasing the quality of the meetings, discussions, conferences or dialogues you participated in (didn’t’ lead or host)? Number of respondents = 70

Survey participants were asked to explain their rating, 52 of the survey participants wrote in comments. Most were positive, making general comments about how the Art of Hosting provides an opportunity for everyone to be heard and express themselves, creates an open environment to generate ideas and stimulate conversation, and results in more effective and productive discussions and meetings. Some comments mentioned how a specific Art of Hosting concept or technique was effective, especially World Café. Here are some examples of verbatim comments: “We started using techniques such as Talking Stick and Check In and / or Check Out and it really changed our meetings in a positive way. Before we were very negative and talked over each other. They are more productive and civil now.” “It is a good way to get lots of ideas, is a great collaborative technique, and creates lots of involvement. The time constraints and the inability to handle large groups were things that were not effective.” “I recently hosted a conversation in Spanish with 50 Hispanic immigrants. A space was created for listening and not controlling. The responded in an amazing way!” “It was helpful, but participants were not given enough information or instruction. Therefore, the World Café was not as powerful as it might have been.” “I use World Café the most because it automatically results in all people becoming active and interested participants in the discussion.” “I have found the techniques to consistently enhance the conversation. A specific example was a high school level policy making task force at the university using World Café. There were a lot of action plans they could follow through with and it is a great set-up for action planning through the coming year.”

Draft 7/1/08

15

Reasons for lower ratings (less than six) focused on needing more time to prepare to implement the Art of Hosting concepts and techniques, people in their organization not understanding the Art of Hosting or not participating in the activities, and the need for more training on specific techniques. Here are some examples of verbatim comments: “It would have been more effective if all the people understood more about the purpose of the techniques, which is to have dialogue that is democratic.” “The desire and effort to incorporate the Art of Hosting concepts and tools is present, but the level of skill, time, and energy to do so is more of a challenge. This is especially so for starting an Art of Hosting team for the agency.” All the verbatim comments are in the Appendix.

Training Applications: Professional Setting Many of the survey participants (72%) said they had the opportunity to apply, lead or host an Art of Hosting concept or technique in their work organization with staff or associates at meetings, discussions and/or dialogues. The survey participants applied, lead or hosted all the Art of Hosting techniques, with the most popular being Circle and World Café (see chart below). 73%

Circle World Café

65%

Check In and/or Check Out

58%

Powerful Questions

58%

Talking Stick

53%

Harvesting

52% 43%

Open Space Appreciative Inquiry

23% 13%

Chaordic Path Divergence, Emergence, Convergence

8% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Q16: What Art of Hosting concepts and techniques did you apply, lead or host in your work organization? Number of respondents = 60

Draft 7/1/08

16

The Art of Hosting concepts and techniques were moderately effective in reaching the goals or outcomes of the meetings, discussions and/or dialogues that participants hosted or led in a professional setting. 100% 90% 80% 70%

46%

60%

43%

50%

32%

40% 30% 20% 10%

0%

0%

2%

14%

9%

0%

1 2 Not Effective

3

4

5

6

7 Very Effective

Q17a: In general, how effective were these Art of Hosting concepts and techniques in reaching the goals or outcomes of the meetings, discussions and/or dialogues you lead or hosted in your organization with staff or associates? Number of respondents = 60

Survey participants were asked to explain their rating, 44 wrote in comments. Survey participants mentioned how a specific Art of Hosting technique was used successfully and resulted in productive outcomes and achieving goals, and how Art of Hosting encouraged participation and provided structure. Here are some examples of verbatim comments: “The techniques prompted participation by reluctant members in a non-threatening manner.” “The meeting participants seem to be growing into the Art of Hosting process. It is a work in progress, with people at various comfort levels actively participating in this new process.” “At a board meeting, I asked the group to Check In by sharing what their desire was to be part of the group. It was incredibly meaningful to share with each other and it created a common bond.” “Asking Powerful Questions brings focus and intentionality to meetings. “When I follow the Art of Hosting fully, it is much more successful. Sometimes relationship building is more important than anything else and can be achieved with the Art of Hosting.” The comments pertaining to lower ratings (less than six) were about co-workers not understanding the Art of Hosting and/or not participating; needing more help implementing a technique; and more time is needed to make progress in their organization; and the lack of follow-up and accountability. Here are some examples of verbatim comments: “There was not enough time for the process to unfold and occur. There was also a lack of people with knowledge of technology.”

Draft 7/1/08

17

“There were lots of ideas generated and the participation was good. Now I ask myself “what happens next?” and “who’s responsible?” Accountability is a major issue.” “I have found the conversation-based approach to a problem is difficult for those having never been through the Art of Hosting training to grasp.” All the verbatim comments are in the Appendix.

Training Applications: Community Setting Thirty-three (33) survey participants (40%) said they had the opportunity to apply, lead or host an Art of Hosting concept or technique in a community meeting, discussions, conference and/or dialogues. As the following chart shows, the survey participants applied, lead or hosted all the Art of Hosting techniques, with the most popular being Powerful Questions, Circle, World Café and Check In/Check Out. 70%

Powerful Questions Circle

64%

World Café

61%

Check In/Check Out

61% 55%

Open Space Harvesting

52% 36%

Talking Stick Appreciative Inquiry

27% 12%

Chaordic Path Divergence, Emergence, Convergence

9% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Q19: What Art of Hosting concepts and techniques did you apply, lead or host in a community setting? Number of respondents = 33

Draft 7/1/08

18

The Art of Hosting concepts and techniques were effective in a community setting, more so than in a professional setting (shown above). 100% 90% 80% 70%

68%

60% 50% 40%

29%

30%

32%

36%

20% 10%

0%

0%

1

2

3%

0%

0%

3

4

5

6

Not Effective

7

Very Effective

Q20a: In general, how effective were these Art of Hosting concepts and techniques in reaching the goals or outcomes of the meetings, discussions and/or dialogues you lead or hosted in a community setting? Number of respondents = 31

Survey participants were asked to explain their rating, 22 wrote in comments. Survey participants mentioned how the Art of Hosting encouraged participation, got people engaged and open to conversation, and enhanced their meetings and discussions to resolve issues and be more productive. Here are some examples of verbatim comments: “I have used the techniques to help lead budget allocation and priority setting at my church. It is a multimonth process and the techniques are very helpful.” “The Art of Hosting techniques give you the courage and the tools to address difficult, but necessary, questions. Even when there is initial resistance, the techniques promote forward movement in the discussion.” “I was able to apply these techniques with a community organization with which we are partnering. It helped us get over some tricky issues relating to territory.” “There was a group of university students who spent most of the day using World Café and Open Space. The evaluations were almost unanimously positive. The group felt connected with each other and had great, participatory conversation all day long.” The comments pertaining to lower ratings (five or less) were about the difficulty in going beyond generating ideas to follow-up and resolution. Here are some examples of verbatim comments: “The concepts generated a lot of ideas, got folks engaged, and created excitement; however, there was no effective and timely follow-up. I ask myself “what’s different?” “The process was effective in bringing forth good ideas, but could not go further.” Draft 7/1/08

19

“Sometimes we didn’t see goals achieved until after the sessions. Sometimes it takes time.” All the verbatim comments are in the Appendix.

Training Applications: Personal Setting Twenty-six (26) survey participants or 32% said they had the opportunity to apply, lead or host an Art of Hosting concept or technique in a personal setting, with family or friends. The survey participants applied, lead or hosted all but two of the Art of Hosting techniques, with the most popular being Powerful Questions and Circle. Interestingly, the technique of Powerful Questions was used more in community and personal settings than in the work organization.

Powerful Questions

69%

Circle

42%

Appreciative Inquiry

35%

Check In/Check Out

31%

Harvesting

19%

Chaordic Path

15%

Talking Stick

15%

Open Space

12%

Divergence, Emergence, Convergence

0%

World Café

0% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90% 100%

Q22: What Art of Hosting concepts and techniques did you apply, lead or host in a personal setting? Number of respondents = 26

Draft 7/1/08

20

These techniques were effective in helping survey participants reach the goals and outcomes in a personal setting. 100% 90% 80%

61%

70% 60% 50%

39%

40%

30%

30%

22%

20%

9%

10% 0%

0%

0%

0%

1

2

3

4

Not Effective

5

6

7

Very Effective

Q23a: In general, how effective were the Art of Hosting concepts and techniques in reaching the goals or outcomes of these personal settings? Number of respondents = 23

Survey participants were asked to explain their rating, 22 wrote in comments. Survey participants had successfully applied the Art of Hosting techniques and concepts with their family members resulting in more open communication, asking better questions and listening more. A few survey participants mentioned the challenges in applying the Art of Hosting concepts and techniques in a personal setting. Here are some examples of verbatim comments: “This has changed the way I try to have conversations with friends and family. Beginning with a question and not knowing the answer has given lots of space for conversation and ease.” “The techniques are very powerful in getting others to drill down to the real questions.” “I have been more confident in asking wicked questions within my family. The resulting communication has been more open and meaningful.” “It is tougher in a personal setting versus a business setting.” All the verbatim comments are in the Appendix. Interestingly, some survey participants lead, hosted or applied the Art of Hosting concepts and techniques at work, in the community and personally.

Personal Change Nearly all the survey participants (85%) said the Art of Hosting training affected them in a way that changed how they interact with others, feel about themselves and others, approach situations or issues or look at the world around them. They were asked to specify what changed, 63 responded. The top ways survey participants changed are: Draft 7/1/08

21

! ! ! ! ! !

Listen much more to what others are saying (and don’t react without thinking) Recognize the importance of questions and formulate them better Look at situations and people differently (e.g., go deeper, greater awareness) Are more comfortable and/or open with others Have a better understanding of others opinions and ideas Look at the world differently since the training experience

Here are some examples of what the survey participants wrote: “My conversation technique changed and I enhanced my listening skills.” “It is difficult to explain. I have less patience for conversations that aren’t meaningful. It sounds bad, but it has provided me a lot of clarity. I am more willing to ask questions and try new ways of being and doing even if the outcomes are unknown.” “I have become a better listener. I am not so quick to offer my opinion. I wait to hear what has been said and only respond if necessary. Silence is okay.” ‘I cannot put this into words. I think that it is a spiritual process for me. I feel closer to my indigenous self. I appreciate silence and meditation more.” “I am more mindful of capturing and holding up other people’s input and reactions to topics at hand.” “It somewhat changed me. It makes me think about the questions I ask in discussions with others. I try to be more open in my responses.” “It helped me look and listen more deeply and to zero in on details, then step back to assess the broader picture. It helped me to realize we cannot assume we know what others are thinking.” “I try to stay in the present and enrich mindfulness. I ask questions not to just look for an answer, but to listen.” “I renewed my hope that our society may be able to work together to solve tough problems.” All the verbatim comments are in the Appendix.

Future Needs Nearly all the survey participants (97%) think that if more organizations and individuals applied the Art of Hosting concepts and techniques, goals and outcomes would be more likely achieved. Survey participants were asked what it would take for them to apply, lead or host the Art of Hosting concepts and techniques more, if at all. As the following chart shows, more than one-half think more exposure to situations where it would be appropriate to apply the Art of Hosting would increase their use of the Art of Hosting concepts and techniques. In other words, survey participants would like to better understand how the Art of Hosting concepts and techniques have been successfully applied, so they can be more successful at applying them to their own situations.

Draft 7/1/08

22

What It Would Take Number Percent More exposure to situations where it would 45 56% be appropriate to apply the Art of Hosting More training on specific techniques 39 49% More training to help me apply the 38 47% techniques to my situation If others in my organization were trained 27 34% Someone to help or supports me 22 28% More support from my manager, my 11 14% organization Note: Survey participants could have attended more than one selection. Over one-half of the survey participants think some type of follow-up activity or get-together after the Art of Hosting training would increase the use of the Art of Hosting concepts and techniques. 100% 90% 80% 70%

55%

60% 50%

34%

40% 30% 20% 10%

9%

2%

0%

Don't Know

No

Maybe

Yes

Q27a: Do you think some type of follow-up activity or get-together after Art of Hosting training would increase the use of the Art of Hosting concepts and techniques? Number of respondents = 83

Survey participants were asked what type of follow-up activities they thought would be best, 51 wrote in suggestions. The two most frequently mentioned were to have more trainings and learning opportunities and to have more meetings, get-togethers and/or discussions. Survey participants also want more support from individuals experienced in the Art of Hosting, more access to information and communications on the Internet and more follow-up with other training survey participants. Here are some examples of what the survey participants wrote: “I think it would be helpful to have more training and opportunities to learn from those who are experienced.” “Practice, practice, practice! It helps build skills and courage.” “Have a day-long training devoted to specific tools. It can cover how to develop Powerful Questions, conducting World Café, Harvesting, and what to do as follow-up.” Draft 7/1/08

23

“Have special interactive “learn-ins” on a topic-by-topic basis.” “Anything via the Internet or email would be best. Face-to-face participation may be limited for me.” “I suggest getting together for training to apply the concept in different settings, and then have a debriefing afterward.” “I would like numerous opportunities to meet and electronic ways to reach out at intervening times.” “Have refresher courses discussing best practices and where the Art of Hosting is being used and how. There can also be newsletter or e-mail communication.” “There should be follow-up gatherings to practice, brainstorm, and share.” “I need support from others who are trained and have an opportunity to share experiences.” All the verbatim comments are in the Appendix. Survey participants were presented with four possible follow-up training activities. As the chart below shows, of the four activities, a two-hour refresher training with Art of Hosting trainers is the activity survey participants would most likely attend.

Two-hour Refresher Training with AOH Trainers

57%

Quarterly Two-hour Gatherings with Other Training Participants

45%

32%

Use of Web Technology

Monthly One-hour Conference Calls With Trainer or Host

14%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Q28: Likeliness of attending? (Percentages of “Very Likely” responses) Number of respondents = 78

ListServ Awareness Two-thirds (67%) of the survey participants are aware of the Art of Hosting ListServ.

Draft 7/1/08

24

Additional Comments Survey participants were asked to provide any additional input, thoughts, recommendations or comments. Twenty-five (25) responded. Most of the comments were positive, thanking the Art of Hosting sponsors and saying how appreciative they are for the opportunity to participate in the trainings. Here are some examples: “Thank you for your continuing support of this work in the community. Thank you for the encouragement to have this kind of listening and healthy training.” “Thank you for hosting the original trainings and for asking me how to support and further the cause.” “Thanks to CMA for providing the opportunities.” “I would like to convey my thanks and appreciation for the opportunity to engage in the Art of Hosting. It is a most respectful and honoring form of meeting and engaging in meaningful dialogue.” “It was wonderful to have a block of time set aside to participate in the Art of Hosting training. Thanks to the planners and participants for all their work, and being willing to take risks and be open to a new way of thinking and doing. Thank you Phil and Diane.” Four survey participants wrote in suggestions: “If there is a downside to the Art of Hosting in organizations it would have to do with getting work done and making decisions in a timely manners. Not all organizations or business processes are conducive to the Art of Hosting. It works best when dealing with the development of mission purpose statements and strategic planning.” “There might be a perception in the community that the Art of Hosting is more of a spiritual retreat in the new age or Buddhist tradition. This will appeal to some, but raise red flags for others.” “The Art of Hosting is very helpful for initial gatherings, but once the question and direction of the group is established the World Café style is not so productive.” “I appreciated the training. Thank you! I had a couple of thoughts. Make activities and concepts less elusive and more concrete. Explain the values and principles upon which the Art of Hosting is based, such as democratic decision-making and working collectively. Explain the benefits, such as enhanced sustainability.” All the verbatim comments are in the Appendix.

Additional Analyses To better understand why some survey participants rated their Art of Hosting training experience somewhat lower than others, a cross-tabulation analysis was conducted. It should be noted that dividing the survey data into various categories could result in numbers too small to detect any significant relationships. For this analysis possible trends or relationships are noted only when a statistically significant difference was found. Draft 7/1/08

25

Attended with Others in Their Organization The one relationship worth noting has to do with whether a survey participant attended the Art of Hosting training with others from their organization or not. When a survey participants attended the Art of Hosting training with others from their organization they were somewhat less prepared to apply what they learned professionally and tended not to lead, host, or apply what they learned in their work organization, the community or personally than those who didn’t attend the Art of Hosting training with others from their work organization. However, those who attended the training with others from their work organization experienced as much of a personal change from the Art of Hosting training as others. Also, attending training with others from their work organization or not didn’t effect participants’ perceptions of training outcomes, such as increasing their knowledge about Art of Hosting concepts and techniques, developing relationships, improving their communications and interactions with others, and improving their effectiveness as a leader or staff member. Participated in Post-training Events or Get-togethers Another relationship worth noting has to do with those survey participants who participated in post-training events or get-togethers versus those who did not. Those who made the effort to continue to be involved in the Art of Hosting after their initial training, reported the training increased their knowledge about the Art of Hosting more, tended to develop more relationships with other training participants and think their communications and interactions with others improved more as a result of the training, compared to those who didn’t participate in follow-up events or get-togethers. Interestingly, there was no difference in how much survey participants’ effectiveness as a leader or staff member improved. In addition, those who participated in post-training events or get-togethers did not feel more prepared to apply what they learned after the training than those who didn’t participate in post-training events or gettogethers. However, they did tend to lead or host Art of Hosting concepts and techniques in their work organizations more and think the techniques were more effective, compared to those who didn’t participate in post-training events or get-togethers.

Conclusions The survey participants represent a variety of Art of Hosting training experiences and organizational backgrounds. Survey participants have learned about the Art of Hosting concepts and techniques and have applied them in professional, community and/or personal settings. The Art of Hosting training appears to be most effective in promoting communication, personal connection, listening and participation. Survey participants’ experiences in leading or hosting using Art of Hosting concepts and techniques has been moderately effective in reaching the desired meeting, discussion or dialogue goals. The training had a positive impact on the way some participants listen to others and ask the “right” questions. Participants have also encountered situations where applying Art of Hosting concepts and techniques had its challenges. These appear to relate to a variety of issues including the following:

Draft 7/1/08

26

• Lacking clarity on how the model takes the conversations and ideas to the next step – action,

decision and accountability • Using the techniques with people who do not like participating • Needing help to effectively apply Art of Hosting concepts and techniques in their own specific situations

To better understand the results, evaluators assessed them using research on training impact. Based on the well known work of Donald Kirkpatrick2, assessing the impact of training is often conducted on four levels: 1. 2. 3. 4.

Satisfaction with training Acquisition of new knowledge or skills Application of what was learned Longer term impact on the organization

These evaluation results show that the Art of Hosting training has had generally positive results on the first three levels. 1. Satisfaction – For the most part, survey participants gave the training moderately high ratings. Participants reported that they appreciated learning new approaches to leadership and working with others. The evaluators would have expected higher ratings based on their experience with this rating scale. Somewhat lower ratings could be explained by some participants being unclear how to apply the concept and techniques to their professional situations. 2. Acquisition of new knowledge or skills – Half of the survey participants gave high ratings regarding their knowledge increase as a result of the training and over half said they were ready to apply what they learned. Less than half of survey participants reported the training improved their communications and interactions and their effectiveness as a leader or staff member. While this is positive, the evaluators would have expected higher ratings based on their experience with this rating scale. 3. Application of what was learned – Despite some survey participants saying their knowledge of the Art of Hosting concepts and techniques did not increase greatly due to their training experience, participants’ applications of Art of Hosting concepts and techniques appear to be at fairly high levels. Survey participants report actually applying what they learned in different settings: 72% in their work organization, 40% in a community setting and 32% in a personal setting. Seventeen percent (17%) applied the Art of Hosting concepts and techniques in all three settings. 4. Longer term impact - It was not within the scope of this evaluation to determine the longer term impact of the Art of Hosting training and use of techniques in organizations and the community. In regards to longer term impact, the World Bank Institute (WBI), which has conducted thousands of training programs intended to increase teamwork and leadership worldwide, reports that evaluation research on the effectiveness of corporate training is very limited and that that only 8% of organizations carry out results or impact evaluations on training. 3 2

Donald Kirkpatrick, Evaluating Training Programs: The Four Levels, 1998

3

“Features of Staff Learning that Contribute to Effectiveness in the World Bank”, Marlaine E. Lockheed and Diana J. Arango, WBI Evaluation Studies, No. EG05-107, The World Bank Institute, The World Bank, Washington, D.C., April 2005

Draft 7/1/08

27

In spite of the lack of training impact information for comparative purposes, these findings lend themselves to the following additional conclusions. • Nearly all the participants think that if more organizations and individuals applied the Art of Hosting

concepts and techniques, goals and outcomes would be more likely achieved. This shows that most participants believe the Art of Hosting is valuable and has promise as an effective leadership tool.

• Many participants emphasized the positive impact of the Art of Hosting training on their listening and

communication styles. They mentioned that rather than react quickly to what others say, they listen intently to others and this has improved their professional and personal interactions.

• Participants reported that the Art of Hosting techniques were more effective in the community and

personal settings as opposed to the professional setting. More exploration needs to occur around what factors could have an impact on the Art of Hosting concepts and techniques being more effective in one setting as opposed to another.

• One possible reason for the lower ratings on some of the survey questions could be that

participants feel they need more training on specific techniques and more training on how to apply and adapt the techniques in their particular setting or organization in order to use them effectively.

• Participant feedback shows a desire for a new way of approaching groups, leadership and problem

solving, but the complexity and unique approach of the Art of Hosting may require additional supports and reinforcements for applying the techniques in the real world.

• Behavior changes slowly, and behavioral change requires consistent feedback to reinforce

messages. Many Art of Hosting training participants expressed the desire and need for additional training and reinforcements to increase their comfort level and effectiveness in using the techniques.

• In order to better assess the impact of the Art of Hosting training in the future, the training and its

desired outcomes need to be more well-defined and consistent. The trainings examined through this evaluation were very different in several ways: how trainees were selected, the length and level of intensity of the training, the written materials provided, the components and featured techniques, the types and mix of participants and the ways participants were invited to attend.

Draft 7/1/08

28