Systems Appraiser Guide

Systems Appraiser Guide Valuable Information for Systems Appraisers

Academic Quality Improvement Program The Higher Learning Commission

Contents The Systems Appraisal Process .............................................................................................. 3 Systems Appraisal Glossary ............................................................................................ 3 Identifying Accreditation Issues in the Systems Appraisal ................................................ 5 Dealing with Systems Portfolio Gaps ............................................................................... 7 Where evidence an institution is meeting the Commission’s common Criteria for Accreditation is likely to appear in a Systems Portfolio ................ 9 Category Summary Statements ......................................................................................10 Systems Appraisal Worksheets ..............................................................................................10 Instructions for Completing the Systems Appraisal Worksheets ......................................10 Individual Worksheets 1 – 9............................................................................................12 Chart of the flow among worksheets 1 – 9 ......................................................................14 An MS Word version of the Worksheet Template is available for download at www.AQIP.org. Select the Downloads link and look under the AQIP Peer Reviewers category. Steps in the Systems Appraisal Process................................................................................15 Systems Appraisal 1: Independent Review .....................................................................15 Systems Appraisal 2: Consensus Review .......................................................................16 Systems Appraisal 3: Consensus Conversation ..............................................................17 Writing Feedback in Appraising a Systems Portfolio ............................................................19 Content Guidelines.........................................................................................................19 Independent Worksheets ....................................................................................21 Effective Comments for Results (R) Questions....................................................22 Consensus Worksheets ......................................................................................23 Strategic and Accreditation Issues Analysis (Draft, Proposed, and Feedback) .....23 Writing “Feedback Ready” Comments.................................................................24 Writing “Actionable” Comments...........................................................................25 Style Guidelines .............................................................................................................25

Grammatical Considerations ..........................................................................................26 Frequently Asked Questions about Systems Appraisals.......................................................28

What’s New in the Systems Appraiser Guide: • • • • •

Using the institution's Self-Evaluation in preparing feedback, page 17. New content guidelines, page 19. New content in Strategic and Accreditation Issues, page 24. The current Appraiser Schedule has been removed from the Guide and is now available in the AQIP Peer Reviewers section of Downloads on the AQIP Website. The contact information for the current Systems Appraisal Process manager can be found in the Staff Contacts section on the AQIP Website.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 2

Systems Appraiser Guide

The Systems Appraisal Process The AQIP Systems Appraisal is a consistent, cost-effective process designed to provide an AQIP organization with professional feedback representing the consensus view of a team of higher educators and others experienced in continuous quality improvement and systems thinking. The process begins when an organization informs AQIP its Systems Portfolio is ready for review, and ends 12 weeks later with the delivery of a 35-45 page Systems Appraisal Feedback Report that includes three components: a Critical Characteristics Analysis, which shows the organization how the team understood its distinctive mission, context, and goals; Category Feedback on each of the nine Categories, identifying what the team sees as the organization’s strengths and opportunities for improvement; and a Strategic and Accreditation Issues Analysis, in which the team identifies what it views as the highest strategic priorities for the organization’s future. In addition, the team will provide the organization with a potentially publishable two-three page Appraisal Summary that captures the team’s appraisal of the organization’s developmental maturity on each of the nine Categories. The goal of the Systems Appraisal Feedback Report and Appraisal Summary is to provide the organization with actionable information it can use to improve its processes and performance. AQIP expects organizations participating in the Program to maintain accredited status to demonstrate overall improvement from one Systems Appraisal to the next. AQIP bases its reaccredidation decisions every seven years on an organization’s overall pattern of improvement (including information from Systems Appraisals, regular reports on Action Projects, and other data), and thus leaves it up to the organization to decide whether, when, and how to address the opportunities for improvement and Strategic and Accreditation issues identified in a Systems Appraisal. In the rare case where there is strong evidence that an organization is no longer committed to continuous improvement or in a position to benefit from AQIP participation, the Systems Appraisal team can alert AQIP. When such a case occurs, AQIP will work with the organization to gather information that will allow the AQIP Standards and Admissions Panel to recommend appropriate action — perhaps even the return of the organization to the more traditional PEAQ accreditation process. The appraisal team may also nominate a specific organizational process or system for AQIP Outstanding Practices recognition, directing AQIP staff to gather additional information about the nominated process. If selected for recognition, the organization will be given a formal opportunity to share its success with the rest of the higher education community.

Systems Appraisal Glossary Category Summary Statement. A statement, crafted by the team (beginning as each team member completes each Independent Category Worksheet) that expresses the team’s overall formative evaluation of the institution’s progress and developmental maturity to date on the Category, and, in the team’s judgment, the institution would be best advised to make the focus of its attention as it continues its journey to continuous quality improvement. These statements should be positive, stressing what the institution has accomplished and explain the benefits of where it needs to focus next. Statements should be written so that the group of nine Category

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08 3

Systems Appraiser Guide

Summary Statements, together with the Strategic and Accreditation Issues Analysis, can be released to the public as an Executive Summary of the Systems Appraisal. Consensus Category Analysis. The final, revised version of the feedback on each Category produced as a result of the Consensus Conversation. These Consensus Category Analyses are based on the Consensus Category Drafts prepared before the telephone discussion, and reflects the suggestions and changes of the team that occurred during that conversation. In other words, each Consensus Category Draft, plus any changes from the Consensus Conversation, will be summarized/combined for the organization into one Consensus Category Summary, which will include a Category Summary Statement expressing in a few sentences the Appraisal Team’s overall formative evaluation of the Category. Consensus Category Draft. A worksheet completed by one team member, summarizing the Independent Worksheets provided by all team members on one particular Category. The Consensus Category Drafts for all nine Categories are the basis for the Consensus Conversation. Consensus Conversation. The conference call held by the team to reach consensus on the Category Feedback and the Category Summary Statement for each Category, the Critical Characteristics Analysis and the Strategic and Accreditation Issues Analysis that will be provided to the organization in the team’s System Appraisal Feedback Report. Consensus Review. The stage in the Systems Appraisal when team members (other than the team leader) are assigned to summarize the Independent Worksheets for one or two of the Categories in order to produce Consensus Worksheets - one Consensus Worksheet per Category. Critical Characteristics Analysis. Analysis produced by the team leader with any modifications made after the Orientation Conversation. (The Analysis may be discussed and modified during the Consensus Conversation.) This Analysis is part of the final Systems Appraisal Feedback Report. Critical Characteristics Draft. Summary consensus proposal prepared from the Critical Characteristics Worksheets produced by the team leader. Produced and distributed to team members in preparation for the Orientation Conversation. Critical Characteristics Worksheet. Worksheet produced by each member of the team and submitted to the team leader, after studying the Organizational Overview and skimming the Systems Portfolio, that attempts to identify the key characteristics that make the organization being appraised unique and distinct. Independent Category Worksheets. Worksheets every team member produces on each of the nine Categories during the Independent Review stage of Systems Appraisal. Independent Review. The stage in the Systems Appraisal when each team member is independently producing first a Critical Characteristics Worksheet - as a review of the Organizational Overview - and second, a set of Independent Worksheets as a review for each of the nine Categories. Organizational Overview. A five-page introduction to the Systems Portfolio in which the organization answers eight direct questions and describes the key factors and capabilities that underlie its strategies for success. This is also the means for the organization to explain its distinctive mission and context to the Systems Appraisal team.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 4

Systems Appraiser Guide

Orientation Conversation. A conference call that occurs after the team leader has produced the Critical Characteristics Summary. This call can identify questions the team needs answered in order to complete its Appraisal. Strategic and Accreditation Issues Analysis. Analysis produced by the team leader with any modifications made after the Consensus Conversation. This Analysis is part of the final Systems Appraisal Feedback Report. Strategic and Accreditation Issues Draft. Summary of the worksheets produced by each member of the team during the Independent Review stage, produced by the team leader as a consensus proposal for the Consensus Conversation. Strategic and Accreditation Issues Worksheet. Worksheet produced by each member of the team and submitted to the team leader, in conjunction with the Independent Worksheets. The worksheet attempts to identify the potential crosscutting or priority issues for the future of the organization being appraised, and identifies any issues related to the organization’s compliance with the Commission’s Criteria for Accreditation. Systems Appraisal Feedback Report. The final report sent to the organization. It includes a Critical Characteristics Analysis, Category Feedback, the Appraisal Summary and a Strategic and Accreditation Issues Analysis. Systems Appraisal Team Leader. The leader of the Appraisal team who is responsible for coordinating the Appraisal team and ensuring that the reviews are completed by the deadlines. The leader may be responsible for collecting any additional information from the organization that the team needs in order to conduct its reviews. And the leader may also be asked to deliver/present the Systems Appraisal Feedback Report to the organization. Systems Appraisal Team. Typically a group of six (including the team leader) AQIP Peer Reviewers who are assigned to complete an Independent and Consensus Review of an AQIP organization’s Systems Portfolio. The team will produce a Systems Appraisal Feedback Report for an organization. Systems Portfolio. The description, created by the organization, of its key processes, their performance effectiveness, and the way the organization improves its processes and performance. The Systems Portfolio is written in response to the questions under each of the nine Categories. The organization also prepares and submits an index to the Systems Portfolio showing how the Portfolio presents evidence for each of the Commission’s Criteria for Accreditation.

Identifying Accreditation Issues in the Systems Appraisal Like all institutions accredited by the Higher Learning Commission, colleges and universities participating in AQIP must demonstrate that they meet the Commission’s Criteria for Accreditation. Every seven years, AQIP determines that a participating institution meets these five Criteria through a formal process called Reaffirmation of Accreditation. This determination is based on a pattern of evidence provided by the institution during the previous seven years of its participation in AQIP. A major source of evidence for the Reaffirmation of Accreditation judgment comes from the comprehensive review of institutional systems and activities that constitute the Systems Appraisal. Not only do institutions create a Systems Portfolio that describes the institution’s current processes and performance results, but every AQIP institution also provides the ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 5

Systems Appraiser Guide

Systems Appraisal team with an Index to the Criteria for Accreditation correlating the evidence in its Systems Portfolio with the Commission’s Criteria for Accreditation and their Core Components. The purpose of this Index is to make it easier for the Appraisal team to determine whether the institution has provided sufficient and persuasive evidence that it is meeting the Criteria for Accreditation. During their Independent Review of the Systems Portfolio, team members should identify potential accreditation issues for the institution to address. As they complete their nine Independent Category Worksheets (Worksheet #4), they should be on the alert for any instances in which an institution’s description of its systems or results signals the institution’s failure to meet an accreditation expectation. Although such instances should be rare, if one occurs the Independent Reviewer should note it in drafting Worksheet #5, Strategic and Accreditation Issues, highlighting the issue with an A (for accreditation) and identifying the specific Criterion for Accreditation that evidence indicates is not met. The team leader will receive these worksheets from each team member and prepare from them a single Strategic and Accreditation Issues Summary (Worksheet #7) which will serve as a “consensus proposal” for the team’s discussion during the Consensus Conversation. If the entire team agrees that an issue identified validly threatens the institution’s accreditability, then that issue should appear in the final feedback report on the Strategic and Accreditation Issues Analysis (Worksheet #9). During its Consensus Conversation regarding accreditation issues, the team will also review the institution’s Index to make certain the institution has presented, in its Systems Portfolio, sufficient and persuasive evidence that it meets each of the Criteria for Accreditation. All team members should review this Index prior to the Consensus Conversation, and note any areas where they believe the evidence is insufficient to show that the institution meets a Criterion. If the team’s consensus judgment is that there are significant gaps in the pattern of evidence presented, then the team leader should be directed to add these gaps to Worksheet #9 as accreditation issues (A), specifying which Criteria are in question, and why the evidence is insufficient. An institution will have ample opportunity to address and correct any accreditation issues identified by a Systems Appraisal prior to its scheduled Reaffirmation of Accreditation. AQIP will conduct a Checkup Visit to each institution two or three years prior to its Reaffirmation, and the institution will have opportunity to provide evidence that issues that concerned the Appraisal team are no longer relevant, either because the institution has provided evidence to fill gaps identified by the team, or because the institution has taken action to correct problems identified by the team. If accreditation issues remain “on the table” after the completion of the Checkup Visit, the institution will have additional opportunities to augment and correct the evidence that it meets the Criteria for Accreditation when its Reaffirmation of Accreditation takes place. Thus the System Appraisal team’s signaling an accreditation issue, through serious, does not imperil an institution’s accreditation because the institution has sufficient opportunity to explain to AQIP why an issue is spurious (by providing the missing evidence) or how a valid issue has been addressed so that it should no longer cause concern — before the Reaffirmation of Accreditation occurs. However Systems Appraisers should use this mechanism judiciously, signaling only accreditation issues (A) that rise to the level, which would threaten the institution’s ability to remain accredited. Other important crosscutting issues for the institution can be identified as strategic issues (S or SS) while issues of more limited scope will be identified as opportunities for improvement (O or OO) in the review of specific Categories.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 6

Systems Appraiser Guide

Dealing with Systems Portfolio Gaps An institution must tackle every P, R, and I item in the AQIP Categories AQIP expects participating higher education organizations to examine themselves on all of the items under AQIP’s nine Categories. Any college or university committed to reaching its full institutional potential must ask itself every one of these questions, seriously and regularly. But not every institution is equally mature in its quality thinking, and the different circumstances in which institutions find themselves force development in some areas before others. An institution may still be behaving reactively in some processes while it develops proactive approaches in others. The Systems Portfolio provides an institution a means of formally stating answers to the questions in each item. The institution should summarize its current level of maturity in the C, or context, items that begin each Category. Its response to the C items should explain to the Systems Appraisal team where it believes it currently stands in respect to the P, R, and I items in that Category. Appraisers should confirm or contest this institutional self-understanding in their analysis of the Category. Examining itself on every item will help an institution discover the gaps that exist between what the institution does now and what it could do to excel. Realistically, an institution won’t be able to write a substantive response for every item and may give a brief “No reportable results” response for some. An institution might not yet be able to describe stable, systematic approaches for some processes, or it might discover that it has no performance result data for certain processes. Many institutions discover that systematic improvement cycles appear only after they have stabilized their processes, and collected and analyzed data on process performance. To ensure it gets useful feedback on the areas where it is currently working to improve, an institution must write substantive responses to no fewer than one-third of the total P, R, and I items within each of the nine Categories. (Institutions may write substantive responses for all items, but must stay below the overall 100-page limit.)

In its Systems Portfolio, an institution addresses at length a subset of P, R, and I items that it selects AQIP encourages an institution to focus its Systems Portfolios on those P, R, or I items where it has something substantive to say about its processes, results, or improvement cycles. For example, if it is developing and deploying strong and effective processes in a particular area, the institution will describe them under the relevant P items. Similarly, it will report performance results under the relevant R items, and actual cycles of improvement under the I items. If there is nothing to report, the institution should say so, clearly and concisely, ideally in a short comment like “No reportable information is currently available about how we set our learning goals for undergraduates. We recognize stabilizing the design of this process represents an major opportunity for educational improvement.” In turn, the Systems Appraisal team may acknowledge that the institution is aware of a gap in this area, and consequently that the item represents an opportunity for improvement. An extensive comment in feedback may not be required in such cases.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 7

Systems Appraiser Guide

In its feedback report, the team should designate a “no reportable…” item with an O symbol (indicating it represents an “opportunity for improvement”) and merely state “No reportable information from the institution at this time.” If the team believes that the institution’s lack of attention (or progress) in an area is critical, it should promote the O to an OO, and concisely explain why continuing to ignore this item would be perilous for the institution. Using the shorthand response of “No reportable…” allows an institution to focus the 75-100 pages of its Systems Portfolio on those areas where it can most benefit from feedback. By not wasting space in describing things that do not now exist the team’s feedback becomes more focused, and more useful. Responding to an item with “No reportable…” does not exempt an institution from confronting the issues the item covers. Every institution striving for systematic quality must deal with all of the process, results, and improvement issues raised in the AQIP Categories. But there is a natural developmental sequence to where an institution puts its efforts. Just as a child can’t learn to run before it learns to walk, an institution can measure the effectiveness of a process it hasn’t formulated, nor can it improve a process whose current performance is unknown. Thus it makes no sense for an institution to write long responses to items it has not yet seriously addressed. At some point in the future, after the institution’s processes become more mature, it will have to develop and use measures of those processes and implement systems for continuous improvement. Some examples An institution examines itself on item 3P5 and concludes that it currently has no systematic or dependable processes for deciding whether it should serve a new student or stakeholder population. For item 3P5 in its Systems Portfolio, it states “No reportable processes in this area.” The team concurs by assigning 3P5 an O, and stating, “No systematic processes for identifying new students or stakeholders are currently in place.” An institution has described good processes for leadership and communication, but examines itself on item 5R1 and concludes that it has no dependable measures or indicators of these processes’ effectiveness. It simply states “No reportable results in this area” for item 5R1 in its Portfolio. The team decides that this is a major gap, assigns the symbol OO to it, and explains in its comment why it is imperative that the institution design measures of leadership and communication effectiveness and gather hard data on these processes. An institution that has not yet developed effective processes for planning is unlikely to have data on the performance of these processes; consequently, when it writes its responses to items in Category 8, it decides to invest its Portfolio space in describing the current state of its processes, thereby inviting feedback from the Appraisal team on that focus. In addition to these correspondences, pay attention to the fact that: •

AQIP Category Context for Analysis (C) and Processes (P) responses often contain evidence that relate directly to the Criteria’s Core Components.

•

AQIP Results (R) responses may provide very strong evidence for any of the five Criteria for Accreditation

•

AQIP Improvement (I) responses may provide excellent evidence for Criterion Two and Criterion Three in particular. ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 8

Systems Appraiser Guide

1. Helping Students Learn

x

2. Accomplishing Other Distinctive Objectives

x

3. Understanding Students' and Other Stakeholders’ Needs

x

x

x

4. Valuing People 5. Leading and Communicating

x

x x

x

x

x

x

x

x

6. Supporting Institutional Operations

x

7. Measuring Effectiveness

x

8. Planning Continuous Improvement

x

9. Building Collaborative Relationships

x

x

Criterion Five: Engagement and Service.

Criterion Three: Student Learning and Effective Teaching. Criterion Four: Acquisition, Discovery, and Application of

AQIP Categories

Criterion Two: Preparing for the Future.

Commission’s Criteria for Accreditation

Criterion One: Mission and Integrity.

Where evidence is likely to appear in a Systems Portfolio that an institution is meeting the Commission’s common Criteria for Accreditation

x

x

x

x

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 9

Systems Appraiser Guide

Category Summary Statements Earlier, AQIP used “rubrics” to communicate to institutions their level of maturity on each of the nine Categories. Because these rubrics were standardized, some institutions interpreted them as grades. AQIP has replaced the rubrics with Category Summary Statements, in which teams craft a separate statement for each of the nine Categories. Each Statement should be a short paragraph consisting of 2-5 sentences that praise the institution for the progress it has made in continuous improvement, including: o o o o o o o

identifying processes that need to be designed identifying ways to improve the design of existing processes deploying robust processes broadly clarifying the goals of existing processes specifying the performance measures used to evaluate how well processes are achieving their goals analyzing the measures gathered from the operation of processes using measured data to improve processes and performance

Be obnoxiously positive — the goal is to help the institution develop its quality culture by letting it build upon past successes.

Systems Appraisal Worksheets Instructions for Completing the AQIP Systems Appraisal Worksheets AQIP compiles all of these worksheets into the Systems Appraisal Feedback Report, and emails each team member a copy. The team leader and team members should not send anything directly to the institution.

AQIP System Appraisal feedback reports are formatted using two different paragraph styles. These two styles are built into the following worksheet templates that AQIP provides to Systems Appraisers. The styles used are identified as CC and SAC. Either of these indicators will appear in the formatting toolbar (which you can turn on using Word’s view…toolbars command. The material on this instructions page, in other portions of the worksheets (such as titles), and in the boilerplate sections of the feedback report appear in normal or headings styles. You cannot make changes or additions to this page.

AQIP uses an unvarying CC style for the Critical Characteristics that Systems Appraisers identify, and also for Strategic and Accreditation Issues Worksheets (#5, 7, and 9). Each Critical Characteristic begins with the letter O, followed by a single digit that refers to one of the eight questions each institution uses to organize its Overview, followed by a lowercase letter (a, b, c, etc.), without intervening spaces). The lowercase letters help to distinguish different Critical Characteristics that the team identified for the same questions. ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 10

Systems Appraiser Guide

The items in Strategic and Accreditation Issues Worksheets begin with a capital SI or AI, a tab, and then the comment. The paragraph format for style uses Arial 11 point type, has a tab set at 0.5 inch from the left margin, a hanging indent of 0.5 inches, 6 points of leading before and after each paragraph, and line spacing set to 1.5 lines.

O3b

This is a sample of a critical characteristic identified by the Systems Appraisal team. The letter indicates it is the second comment that the team identified as responding to Overview question 3. You will use this format for Worksheets 1 – 3, and when reprinting relevant Critical Characteristics on Worksheets 4, 6, and 8.

AI

This is a sample of an accreditation issue as it might appear on worksheets 5, 7, or 9.

AQIP uses an unvarying SAC style for System Appraiser comments on each Category, the comments appraisers identified as strengths (SS or S) or opportunities (OO or O). Each comment begins with the Category item number (a single digit, then a letter — C, P, R, or I — and then a one or two digit number, without intervening spaces). This item identifier is followed by a tab, the rating — SS, S, O, or OO — another tab, and then the comment, written without tabs, carriage returns, or extra spaces between words or sentences. The paragraph format for style SAC that is employed for Appraiser comments uses Arial 11 point type, has two tabs set at 1 and 1.5 inches from the left margin, a hanging indent of 1.5 inches, 6 points of leading before and after each paragraph, and line spacing set to 1.5 lines.

1P2

OO

This is a sample of a comment on a single Category item. Most appraiser comments will look like this one. Style SAC is used to format System Appraiser comments like these.

1P3-1P6

SS

This is a sample of a comment on a range of adjacent Category items that the institution has addressed with a single section of its Portfolio.

1P7,1R4

O

This is a sample of a comment on two separate Category items that the institution has addressed in a single section of its. This occurs rarely.

If in doubt (or if you mistakenly change the format) use the format painter (the little paintbrush icon) to copy the format from the appropriate sample item above: highlight the sample item, click the paintbrush, and paint it on all of the items you have created for which that style is appropriate.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 11

Systems Appraiser Guide

Individual Worksheets 1 – 9 1: Critical Characteristics Worksheet

Who prepares or completes it? Every team member (except the team leader). When is it prepared? As soon as AQIP sends you the Systems Portfolio. Then what happens? Team members email their completed Worksheets #1 to the team leader. 2: Critical Characteristics Draft

Who? The team leader prepares this consensus proposal. When? As soon as all team members email the team leader their completed Worksheets #1. Then? The team leader emails all team members Worksheet #2 so that they can prepare for the Orientation Conversation, reaching agreement on the institution’s critical characteristics. 3: Critical Characteristics Analysis

Who? The team leader incorporates the entire team’s agreements into this analysis. When? After the Orientation Conversation, prior to the Independent Review of the Systems Portfolio. Then? The team leader emails all team members Worksheet #3 to guide their Independent Review of the nine Categories as they prepare Worksheet #4. 4: Independent Category Worksheet

Who? Every team member (including the team leader). When? After team leader distributes Worksheet #3, which marks the beginning of the Independent Review stage of the Systems Appraisal. Then? Each team member emails nine Worksheets #4 (one for each Category) to the team leader. (Save the file for each worksheet with a name in this format: Crit2_ABC.doc or Crit7_ABC.doc, where “ABC” represents the initials of the team member who prepared the worksheet.) 5: Draft Strategic and Accreditation Worksheet

Who? Every team member drafts an independent list of these issues. When? During the Independent Review stage, concurrently with the preparation of Worksheets #4, team members prepare Worksheet #5, indicating strategic issues with an S and accreditation issues with an A. (By definition, all issues that threaten an institution’s accredited status are also strategic for that institution.) Accreditation issues can include those where the evidence is missing that the institution meets one or more accreditation criteria and those where the evidence indicates that the institution fails to meet an accreditation expectation. Comments must refer specifically to one or more of the Commission’s Criteria for Accreditation (and the relevant Core Components of each) when identifying accreditation issues.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 12

Systems Appraiser Guide

Then? Each team member emails Worksheet #5 to the team leader. (Save the file in this format: Wksht5_ABC.doc, where “ABC” represents the initials of the team member who prepared the worksheet.) 6: Consensus Category Draft

Who? Team members each prepare Worksheet #6 for one or two Categories, as assigned by the team leader. When? After the completion of the Independent Review stage, when all team members have sent their Worksheets #4 to the team leader, the team leader will make consensus assignments and distribute the appropriate Worksheets #4 to each team member. Then? After completing one or two Consensus Category Proposals (#6), email them to all team members, who will study them in preparation for the Consensus Conversation. 7: Consensus Strategic and Accreditation Draft

Who? The team leader prepares this consensus proposal from the Worksheets #5 prepared by each team member. When? Prior to the Consensus Conversation. Then? The team leader emails all team members Worksheet #3 to guide their Independent Review of the nine Categories using Worksheet #4. 8: Consensus Category Analysis

Who? The same team member who prepared Worksheet #4 is responsible for the corresponding Worksheet #8. When? The final version of Worksheet #8 is prepared following the Consensus Conversation; it incorporates what the team decided in that discussion. Then? The team member emails the completed Worksheet #8 to the team leader, who then forwards it to AQIP. 9: Strategic and Accreditation Analysis

Who? The team leader completes Worksheet #9. When? After the Consensus Conversation, which discussed Worksheet #7 (which the team leader had prepared from the Worksheets #5 prepared by each team member), the team’s decisions are incorporated into Worksheet #9. Then? The team leader forwards the completed version of Worksheet #9 to AQIP.

Chart showing flow among the nine worksheets The chart on the next page shows how the worksheets combine to make up the process for Systems Appraisal.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 13

Orientation Conversation

Systems Appraiser Guide

Each team member creates a Critical Characteristics Worksheet

➙

❷

Team leader prepares a Critical Characteristics Draft

➙

❹

Each team member creates nine Independent Category Worksheets

➙

❻

Team members combine all work into nine Consensus Category Drafts

➙

❺

Each team member independently creates a Strategic and Accreditation Issues Worksheet

➙

❼

Team leader prepares a Strategic and Accreditation Issues Draft

➙

❸

Team agrees on a consensus Critical Characteristics Analysis

❽

Team agrees on a final Consensus Category Analysis for each Category

❾

Team agrees on a final consensus Strategic and Accreditation Issues Analysis

Consensus Conversation

Consensus Conversation

❶

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08 14

Systems Appraiser Guide

Steps in the Systems Appraisal Process Systems Appraisal 1: Independent Review (Estimated Total Time: 13 - 20 Hours) 1. Quickly skim the entire Systems Portfolio. 2. Carefully read the Organizational Overview (5 pages) and the organization’s responses to the Context (C) questions for each of the nine Categories. In these sections, the organization provides foundational information about its distinctive features, scope, students, faculty, staff, facilities, technologies, environment, competitors, opportunities, and vulnerabilities. Draft a 1-2 page Critical Characteristics Worksheet in response to this reading in which each sentence should identify a single factor or characteristic that you believe it is important for the Appraisal team to keep in mind as its members review the Systems Portfolio. The goal here is twofold: (1) to make sure the Appraisal team, as it appraises each Category, bears in mind a shared understanding of the most important aspects of the organization’s identify, its current dynamics and the forces surrounding it, and its internal momentum and aspirations; and (2) to make sure the organization recognizes that the team understands and knows what makes the organization distinctive when it receives the Systems Appraisal Feedback Report. Number each sentence with the appropriate Organizational Overview identifier (i.e., if you write two statements in response to what the organization says in item OO5, number them OO5a and OO5b). Keep your statements in order, and skip a line (one carriage return) between each statement. 3. Receive back, by email, a consensus Critical Characteristics Draft from your team leader, and read it carefully. (In the interim, analyze the complete Systems Portfolio.) Participate in a one-hour telephone Orientation Conversation, in which the team members agree on (or agree to modify) the Critical Characteristics Analysis. If crucial questions emerge from this conversation, questions whose answers are needed before the Independent Reviews can proceed, the team leader will contact the institution and then communicate the answers to all team members. The leader will then make any modifications to the Draft and email each team member the Critical Characteristics Analysis. 4. Complete an Independent Category Worksheet for each Category. The Organization’s Systems Portfolio is divided into Category sections, which should contain the answers to each of the questions within that Category. Each answer should be identified with the section/number of its corresponding question in the AQIP Principles and Categories for Improving Academic Quality booklet. The organization’s Systems Portfolio probably will not repeat the question so keep the Categories handy for reference. To complete an Independent Category Worksheet for each Category section in the Portfolio, •

choose a Category section in the Portfolio to focus on, read the answers in that section as a whole first.

•

review the Critical Characteristics Analysis and indicate, on the worksheet, the 3 - 4 factors that you believe are most important in relation to that Category as a whole.

•

examine each answer individually and write a series of bullets that respond to a single answer or set of answers that the organization has grouped together to respond to the corresponding question(s) found in the corresponding AQIP Category section of the booklet.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 15

Systems Appraiser Guide

•

assign each bullet a symbol to identify whether it represents one of the organization’s strengths in meeting the requirements of the Category (SS or S) or one of the organization’s opportunities for improvement of its systems and processes that fall under this Category (OO or O). Discriminate issues carefully, and decide whether each is a strength or an opportunity for improvement. Do not write bullets treating a single issue as both strength and an opportunity for improvement.

•

choose one Category Summary Statement (from a formal, graded array of seven Category Summary Statements) that best captures your perception of the organization’s current overall level of achievement on that Category. Considering everything you have learned and discussed about the organization, complete a Strategic and Accreditation Issues Worksheet, identifying up to three potential crosscutting or priority issues for the future of the institution. Email your nine completed Independent Category Worksheets and your Strategic and Accreditation Issues Worksheet to the team leader.

Systems Appraisal 2: Steps in the Consensus Review (Estimated total time: 4 - 6 hours) 1. Your team leader will have assigned you either a single Category (Category 1, “Helping Students Learn”) or two Categories (any two of Categories 2 - 9) for consensus, and have emailed you the appropriate Independent Category Worksheets completed by the five other members of your Appraisal Team. Your task is to combine the contents of the Independent Worksheets corresponding to an assigned Category into a Consensus Category Worksheet that will be your consensus proposal during the team’s Consensus Conversation. You will complete one or two Consensus Category Drafts depending on how many you are assigned. 2. To complete a consensus worksheet, first review the Critical Characteristics Analysis section on each Individual Worksheet, and indicate on the worksheet the 3 -5 items that reflect the consensus of the team as to which Critical Characteristics are most relevant to evaluating that Category. 3. Read the bullets section on each Independent Worksheet and look for commonalities among them. Craft a comment (1-2 sentences) for each strength and for each opportunity for improvement, that all team members could support. Propose a symbol (SS, S, O, OO) for each comment. Write your proposed comments and corresponding symbols on the Consensus Worksheet. Your comments should summarize the shared views of all team members concerning the strengths and opportunities for improvement in this Category. To do this, sort all of the bullets on the six worksheets first by the Category questions, and then (if there are multiple bullets for a single question) by theme. Comments suggested by only one or two team members should be captured on the worksheet, but list them at the bottom and do not give them a symbol. A good strategy is to select, from the assembled worksheets, the best comment for a particular theme, and use it as the basis for your consensus comment, adding or modifying it if necessary. ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 16

Systems Appraiser Guide

Discriminate issues carefully, and decide whether each issue overall is a strength or an opportunity for improvement. Do not propose a comment that treats the same issue as both a strength and an opportunity for improvement. 4. Record on the Consensus Category Draft (s) the independent reviewers who echoed each consensus comment you have proposed by filling in the symbol (SS, S, O, OO) each reviewer gave that comment. Put an “X” in the column for any reviewer who had no comment similar to the one you are proposing. In the last row of the worksheet, fill in the Category Summary Statement numbers suggested by each independent reviewer and then fill in your proposed Category Summary Statement number (in the left column). 5. Email your one or two completed Consensus Category Drafts to the team leader who will make sure all members of the team receive copies prior to the Consensus Conversation during which the team will come to agreement on the feedback it will provide to the institution. The team leader will combine the individual Strategic and Accreditation Issues Worksheets into a Strategic and Accreditation Issues Draft, and all members of the team will also receive copies of this document prior to the Consensus Conversation. After receiving and studying all Consensus Category Drafts and the Strategic and Accreditation Issues Draft, you will participate in a Consensus Conversation (probably a teleconference) in which each Category is discussed, in turn, with the discussion led by whichever team member crafted the Consensus Category Draft under discussion. Following this conversation (estimated time: four hours), each team member will incorporate suggested changes in each Category consensus proposal and send all revised work to the team leader.

Systems Appraisal 3: Steps in the Consensus Conversation (Estimated total time: 4 - 6 hours) 1. After receiving and studying the Consensus Category Drafts from the other members of your team, you will participate in a Consensus Conversation in which each Category is discussed, in turn, with the discussion led by whichever team member crafted the Consensus Category Draft under discussion. After discussing each Category, the team leader will lead a discussion of the Strategic and Accreditation Issues Draft. Your team leader will arrange and communicate to you the time scheduled for the conversation (or conversations, if the conversation must be scheduled in two installments). 2. Typically, the team will go through the nine Categories in a prearranged order suggested by the team leader, and the team member who prepared the Consensus Category Draft will lead the discussion on that Category, running quickly through: •

each comment that was crafted to capture the consensus view of the team, asking for suggestions and objections from the rest of the team concerning each proposed consensus comment and symbol (SS, S, O, OO)

•

the set of comments rejected as not capturing the team’s consensus view, asking team members who feel passionately that their comment should not be rejected to argue for their inclusion.

Throughout these discussions, the goal is for team consensus — unanimous agreement or agreement of all but one team member. With all team members well prepared in advance, ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 17

Systems Appraiser Guide

the discussion of each Category should take about 20 minutes, with twice that amount of time, perhaps, for Category 1, “Helping Students Learn.”

Using the institution's Self-Evaluation in preparing feedback At this point in the process, in preparation for the Consensus Conversation, the Team Leader also distributes to each team member copies of the institution’s Self-Evaluation of the responses it has provided in its Systems Portfolio. (A sample of this two-page form is pictured below). AQIP asks each institution to rate (using SS, S, O, OO — or “?” if it is unsure) how it currently views its activity and performance for each P, R, and I item to which it has provided an in-depth response — indicating whether it perceives this areas is an institutional strength or opportunity for improvement. (Download Self-Evaluating Your Systems Portfolio for AQIP’s instructions to institutions.) The Self-Evaluation’s goal is to stimulate each institution to decide where it thinks its current processes and performance are good, and where it thinks its improvement opportunities lie. The institution’s Self-Evaluation should help your team give better and more useful feedback to the institution by showing where team and institutional appraisals differ. For example, if your team agrees with the institution that its personnel evaluation process is an outstanding strength (OO), you may need to write only a short sentence or two explaining why you agree. But if the institution thinks its system for new program development is a strength (SS or S) but your team sees it as an opportunity for improvement O or OO), it is the team’s responsibility to explain clearly why it sees the system as an opportunity that it is in the institution’s interest to improve. Differences between the institution’s self-evaluation and your team’s appraisals alert your team to explain, clearly and specifically, how and why it reached a different judgment than the institution did. This is true whether your team’s appraisal is more or less favorable than the institution’s. At the end of the discussion for each Category, the team must also agree on a Category Summary Statement that captures the team’s view of the organization’s maturity for that Category. The team member who prepared the Consensus Category Draft (having considered the independent ratings of all team members) suggests a Category Summary Statement number, and others can argue for a higher or lower number until the team reaches an agreement. 3. After all Categories have been discussed, the team leader facilitates a discussion of the Strategic and Accreditation Issues Draft that has been distributed to all team members, and the team reaches consensus on the contents of a Strategic and Accreditation Issues ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 18

Systems Appraiser Guide

Analysis that will be part of the final Systems Appraisal Feedback Report. The team should strive to identify no more than 3-4 strategic issues that are crucial for the future of the organization. The team also considers whether it should make any modifications in the Critical Characteristics Analysis that will provide the organization assurance that the team’s feedback is based on a clear understanding of its distinctive organizational identify and dynamics. 4. Following the conversation, each team member incorporates suggested changes in the comments for their assigned Categories, and sends a finished Consensus Category Analysis to the team leader. The leader assembles these together with the Critical Characteristics Analysis and Strategic and Accreditation Issues Analysis into the final Systems Appraisal Feedback Report, and sends each team member and AQIP an electronic copy.

Systems Appraisal Timetable A timetable for the current years’ Systems Appraisals can be found in the AQIP Peer Reviewers section of Downloads on the AQIP Website at www.AQIP.org.

Writing Feedback in Appraising an AQIP Systems Portfolio Content Guidelines Focus primarily on content in writing comments, on giving useful feedback to the organization whose Systems Portfolio you are appraising. But how you say things can enhance — or undercut — the credibility and value of the feedback you give. Here are some pointers about content and style to guide your comment writing. •

Don’t write comments for Context items. These are included to provide the minimum context the team needs to interpret the P, R, and I items. If the C items for a category are inaccurate, incomplete, or unhelpful, provide constructive feedback in your Category Summary Statement (e.g., “The Context items for Category One need to spell out the specific current educational goals the institution has for all its students, even if these are undergoing constant revision.” or “The student groups the institutions now serves and wishes to serve are not clear in the Context for Category Three, making it difficult to see whether the processes in place can assess stakeholder groups’ needs.)

•

Draw linkages between the organizations responses to Categories questions, and between a Category question response and the organization's Organizational Overview.

•

Appraise whatever the organization has provided, giving honest and frank feedback. If the organization has not responded in depth to a P, R, or I item, briefly identify this as a future opportunity for improvement (O or OO), explaining to the organization why addressing it now is urgent or whether it could be scheduled for some future improvement cycle. Organizations should address in depth at least 1/3 of the total P, R, and I items in each Category, and should signal where they want detailed feedback by the items to which they devote the most attention. If you think the organization’s focus is wrong, or that they’ve given a Category too little attention, say so in your Category Summary Statement.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 19

Systems Appraiser Guide

•

Give short responses to items where the organization’s response is short. AQIP encourages an organization to focus its Systems Portfolios on those P, R, or I items where it has something substantive to say about its processes, results, or improvement cycles. For example, if it is developing and deploying strong and effective processes in a particular area, the organization will describe them under the relevant P items. Similarly, it will report performance results under the relevant R items, and actual cycles of improvement under the I items.

•

If there is nothing to report, AQIP urges organizations to say so, clearly and concisely, ideally in short comments like “No reportable information is currently available about how we set our learning goals for undergraduates. We recognize stabilizing the design of this process represents an major opportunity for educational improvement.” In turn, the Systems Appraisal team may acknowledge that the organization is aware of a gap in this area, and consequently that the item represents an opportunity for improvement. Extensive feedback comments may not be needed in such cases.

•

In its feedback report, the team should designate a “no reportable…” item with an O symbol (indicating it represents an “opportunity for improvement”) and merely state “No reportable information from the institution at this time.” If the team believes that the organization’s lack of attention (or progress) in an area is critical, it should promote the O to an OO, and concisely explain why continuing to ignore this item would be perilous for the organization.

•

Give the organization feedback on performance measurement in every Category. If an organization provides data for Result items, comment on whether its performance is “good enough” and suggest how it can set stretch targets for future performance. If an organization provides no in-depth responses to any of a Category’s R items, then it must provide an in-depth response to the final P item, the one that asks what measures it uses or plans to use to measure its performance in that Category. If this response is weak, suggest how it might articulate the goals and objectives of various processes in that Category, and possible measures of the performance of key processes.

•

Don’t copy groups of sentences or paragraphs directly from the Systems Portfolio into your report. The organization already has a copy of its Portfolio — it wrote it! Quoting short phrases from the Portfolio, using organizational terminology, or citing tables and illustrations by number all provide the organization with assurance that your team has read their Portfolio carefully. Colleges and universities want to know your reaction and counsel. But quoting at length what they’ve written back to them provides little value. Yet be like a mirror — assure them you see and understand who they really are.

•

Remember that O’s and OO’s are not faults, but Opportunities for improvement (or Outstanding Opportunities). Often, an organization has not yet dealt with an area because it hasn’t yet reached the level of development where that area is on its radar. For example, organizations that haven’t yet measured their own performance in, say, advising, aren’t at fault because they don’t compare their performance with others’ — they simply haven’t yet reached the point where such comparisons would be possible — or useful to them. Telling them that such situations represent opportunities for improvement is not negative, but rather a way of coaching the organization to reach the higher levels of excellence that it wants. Getting actionable feedback on how to improve is a major reason why organizations join AQIP. (OO is pronounced “Oh….OOOH!” with rising intonation, the way someone who finally understands with wonder the

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 20

Systems Appraiser Guide

opportunities ahead — not “uh-oh,” which is what you might say to someone who made and error. For shorthand, call OO’s “outstanding opps.”)

•

Do not contradict other comments found elsewhere in the feedback. Contradictions are most likely to occur when an appraiser does not clearly specify the strength or opportunity.

•

Avoid prescriptions that simply say, in effect, “do this because we say to”; instead, tell them why following your advice makes good sense. Don’t be afraid to suggest — even to prescribe or proscribe — things an organization should consider doing (or stopping), so long as you explain why doing (or refraining from) something would benefit (or hurt) the organization. An organization is free to accept or reject your counsel, but will find it difficult to reject a shrewd directive that includes an explanation of why obeying it is in the organization’s self-interest.

•

Avoid subjective judgmental terms (good, terrible, repulsive, interesting) that serve primarily to describe your internal reaction to what the institution describes. Instead, state your observation in a factual manner (e.g., customer satisfaction rates have increased over the past three years) or communicate the reason for your positive or negative reaction (e.g., continuing declines in enrollment are likely to produce serious budget shortfalls).

•

Do not refer to individuals by name in your comments or feedback, and be hesitant to identify positions (VPAA) held by individuals as the subject for personal praise or blame.

Critical Characteristics Analysis (Draft, Proposed, and Feedback) •

Limit your draft worksheet to 1-2 pages.

•

Use phrases rather than complete sentences.

•

Begin each comment with the number of the item (O1 – O8) to which it refers followed by a tab, then the comment itself:

O4

The institution clearly identifies its common learning goals for baccalaureate graduates, thereby aligning all faculty members’ efforts in helping students learn essential skills and content.

Independent Category Worksheets Expect every member of the appraisal team to do an Independent Review of the Critical Characteristics and all nine of the AQIP Categories in a Systems Portfolio. Every team member, including the team leader, must review every Category. Although it might seem quicker and less burdensome to have each Category reviewed by only a subset of the team, doing so would seriously weaken the quality of feedback the organization receives. It would limit the number and quality of ideas the team generates in identifying strengths and opportunities for improvement, and make it difficult for some team members to participate in the consensus conversation (on Categories they did not independently analyze), thus making the feedback report something other than the consensus of the team’s views. •

Complete one worksheet for each Category.

•

Limit each worksheet to 1-3 pages.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 21

Systems Appraiser Guide

•

Do not write comments responding to C (or Context) questions, which are included so that you can place the institution’s response to other questions in the appropriate institutional context

•

Use a single, simple, complete thought per comment.

•

Identify each comment you write as a SS (outstanding strength), S (strength), O (improvement opportunity), or OO (outstanding improvement opportunity).

•

Judge things as you see them: don’t try to balance the number of strengths and improvement opportunities.

•

Begin each comment with the number of the item (e.g., 3P2) followed by a tab, then an identifier (SS, S, O, or OO), another tab, and then the comment itself:

3P2

•

SS

The institution clearly identifies its common learning goals for baccalaureate graduates, thereby aligning all faculty members’ efforts in helping students learn essential skills and content.

Sort your comments into numerical order (within P, R, and I batches), and leave space between comments.

The easiest way to do this is to follow the built in styles within the Systems Appraisal Template. CC for Critical Characteristics comments and SAC for Systems Appraiser comments. Effective Comments for Results (R) Questions. Well-written Results comments frequently address the following questions: •

Has the organization analyzed trends? Is the trend direction positive or negative? What is the desirable direction (i.e., more or fewer)? Are explanations provided for significant positive or negative changes?

•

In each Category are data presented for measures of the key processes that were described in the Process questions? How do the results from one Category link to other Categories (e.g., strategic processes, supplier and partner relationships)?

•

Are all important results presented? Are data focused on the critical organization performance results (e.g., customer requirements, compliance with regulatory requirements)? Are there any gaps in the data?

•

Is the amount of data provided sufficient (e.g., number of cycles of data for trend data, percentage of stakeholder population)?

•

Are the data appropriately segmented?

•

Do the data represent both short- and long-term priorities?

•

How does the organization measure effectiveness, and are these measures presented?

•

Are comparative data presented, and are they appropriate?

•

What are the standard measures in this field? Is there any significance to the lack of any of these measures in the Portfolio?

•

Are the data normalized (presented in a way that takes into account the various size factors)? ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 22

Systems Appraiser Guide

Consensus Category Worksheets •

Create a “consensus proposal” for each Category assigned to you by suggesting which comments belong in your team’s feedback report, and providing proposed wording for each comment.

•

Begin your work on each Category by sorting the comments from a set of Independent Category Worksheets into groups according to the questions on which they comment.

•

If multiple comments deal with a common issue, select the most thoughtful one and use it as a foundation upon which to build. Add ideas and words from similar comments to this base.

•

If several (3 or more) members of your team commented on the same issue, it probably belongs in the consensus proposal. If only one person commented on an issue, include it In consensus only if the comment has clear value and will benefit the institution.

•

Even if only one or two team members suggest comments on an issue, include the comment on your Consensus Category Worksheet. But if you propose that the team’s final feedback omit this issue, recommend no symbol (SS, S, O, OO) for the comment. Your worksheet “consensus proposal” in effect suggests that the team endorse only those items for which you include symbols.

•

In addition to following the advice for the Draft Category Worksheet (above), work to craft consensus comments that are feedback ready (see below).

Strategic and Accreditation Issues Analysis (Draft, Proposed, and Feedback) •

Provide 3 – 5 Strategic Issues for every organization. Strategic issues are not dings or faults — they constitute advice on where the team believes an organization would be wise to invest its energies in the immediate future to continue (and accelerate) its quality journey. These items should not repeat single SS, S, O, or OO comments, but rather should summarize a few key themes that run through the items and Category analyses you have done.

•

A Strategic Issue might be complimentary — a suggestion that capitalizing on a set of strengths (rather than addressing a set of opportunities) could move the institution in beneficial ways. But observations like this typically occur in Category Comments, not as Strategic Issues.

•

Fit Strategic Issues to the organization current place and culture. Strategic issues are typically broad, and can generate a variety of actions (action projects, initiatives, plans, tasks, or undertakings). If we expect an organization to concentrate its efforts on a few broad strategic issues, addressing them through a variety of projects, then it makes sense to limit the number of issues we identify as its highest priorities. Three or four is thus probably a sensible number. The rare organization that is highly practiced at continuous improvement, with a culture that can generate and attack new projects easily and complete them quickly, may be able to deal with more than three or four strategic issues.

•

Identify any accreditation issue that needs attention by specifically referring to one of the Commission’s five Criteria for Accreditation, and to the relevant Core Component of that Criterion. Accreditation-related issues should be serious, and imperil the institution’s

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 23

Systems Appraiser Guide

accreditation if left unaddressed. Do not associate a strategic issue with accreditation simply to get the institution’s attention. Most colleges and universities are using AQIP to maintain their accreditation, so staying accredited is of the utmost importance. Any opportunity or weakness that might endanger accreditation automatically becomes a major strategic issue. Moreover, accreditation issues are almost invariably critical issues in their own right — issues about clarity of mission, educational purpose and effectiveness, sufficiency and effective allocation of resources, and similar core concerns. An appraisal team that perceives a central issue of this sort needs to place it high on the organization’s agenda. The team should explain how the issue might endanger the organization’s accredited status, and why the substance of the issue matters — how ignoring it would handicap the organization’s ability to accomplish its mission, serve its constituents, and, perhaps, maintain its vitality and integrity. In your feedback, use letters to identify strategic and accreditation issues. Put A before any issue that jeopardizes accreditation (even if it is also of strategic importance), and tie it to a specific Criterion and Core Component. •

An accreditation-related issue can be raised either (a) because there is evidence in the Systems Portfolio that the institution is on a course that may result in its not meeting a Criterion, or (b) because the evidence that the institution meets every Core Component of all five Criteria is missing or insufficient. In the latter case, alerting the institution to gaps in evidence provides it with an opportunity to submit additional evidence before its Reaffirmation of Accreditation.

•

Have the team leader talk with AQIP’s Systems Appraisal Process Manager — before you submit your final Appraisal — if your team identifies a serious accreditation concern that is a “sin of commission” (rather than an instance where the organization has simply not yet provided enough evidence in its Portfolio to substantiate that it meets every Core Component). The current Systems Appraisal Process Manager’s name and contact information is listed on the AQIP website under Staff Contacts (www.AQIP.org/Staff).

•

Label strategic and accreditation issues distinctively (e.g. S1, S2, S3, A1, A2, etc.) so the institution can distinguish which is which.

•

Limit the worksheet to 2-3 pages.

Writing Feedback Ready Comments Comments that can easily be integrated with others make the team leader’s job easier, and make the feedback report more valuable to the organization. Feedback ready comments: •

are in complete sentences, frequently containing a subject and a verb “lifted” from the Categories question. For example, o

5P5 asks, “How does communication occur between and among institutional levels?” and a comment might respond, “Communication between administration and faculty occurs when…”

o

6P1 asks, “How do you identify the support service needs of your students?” and a comment might respond, “Each department identifies its students’ support service needs through a monthly…”

[Italics are used in these examples only to emphasize the subject and verb lifted from the question.] ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 24

Systems Appraiser Guide

•

mention specific examples from the organization’s Systems Portfolio

•

cite diagram, table, or figure numbers from the organization’s Systems Portfolio

•

link to key factors that appear in the Organizational Overview section of the Portfolio (and are likely noted in the team’s Critical Characteristics Summary)

•

follow content and style guidelines

•

include “So what’s”

•

consist of 1-3 complete, actionable, nonprescriptive sentences

•

cite specific examples from the Portfolio

•

link to the organization's Critical Characteristics as captured in your team’s Critical Characteristics Summary

•

mention, as appropriate, AQIP’s Principles of High Performance Organizations (e.g., Learning, Collaboration, Agility, Integrity, etc.) as values that inform or reflect institutional activities and strategies

•

meet both the content and style requirements of the Comment Guidelines

Writing Actionable Comments The Systems Appraisal’s goal is to give the organization feedback it can use to (a) take actions that will help it sustain the good things it is already doing, and (b) take actions to improve the things it could do better. In both situations, the stress should be on action. Without being prescriptive, you need to give advice and feedback that the institution can translate into a course of action. •

In citing a strength, say or illustrate (with an example) what it has accomplished for the organization — the “so what?”

•

Suggest how strengths could become more valuable by deployment throughout the organization.

•

Explain the “so what?’ by linking a strength with one of the AQIP Principles of High Performance Organizations, and suggest how the strength can be leveraged into organizational culture change.

•

For opportunities for improvement, explain what positive consequences could happen (or what negatives could cease happening) if the opportunity was exploited — the “so what?”

•

Note the costs of not seizing an opportunity for improvement — particularly the cost of standing still while others (the competition) is moving on improvement.

Style Guidelines •

Use the full name of the organization or its own shortened name to refer to the organization. For variety, use the institution or the organization as synonyms.

• •

Echo the organization's terminology whenever appropriate. Make it clear that you have read its Systems Portfolio and are writing in reaction to what it says. Keep your tone polite, professional, and positive.

•

If something is unclear, explain what is missing and why its omission is important. ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 25

Systems Appraiser Guide

•

Focus on the organization's substantive strengths or improvement opportunities, not its writing style or graphics. Do not comment on writing errors or style (e.g., the Portfolio needs a good proofreading or writing overuses the passive voice) in your feedback (although you should alert AQIP staff if poor writing becomes an obstacle to your appraisal).

•

If there are problems in understanding graphs, tables, or figures, write a single comment summarizing the difficulties encountered (e.g., variables in tables are presented inconsistently, making it impossible to distinguish counts from percentages or readers cannot determine which direction represents improvement in line graph figure 2-3).

•

Identify strengths or improvement opportunities according to where the issue falls in the Categories, not by where the organization places the information in its Portfolio. For example, if an institution discusses its faculty’s needs under Category 3 (rather than Category 4, where the discussion should occur), the appraiser should read this and include it in responding to Category 4.

• •

Use vocabulary and phrases from the Categories, Principles of High Performance Organizations, and Category Summary Statement Guidelines in your feedback comments. Avoid jargon and acronyms, unless they are used by the organization.

•

When referring to a figure, table, or chart, provide the figure number used in the Portfolio.

Grammatical Considerations Focus on the substance of the comments you write, keeping the following considerations in mind. This will save time and effort in editing and writing the final feedback report. Examples are in italics type.

Tense

When possible, use the present tense of all verbs throughout the feedback.

Active Voice

Prefer active voice to passive. With active voice, the subject of the sentence names the actor. With passive voice, the subject of the sentence is the noun acted upon, and the actual actor is often not explicitly named.

All students take a writing assessment annually, and the university summarizes these results each year, but the institution collects no comparative results for achievement in student writing, making it difficult for faculty to assess program effectiveness relative to that of competing institutions. Active voice: The Director of Human Resources completes the survey process for assessing faculty satisfaction each May. Passive voice: The survey process for assessing faculty satisfaction is completed each May.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 26

Systems Appraiser Guide

Acronyms

Numbers

Agreement

Capitalization

Capitalization

Spell out all abbreviations and acronyms the first time they are used and indicate the abbreviation in parentheses. If you use a term only once, do not abbreviate or substitute an acronym.

Human Resource (HR) managers are responsible for leading the Dewey University Action Projects (DUAPS).

Spell out numbers from one through ten. If a number is used as the first word of a sentence, spell it out. Otherwise, use figures for numbers above ten.

There were five union workers in ten meetings. Eleven meetings are held throughout the quarter. The new library holds 200,000 more books than the facility it replaced.

When numbers greater and less than ten occur in the same sentence, use figures for the numbers.

The program has graduated only 15 students in the last 8 years.

Refer to an organization, institution, board, or committee as "it," not "their." Don’t let the fact that a singular body includes many people confuse your selection of the right pronoun or verb.

The institution rewards its faculty for attention to students. The Faculty Senate has not revised its operating principles in 15 years. Some faculty members are unhappy, but, as a whole, the Faculty is satisfied with the situation.

Follow the organization's own conventions for capitalizing organizational divisions, departments, staff titles, and the names of teams and internal processes.

Director of Human Resources, Data Process Improvement Team, English and Physics faculty, the new library, the President’s Office, etc.

Do not capitalize "the organization" or "the institution." Capitalize references to the AQIP Categories, their names (e.g., “Helping Students Learn”), and AQIP processes.

The institution began two Action Projects relating to AQIP Category 1, “Helping Students Learn,” in 2002-03.

Capitalize “Criteria” or “Criterion” when referring to the Commission’s Criteria for Accreditation, and capitalize “Category” or “Categories” when referring to the AQIP’s nine Categories.

The college meets the Commission’s first Criterion, in part, because of its strong processes for addressing AQIP Category 3, Understanding Students’ and Other Stakeholders’ Needs.

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 27

Systems Appraiser Guide

Capitalize references to specific AQIP processes or activities.

Strategy Forum, Action Projects, Annual Updates, Reaffirmation of Accreditation, Organizational Overview, Systems Portfolio Awkward: Each staff member is responsible for his or her own professional development.

he/she & (s)he

Avoid the awkwardness of he or she, he/she, (s)he, his and her, him and her by making nouns plural.

Sexist language

Avoid sexist terms like chairman, faculty wives, etc.

The chairs of departments make special efforts to include faculty spouses in social activities.

Items in series

Put a comma before the conjunction (and, or, or nor) when three or more items are listed in a series.

No federal, state, or local sanctions have been imposed against the organization for the past five years.

Figure References

In Figure references, spell out the word "Figure."

Write Figure 1.1-1, not Fig. 1.1-1.

Better: All staff members are responsible for their own professional development.

Frequently Asked Questions for Systems Appraisers I don’t understand what I should be doing next in the Systems Appraisal. Who can help me? Your team leader is the first person to call when you have questions, but the AQIP Systems Appraisal Process Manager is also a source of information for Systems Appraisal Team members and leaders. The name and contact information of the current process manager is available on the AQIP website at www.AQIP.org.

Must every member of the appraisal team do an Independent Review of all nine of the AQIP Categories in a Systems Portfolio? Wouldn’t it be more efficient to have two or three team members review each Category? Every team member, including the team leader, must review every Category. Although it might be quicker and less burdensome to have each Category reviewed by only a subset of the team, doing so would seriously weaken the quality of feedback the organization receives. It might limit the number and quality of ideas the team generates in identifying strengths and opportunities for improvement, and it might make it difficult for some team members to participate in the consensus conversation (on Categories they did not independently analyze), thus making the feedback report something other than the consensus of the team’s views.

Should our team limit the number of comments we provide to the organization under each AQIP Category? Should we balance the number of comments pointing out strengths (S, SS) with those pointing out opportunities for improvement (O, OO)? Include in your feedback all comments that the team agrees on that respond to the Systems Portfolio. Typically team members in their Independent Review will generate 10 - 15 comments for Category 1 and about 5 - 10 comments for Categories 2 - 9. (Systems Portfolios usually ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 28

Systems Appraiser Guide

devote about twice as much attention to Category 1 as to any other Category.) After the consensus review, about the same number of agreed-upon comments will remain for each Category, but usually the consensus comments themselves will be fuller, particularly those that represent opportunities for improvement (O, OO). If you arbitrarily limit the number of your comments, two problems occur. First, the insights and creativity of the team members is stifled in their Independent Reviews. Second, the organization is prevented from getting the full range of consensus ideas that it deserves and needs to maximize its effectiveness. Don’t worry about the ratio of strengths and opportunities. Call ‘em as you see ‘em, honestly and candidly. Typically, higher Category Summary Statement scores on a Category (5, 6, or 7) go to organizations whose strengths greatly outnumber their opportunities for improvement, but Category Summary Statements depend heavily on the significance of the particular strengths or opportunities identified. One OO comment might outweigh several S comments.

AQIP Category 7 is about data and measurement, but there are also questions about measures and results in each of the Categories. Where should we expect a college or university to talk about these issues? Where should we comment about measurement, information, or data? AQIP expects that colleges and universities will specify, under each of the nine Categories, the ways that they measure the processes and goals for that Category. Thus, the Systems Portfolio section on Category 1 will explain how the organization measures student learning, and will present performance data that these measures produce. The section on Category 5 will explain how the organization measures and evaluates leadership and communication, and present the results. There should be an array of measures and results for each of the nine Categories. The appraisal team needs to point out any gaps — missing measures or missing results — and explain why their absence is a problem, or how plugging the gaps can stimulate improvement. Category 7 asks about the organization’s processes for selecting, gathering, storing, retrieving, and using data and information — of any kind. Where the other Categories ask about specific measures related to the processes that fall under each Category, Category 7 asks about the processes for deciding what information to collect and what not to collect. Category 7 asks about the security, currency, accuracy, and availability of stored data. It asks how the organization improves its data and information collection and retrieval processes. Like the other Categories, a college or university should have metrics for Category 7 (e.g., total cost of data and information storage, satisfaction of employees with data they need, frequency and extent of the use of information, and similar measures) and performance results data for these measures.

How many strategic issues should the appraisal team identify for a college or university? Strategic issues are typically broad, and can generate a variety of actions (action projects, initiatives, plans, tasks, or undertakings). If we expect an organization to concentrate its efforts on a few broad strategic issues, addressing them through a variety of projects, then it makes sense to limit the number of issues we identify as its highest priorities. Three or four is thus probably a sensible number. The rare organization that is highly practiced at continuous improvement, with a culture that can generate and attack new projects easily and complete them quickly, may be able to deal with more than three or four strategic issues. Similarly, too many OO items (or too few) may make it difficult for an institution to prioritize what is most important in the feedback. Reserve OO for those things that need immediate attention, typically those connected with a Strategic or Accreditation issue. ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 29

Systems Appraiser Guide

How should we relate strategic issues and accreditation issues? Most colleges and universities are using AQIP to maintain their accreditation, so staying accredited is of the utmost importance. Any opportunity or weakness that might endanger accreditation, automatically becomes a major strategic issue. Moreover, accreditation issues are almost invariably critical issues in their own right — issues about clarity of mission, educational purpose and effectiveness, sufficiency and effective allocation of resources, and similar core concerns. An appraisal team that perceives a central issue of this sort needs to place it high on the organization’s agenda. The team should explain how the issue might endanger the organization’s accredited status, and why the substance of the issue matters — how ignoring it would handicap the organization’s ability to accomplish its mission, serve its constituents, and, perhaps, maintain its vitality and integrity. In the feedback on strategic and accreditation issues, use letters (similar to using SS, S, O, and OO) to distinguish issues. Put “A” before any issue that jeopardizes accreditation (even if it is also of strategic importance), and “S” before an issue the team believes is strategic but not endangering accreditation.

We’ve identified an issue we think is very serious for this organization. In order to impress the leadership and faculty with the need to take action, should we identify it as an “accreditation issue” instead of simply calling it “strategic?” Accreditation is based on meeting the Commission’s five Criteria for Accreditation, and the twenty-one Core Components under those Criteria. Identify as “accreditation” only issues that truly endanger an organization’s ability to demonstrate it meets the Commission’s Criteria for Accreditation. Remember: accreditation standards are a floor that must be met by all accredited colleges and universities, and represent minimum requirements only. Organizations serious about quality and continuous improvement should hold themselves to far higher and more demanding expectations than those embodied in accreditation criteria, and so should an AQIP Systems Appraisal team. Therefore, it is quite legitimate for your appraisal team to identify for a college or university a strategic weakness that traditional accrediting processes would ignore. To participate in AQIP, an organization must meet all of the regular accreditation standards. But AQIP expects an organization serious about quality to do far more than just meet minimum standards. If your team identifies a serious accreditation concern that is a “sin of commission” (rather than an instance where the institution has simply not yet provided enough evidence in its Portfolio to substantiate that it meets every Core Component, your team leader should call the AQIP staff immediately — before you submit the final Appraisal. Call Steve Spangehl at 800-621-7440, x106, or, if you can’t reach Steve, call another staff member and ask them to have Steve call your team leader directly.

What might be an example of a strategic issue that is not also an accrediting issue? To take just one example, few accreditation agencies have standards that require a formal process for leadership succession — a thoughtful, proactive approach to developing new leadership talent, nurturing or mentoring future leaders. Yet quality organizations typically take leadership succession and continuity very seriously, and work hard to address these areas with effectives processes and plans. If your team sees an important concern like this, don’t hesitate to identify it as a strategic issue for the organization. But resist associating it with accreditation ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 30

Systems Appraiser Guide

simply to “get the organization’s attention.” Instead, get the appropriate attention by explaining in your comment why the issue is critical and what might happen if it continues to be ignored. AQIP places demands on organizations that go far beyond accreditation requirements. AQIP expects colleges and universities to gather comparative data and use it to set targets for performance, and to identify best practices or benchmarks to study to learn how to improve specific processes. A failing here would be a serious strategic issue, but unrelated to any accrediting requirement contained in the Criteria for Accreditation. Accrediting standards are focused on specific key areas. There are many, many issues of serious concern to colleges and universities that accreditors do not address at all. AQIP wants these issues “on the table” if they are serious, and if addressing them can help an organization better meet and serve the needs of its stakeholders. If your team identifies an accrediting issue, the team leader should speak with Steve Spangehl, AQIP Director (at 800-621-7440, ext. 106, or [email protected]), either before the Consensus Conversation at which the issue will be discussed or (if the issue emerges as the result of a Consensus Conversation) immediately afterwards. Issues touching accreditation are serious, both for institutions and the Commission, and often require immediate action by the AQIP staff.

As I was working on my Independent Review, I discovered there was critical information missing from the Portfolio that I need to provide the organization helpful advice. Can I call them to get clarification on how they currently do things? No! It is critical that appraisers not contact the organization during or after the appraisal. AQIP does not reveal the identity of specific appraisers to colleges and universities; by maintaining confidentiality on the identity of reviewers, AQIP forces the organization to grapple with the reviewers’ feedback, preventing the organization from engaging in the tempting practice of dismissing advice or criticism from reviewers whom organizations might reject as “people not like us” or “those who can’t appreciate our position”. Often it is just those reviewers with the freshest perspectives who make the most insightful and helpful comments. If you think critical information is missing, write a comment explaining what is missing, why its absence is critical, and what the consequences of omitting (or including) it might be. If the missing material makes it impossible to determine whether the organization meets accreditation criteria, say so: the organization will have ample opportunity, before the Commission next judges its accreditability, to fill the gap, but only if you tell it the gaps exists. Finally, if you believe that a review is impossible without specific additional materials, contact your team leader. Under certain conditions, team leaders can contact colleges and universities to request specific additional information addressing specific questions. This is most likely to happen when a team lacks the information it needs to agree on a Critical Characteristics Analysis of the organization, a task which must be completed before the Independent Review of Categories can begin. For example, if nothing in the Portfolio indicated the size, enrollment, or total budget of the organization, the team might direct the team leader to call the organization and get this vital information before proceeding to the rest of the Appraisal.

How does AQIP avoid conflict-of-interest situations in the System Appraisal process? Before it makes assignments to Systems Appraisal teams, AQIP asks all appraisers to identify those colleges and universities they could not fairly and objectively review. Organizations are also given the roster of all AQIP Systems Appraisers, and are asked to identify any the ©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 31

Systems Appraiser Guide

organization believes could not review it objectively. Thus, when teams are formed, no conflict of interest situations should arise. However, it sometimes happens that an appraiser does not become conscious of a conflict of interest until he or she reads the organization’s Systems Portfolio. Because of this possibility, it is critical that appraisers read (skim thoroughly) the entire Portfolio the moment it arrives, and call AQIP immediately if a conflict-of-interest becomes apparent. This foresight can prevent the disruption caused by replacing an appraiser midway through the process. However, any appraiser who comes to recognize a conflict of interest, no matter when the realization occurs, should contact AQIP immediately.

Whom should I call if I feel the Systems Appraisal is off-course: behind schedule, improperly organized or conducted, or seriously flawed in some other way? Contact the Systems Appraisal Process Manager (current manager’s name and contact information available in the Contact AQIP Staff section of the AQIP website at www.AQIP.org); Telephone AQIP at 800-621-7440, ask for Steve Spangehl, Director (x106) or Babatunde Alokolaro, AQIP Program Facilitator (x109), and explain concisely and frankly whatever concerns you. (Alternatively, email both [email protected] and [email protected].)

©2008 Academic Quality Improvement Program, The Higher Learning Commission. All rights reserved. Last updated 2/08. 32