1 / 55

CONSENSUS TRAINING July 10, 2019

Learn how to synthesize individual scorebooks into a team scorebook, prepare for site visits, and provide actionable feedback to applicants.

arlenef
Download Presentation

CONSENSUS TRAINING July 10, 2019

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CONSENSUS TRAINING July 10, 2019

  2. SEPTEMBER TODAY

  3. Independent Review Independent Review Consensus Site Visit

  4. The Consensus Process

  5. Overview Purpose: Create high quality, actionable feedback for the Applicant Action Plan: Synthesize individual scorebooks into one team scorebook which will become the basis for Site Visit Strategy and Final Feedback Report Keys to success: • Strength comments identifying criteria nugget, key factor(s), application examples and process maturity (ADLI or LeTCI) • Next step OFI Concepts or Questions to guide site visit planning

  6. Roles • Team Lead – manages team & consensus process • Item Lead – responsible for synthesizing comments for specific assigned Item(s) • Item Backup – reviews and provides feedback to ItemLead to validate and improve the final output

  7. Steps • Item Lead Synthesizes Team IRs in Navigator • Item Lead Score • Prepare Script • Item Lead follows Script • Discuss comments • Team Score • Take notes to edit comments later • Edit comments per consensus • Report due to MEI Office w/in 7 days • Site Visit Planning

  8. Preparing for Consensus

  9. Preparing an Item for Consensus Meeting

  10. Synthesize Comments Examiner 6 Item 1.1 Examiner 7 Item 1.1 Examiner 5 Item 1.1 Examiner 4 Item 1.1 Examiner 3 Item 1.1

  11. Navigator - Stage 2

  12. Navigator – Review Stage 1 Comments For each of your assigned items in the Stage 2 scorebook Click on this icon to show the entries from all the Stage 1 IRs

  13. Export from Navigator Click on this icon to open the print preview/report for this item. Click on this icon to export the report data (Key Factors, Strengths and OFIs) to excel or word.

  14. Ways to Sort/Review:Paper OR Excel Method Team Member IRs Team Pre-Consensus See Cheat Sheets in Examiner Resource Center

  15. Volunteer Resource Center

  16. Synthesis: Excel tool • Export the Item to Excel • Sort comments by Item Ref • Cut and paste “same item ref” comments into a new tab • Read/review the common comments • Create the “best” comment by selecting one from IR or combining elements from several IR comments • On consolidated comment…indicate which examiners (initials) agree with comment concept - not word for word • Repeat until you have the best 6-8 proposed Comments for the item (AA, BB, CC)

  17. EXAMPLES

  18. EXAMPLES

  19. How to write a Consensus Comment Consensus Comment may be: A single IR comment which best expresses a unique or a shared IR idea (for example: 3 examiners comment on the same criteria with similar ADLI evaluations  select the best written comment) OR Combined elements and/or language from several IR comments OR A new comment composed using the ideas expressed by one or more IR comments

  20. Frequently Asked Questions Q - Do I have to use all the IR Comments? A – No, select the “best” IDEAS from the IR comments Q – What should I do with IR Comments I didn’t use? A – List them on the Comments Not Used form Q – What are the “best” IR comments? A – Comments which align with the Criteria and the applicant’s Key Factors; clearly communicate the organization’s next step toward process maturity Q – What if the IR comments conflict (for example, a strength and an OFI addressing the same criteria)? A – The Item leader should evaluate the comments based on the Criteria, the application responses, and Key Factors to determine whether the organization will be better served if the comment is a strength or an OFI, for example: a Strength for Deployment versus an OFI for Learning.

  21. Comment Content Comments Include 4 elements

  22. Characteristics of GOOD Comments • GOOD COMMENTS… • Are clearly aligned with the Criteria • Contain only one main idea supported by 2-3 examples • Are evaluated on maturity of process (ADLI) or results (LeTCI) • Use the applicant’s terminology – not “Baldrigese” • Are actionable and relevant to the applicant • Add value - not just restate application • Are factual not judgmental

  23. Entering Synthesized Comments in Navigator • The Consensus Scorebook’s format is slightly different than the Independent Review Scorebook: “As Evidenced by” is now called “Rationale”. And there are no checkboxes for ADLI or LeTCI. • Select 4-6 Key Factors for each Item at the top of the Scorebook • Paste all final comments into the Strength or OFI areas • Check the Key Factor Reference and Item Reference boxes for each comment. • Use “Rationale” for notes, ADLI/LeTCI, site visit issues • Enter your score • Let your team leaders and item backup know the item is ready for review.

  24. Comments Used/Not Used Comments Not Used Matrix.xls • Download form from Examiner Resource Center • Indicate why comment wasn’t used (examples include: not actionable, lower level/not next step, not criteria based, etc.) • These may be brought back later (either at consensus or site visit)

  25. SCORING

  26. Scoring Guidelines

  27. Process Category Scoring Guidelines Results Category Scoring Guidelines

  28. Score the Item • Thinkholistically at the ITEM Level • Score is determinedby the Strengths (& SOFIs) • No strengths  score is 0 • No OFIs  score is 100% • Process or Results (ADLI/LeTCI) • OFIs should describe what is needed to get to the • next higher box = applicant’s next steps • Scorein 5% increments (50, 55, 60, etc.) NOTE: This is your PROPOSED Score based on synthesized Item Comments

  29. Example: Item 6.2

  30. Entering Scores in Navigator 1. Start in the 50 -65% column. 2. Determine if 50-65% is too high, too low, or just right. Select best score range. 3. Then choose the best score within that range (in increments of 5%).

  31. Consensus Meeting Preparations

  32. Review YOUR Item(s) Review synthesized comments: • Covers the main objectives of the Criteria (a1, a2, b1, …) • Contains: Criteria nugget, Key Factor, evaluation of process maturity, 2-3 examples from application or what’s missing • Provides actionable and relevant feedback • No conflicts or contradictions between comments (Strengths vs OFIs or other items) • Verify that the score accurately reflects the Strength comments

  33. Share your Item(s) & Make Edits from Feedback • Enter your proposed Comments and Score in Navigator • Share with Item Back-up and Team Leaders with adequate time for feedback • Incorporate B/U and TL feedback into Navigator comments and rescore if needed

  34. Item Backup - Role & Responsibilities The role of the item Backup is to provide a critical review of the Consensus comments: • Requires use of: Criteria book, Application, Key Factors • Proofread for grammar, spelling, punctuation • Wordsmith – for clarity and accuracy • Do Strength Comments Conform to the Comment Guidelines? • Criteria, key factor(s), relevant examples and process maturity (ADLI/LeTCI)? • Are they actionable and appropriate for organization? • Adequate length (not too short or too long) • No conflicts or contradictions between comments (Strengths vs OFIs or other items) • Does the score seem appropriate for the Strength comments

  35. Script Template (1 Item per form) Presentation Script Template See Consensus Forms on the Volunteer Resource Center website

  36. Complete Scripts Presentation Script includes: • Criteria Intent (Summary only) • Key Factors for this Item • Comment Summary…DO NOT paste the full comments just the intent (ADLI/LTCI) • Proposed Score and Rationale • Which Box, why? • Why not lower band/Why not higher band? Backup/Team Lead review/edit script

  37. 1-2 Days Before Consensus

  38. Familiarize yourself with Consensus Scorebook– Every Item EVERYONEmust be knowledgeable about the NEWScorebook • Read the complete Consensus scorebook – it’s not the same as your IR Scorebook • Look for: • Conflicts - between Strengths and OFIs both within and across Items • Clear, Good grammar, spelling, etc. • Do you Agree? (evaluation of process maturity, balance of Strengths & OFIs, scoring, etc.) • Do you see cross-cutting ideas or themes? (Strengths or OFIs) • Accurate?...based on Application, uses Applicant’s terminology • Actionable? … clear and concise next steps • Aligned? … with criteria ...with scoring box • Appropriate? … professional tone, non-judgmental • Make notes of your questions or things to discuss during Consensus and potential questions for site visit

  39. Consensus Call

  40. Consensus Call – Roles & Responsibilities Team Leader • Present Agenda/Ground Rules • Revisit Key Factors • Discuss Key Themes Overseer • Ensure integrity of the process • Mediate Criteria questions Item Lead • Read prepared script summarizing Comments (Str & OFIs), Proposed Score and rationale • Lead discussion of conflicts or concerns • B/U takes notes Team Members • Discuss item (questions, conflicts, etc.) • Resolve issues • Agree on changes • Listen for conflicts and integration with other Items throughout Report • Offer wordsmithing suggestions in writing only, share with Item lead • Poll Individual Scores - Adopt average of scores if in “same box”, else discuss until consensus is reached • Identify Key Theme concepts • Repeat for all Items • Revisit any Parking Lot issues

  41. Tips and Ground Rules • Allow time and follow Agenda 8 hours (or more) – one day or two half days? • Choose a timekeeper to stay on schedule • Schedule and take breaks • All other devices turned off while on call - no multi-tasking • Everyone in a private/quiet room and use Mute when not talking • Say your name before speaking • “Not used” comments are not discussed unless someone asks • If you disagree or don’t understand, ask questions - don’t cave. Everyone has an equal voice on every item • Go around “room” asking for input, scores, or questions (rotate sequence) • Overseers make good scorekeeper • Use a Parking Lot for longer unresolved discussion items; come back to them later using Consensus Resolution Form • It’s not over until all the work is complete • Discussion stops if anyone leaves the call

  42. What to have with you during Consensus Call • Quiet room, good chair, snacks, phone • If using cell phone, have charger cord • Headset/speaker and Mute function • Computer with Navigator or Printout of FULL Pre-Consensus Comments and Scores • List of your comments/questions/issues to discuss (All Categories) • Criteria Book • Application • Key Factors • Results Spreadsheet • Your Presentation Scripts, Comments Not Used, and any materials for reference • Paper, pens, highlighters, post-it-notes, etc.

  43. Team Scoring • If everyone’s scores are within the same band (i.e. 50-65%) - use the average of the scores (rounded to nearest 5%) • Otherwise, the Overseer will lead a discussion to understand the reasons some examiners gave higher and/or lower scores • The goal is to move the team toward a “consensus” score where variance is <= 20 points and everyone is comfortable within a single band

  44. Key Themes

  45. What are Key Themes? • “Executive Summary” of Feedback Report • Address important strengths or OFIs as reflected in Key Factors – usually 2-3 Strengths, 2-3 OFIs, 2-3 Results (+/-) • Traceable to Comments/Key Processes • Are cross-cutting (appear in multiple Items) OR • Address a significant issue in one Item (role-model practice or threat to sustainability) • May address a Core Value of the Criteria • Summarize how well Criteria requirements are addressed • May address ADLI or LeTCI

  46. Key Themes Matrix Core Value & Key Theme Matrix.xlsx

  47. How to Fill Out the Key Themes Matrix O2 O1 O3 O2 S1 S3 S2 S4 S5

  48. Consensus Comment #s Where are groups of OFI comments? Where are groups of Strength comments?

  49. Writing Key Themes Writing a Key Theme is similar to writing a comment:

  50. After the Consensus Meeting

More Related