Fifteen tools for efficient performance auditing l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 70

Fifteen Tools for Efficient Performance Auditing PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

Fifteen Tools for Efficient Performance Auditing. Drummond Kahn Director of Audit Services City of Portland, Oregon Webinar Presented to NASACT September 2010. Web Conferences Are Tough. You can’t see who’s talking But you can imagine what people might look like!.

Download Presentation

Fifteen Tools for Efficient Performance Auditing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Fifteen tools for efficient performance auditing l.jpg

Fifteen Tools for Efficient Performance Auditing

Drummond Kahn

Director of Audit Services

City of Portland, Oregon

Webinar Presented to NASACT

September 2010


Web conferences are tough l.jpg

Web Conferences Are Tough

  • You can’t see who’s talking

  • But you can imagine what people might look like!


The city of roses portland l.jpg

The City of Roses -- Portland


Portland s audit services division l.jpg

Portland’s Audit Services Division

  • www.portlandoregon.gov/auditor/auditservices

  • City has over 5,000 employees and $2 Billion budget

  • ALGA member office, follows Yellow Book standards


Greetings l.jpg

Greetings!

  • Thanks for participating in today’s conference!

  • We’ll have time for questions at the end of the session.

  • My contact information is on the final slide of today’s presentation.


What s a performance audit l.jpg

What’s a Performance Audit?

  • An objective, systematic examination of evidence.

  • Makes recommendations to decision-makers and to enhance public accountability.

  • Distinct from financial audits, which express opinions on financial statements.

  • Conducted in accordance with standards.

  • Has significant “barriers to entry”. Not everyone can call themselves a performance auditor!


Barriers to entry l.jpg

Barriers to Entry

  • Economists talk about “barriers to entry” as the rules, logistics, or processes that keep new companies from entering a marketplace easily.

  • Think about how tough it would be to start up a hospital, an insurance company, or an airline!

  • Audit “barriers” involve:

    • External quality control review (peer review)

    • Continuing education requirement for staff

    • Independence in fact and appearance

    • Operational structures like supervision

    • Reporting structure to those charged with governance


Special and unique l.jpg

Special and Unique

  • Performance audits are expensive, since:

    • Staff are highly-trained (CPE requirements),

    • Staff are well-supervised (which increases hourly costs compared to less supervision), and

    • Audits take more time than similar analysis done outside of an audit environment due to our internal and external quality control processes

  • Not every organization can field a performance audit function.

  • Government and the public should understand and appreciate the performance audit function.


  • Importance of performance auditing l.jpg

    Importance of Performance Auditing

    • Auditing has barriers to entry, may cost more than other analytical efforts, and has an important audience of the public and decision-makers!

    • We need tools to keep this important field timely and as efficient as possible.


    Our overture tools for efficiency l.jpg

    Our Overture – Tools for Efficiency

    • Like baseball, how more “at-bats” can help auditors effect organizational change

    • From Star Wars to Lord of the Rings, how movie trilogy production can inform auditing

    • Why the audit process should work more like Southwest than Northwest

    • Why audits should read more like Hemingway than Michener

    • Why audit concerns about reporting and timeliness date back to Shakespeare


    Today s sponsor l.jpg

    Today’s Sponsor

    • 15 Tools for Efficiency in performance auditing

    • Today’s presentation is brought to you by the number 15 and the letter “E”


    Conducting performance audits efficiently l.jpg

    Conducting Performance Audits Efficiently

    • “Efficiency” is the ratio of inputs per output:

      • Hours spent per audit report

      • Years spent per audit report (??!?!)

  • Highly efficient auditors and offices have “traction” to create audit yield with each hour of audit time invested

  • Core idea: Make each time investment count in your audit office

  • This work needs to lead to reports!

  • Public reports help us fulfill the promise of auditing by leading to change and by placing management “on the record” through written, published responses.


  • Initial tool kit l.jpg

    Initial Tool Kit

    • Develop clear objectives near beginning of audit. For unclear or new areas, slate a specific amount of time for survey and don’t exceed it without a “scoping meeting”


    Initial tool kit14 l.jpg

    Initial Tool Kit

    • Develop clear objectives near beginning of audit. For unclear or new areas, slate a specific amount of time for survey and don’t exceed it without a “scoping meeting”

    • Set and stick to a clear scope for the audit


    Initial tool kit15 l.jpg

    Initial Tool Kit

    • Develop clear objectives near beginning of audit. For unclear or new areas, slate a specific amount of time for survey and don’t exceed it without a “scoping meeting”

    • Set and stick to a clear scope for the audit

    • When scope is fulfilled and objectives are met, write the report!


    Initial tool kit16 l.jpg

    Initial Tool Kit

    • Develop clear objectives near beginning of audit. For unclear or new areas, slate a specific amount of time for survey and don’t exceed it without a “scoping meeting”

    • Set and stick to a clear scope for the audit

    • When scope is fulfilled and objectives are met, write the report!

    • Objectives can met and the report can be written, even when literal fulfillment of the objectives is impossible!


    Unmet objectives can be findings l.jpg

    Unmet Objectives Can Be Findings!

    If audit objective can’t be met, that’s a reportable condition.

    • “Determine which laws govern the State’s practices” – no laws mean that we’ve determined that “no laws govern practices”, NOT that additional audit work is needed right then.

    • “Determine whether policies are properly reported” – if not, that’s the finding.

    • “Determine whether agency expenses are properly supported” – if not, that’s the finding.


    Determining when you re done l.jpg

    Determining When You’re Done

    • In each of these examples, the objective was answered by a lack of data, not by additional audit work.

    • This time-saving tool simply has us answer the objective and write the report, rather than performing more work to prove a negative.


    Whose fault anyway l.jpg

    Whose Fault, Anyway?

    • “We were unable to determine. . .” can also be worded as “Management may lack the information it needs. . .” – and not just from the report, but from their own mechanisms!!!

    • Remember that data limitations aren’t the same as scope limitations! (Data not answering your objective is very different from someone restricting your access to that data)


    Substitute superauditor for auditor l.jpg

    Substitute “Superauditor” for “Auditor”

    • You’d never see this phrase --“Superauditor™ was unable to determine whether travel claims were supported”!

    • Try placing responsibility on management, rather than on the auditor.


    Managers manage auditors audit l.jpg

    Managers Manage, Auditors Audit

    • It’s in Auditing 101: Managers are responsible for managing, while auditors are responsible for auditing.

    • Shortcomings in the organization are NOT due to lack of data or a shortage of audit information. They are due to management choices.

    • Keep this in mind while framing findings.

    • Reports shouldn’t be about the audit process and its difficulties, they should be about the performance of management.


    Whose problem is missing data l.jpg

    Whose Problem is Missing Data?

    Consider these examples:

    1. “Data on the audit objective was lacking. Therefore, we were unable to conduct our audit or determine the condition that we sought to review” (No Finding)

    2. “Data on the audit objective was lacking. Therefore, neither auditors nor management have the information they need to make good decisions.” (Finding!)


    Victory from the jaws of defeat l.jpg

    Victory from the Jaws of Defeat

    • In one case, missing data means no finding.

    • In the same case through another lens, missing data becomes a finding and can guide management to improvement.


    Don t limit focus l.jpg

    Don’t “Limit”! -- “Focus”!

    • Instead of scope choices reading like limitations, consider wording scope choices as a focus –

    • “We limited our audit to XYZ” versus “We focused our audit on XYZ”.


    Audits shouldn t have birthdays l.jpg

    Audits Shouldn’t Have Birthdays!

    • Birthday parties are great for little kids.

    • Not so great for audit reports.

    • Audits taking longer than a year to complete put a longer distance between the problem and the report.


    Birthday conundrum l.jpg

    Birthday Conundrum

    • As audit work ages, you might need to repeat work to collect newer data, since the report will now be issued in a new year!

    • So a report set for 2010 but issued in 2011 would need to “refresh” its 2009 fieldwork.

    • This is bad.


    Really no birthdays l.jpg

    Really. No Birthdays.

    • I’m OK if I never have to take another audit report out for a pizza party.

    • Let’s celebrate concise, focused, early reports.

    • And let’s only have pizza when reports are released, not just when they turn one or two.


    Audit birthday avoidance protocol l.jpg

    Audit Birthday Avoidance Protocol

    • Consider the Audit Birthday Avoidance Protocol (ABAP):

      • Audits should be scoped tightly to take less time (remember that one full-time equivalent auditor spends about 1,500 billable hours per year).

      • Basic and regular team communication, including update meetings on scope and progress.

      • Adopt tools to quickly and cleanly decide on message/focus.

      • Positive organizational change is our goal, rather than only focusing on the report itself.


    Blueprints for improvement l.jpg

    Blueprints for Improvement

    • Assign both the responsibility and the positive results of being responsible to management!

    • Findings are not negative or “gotcha” results – they are blueprints for positive improvement.


    Not bad 80 to 90 percent l.jpg

    Not Bad – 80 to 90 Percent

    • Many audit offices have 80 to 90 percent implementation of audit recommendations!

    • An excellent process – auditors suggest improvement, and then management improves!

    • Audit reporting and results are uniquely bi-partisan!


    Missing data finding structure 1 l.jpg

    Missing Data Finding Structure “1”

    • Condition: Management data lacking or missing

    • Criteria: To do an audit, we need data as auditors

    • Cause: We can’t find the data

    • Effect: Audit on hold or cancelled

    • Recommendation: None – no published report


    Missing data finding structure 2 l.jpg

    Missing Data Finding Structure “2”

    • Condition: Management data lacking or missing

    • Criteria: Data was needed to make good decisions (decisionmaking, accountability)

    • Cause: Too expensive, not available, etc.

    • Effect: Audit ended early, management proceeding without complete information

    • Recommendation: Improve data!


    Use everything l.jpg

    Use Everything!

    • Just like a barbecue can use all parts of an animal, an audit should make the best use of its fieldwork.

      • Bolster OSM section

      • Use findings to propel the report

      • Use “absence of evidence” to show concerns


    Boosting yield l.jpg

    Boosting “Yield”

    • Use and bolster an “Objectives, Scope, and Methodology” (OSM) section

    • Tell the reader about all the work you did

      • Interviewed officials, reviewed data, analyzed information, compared/evaluated, reached conclusions – using your work to boost yield in the audit report.

      • This builds credibility and confidence in the report by users because they can “see inside” your work!


    Where to put the osm section l.jpg

    Where to Put the OSM Section?

    • For most audits with straightforward scope and methodology, putting the OSM section near the end of the report is OK, since readers won’t be slowed by it as they review the findings and recommendations.

    • For audits with extra controversy or anticipated reader concerns, OSM can be closer to the beginning.

    • Putting OSM earlier in the report is like taking an antacid before you eat a bowl of chili.


    Findings and results propel reports l.jpg

    Findings and Results Propel Reports

    • Readers like to see results (which we structure as audit findings) and the conclusion to work completed.

    • Tips later will discuss writing, but remember that results are most interesting.


    Absence of evidence l.jpg

    Absence of Evidence

    • As discussed on The X-Files,

      “absence of evidence

      is not

      evidence of absence”.

    • Lack of data does not mean that a condition doesn’t exist, just that there is no data.


    Match amount of evidence to risk l.jpg

    Match Amount of Evidence to Risk

    • Unusual claims require unusual evidence – the higher the risk of the audit statement, the higher the level of evidence needed.

    • But reviewers and QC should not require a huge amount of evidence for an ordinary finding.


    Lower risk lower evidence l.jpg

    Lower Risk = Lower Evidence

    • “Management processes could be improved” requires less evidence than “Management could be improved 73% and save $1,630,022.10.”

    • More detail and more risk increase the demand on the audit evidence supporting the report.


    Use the information you find l.jpg

    Use the Information You Find

    • If management data is missing, report it!

    • If management can’t use its own information, report it!

    • If data and analysis could be improved, report it!

    • If data, processes, and analysis are accurate, report what they show!


    Increase plate appearances l.jpg

    Increase “Plate Appearances”

    • Specific audits on specific subjects are easier to scope, plan, conduct, write, and review!

    • More specificity = more reports better tailored to each audience


    Increase plate appearances part 2 l.jpg

    Increase “Plate Appearances” Part 2

    • More reports allows your office to show a series of efforts on a topic, rather than bunching different findings into one report.

    • Each topic can create momentum for change.


    More contacts more results l.jpg

    More “Contacts” = More Results

    • Business is big on the notion of “touches” and “contacts” with clients.

    • Extended to auditing, it’s a good goal to have more contacts with decision-makers, the public, and the media.

    • Audit reports are the perfect vehicle to do this, but we need more reports to have more contacts!


    Which works better l.jpg

    Which Works Better?

    • High costs

    • Had declared bankruptcy

    • Charged for snacks

    • Charged for checked bags

    • Flew fewer, longer flights per day


    Which works better45 l.jpg

    Which Works Better?

    • Lower costs

    • Profitable

    • Free snacks

    • Free checked bags

    • Flies more short flights per day


    Which works better for auditing l.jpg

    Which Works Better for Auditing?

    • Producing just a few large reports and contacts per year?

      OR

    • Producing more short reports, leading to many more contacts per year?

    • “Contacts” include public, decision-makers, media, and other audit report users

    • More reports may enhance meeting our goal of informing the public and enhancing decision-making


    Quick response and efficiency l.jpg

    “Quick Response” and Efficiency

    • Briefly, “quick response” audits limit scope and fieldwork to focus only on the objectives of the audit.

    • So should all audits!!!

    • Consider your scope early on – it will dictate fieldwork, writing, and review!


    Setting a clear scope l.jpg

    Setting a Clear Scope

    • The clearer the focus of the planned audit, the easier fieldwork will be and the better you’ll know when you’re finished.

    • Discuss scope regularly, and whether current fieldwork is still within the planned scope.

    • Avoid “scope creep”!


    Focused scope focused fieldwork l.jpg

    Focused Scope, Focused Fieldwork

    • Summarizing the tools so far, a specific, focused scope can lead to specific, focused fieldwork.

    • With a clear scope, you can tell when you are finished.

    • Clear scope makes the report easier to write, helping with everything from the title to the summary to the recommendations.


    Focused fieldwork focused report l.jpg

    Focused Fieldwork, Focused Report

    • Clear fieldwork leads to an easier-to-write report, and a better focus in the report on the important issues.

    • Deadlines, fieldwork plans, and sticking to them actually helps complete an audit report.

    • Does focus squelch creativity?


    Big dilemma on creativity l.jpg

    Big Dilemma on Creativity?

    • Our offices hire smart, nice, thoughtful people with lots of great ideas.

    • We also have deadlines and deliverables.

    • Creativity and brilliance in auditing need not mean that we miss deadlines.

    • What’s more insightful than issuing reports and inspiring change???


    Clear reporting helps prompt review l.jpg

    Clear Reporting Helps Prompt Review

    • The more focused the report is, the easier and faster it is to review.

    • Reviewers may have additional questions and ideas for future audit topics.

    • Consider these in future audit planning, but don’t curtail reporting what’s in hand.


    Action from audits requires reports l.jpg

    Action From Audits Requires Reports

    • It’s issuing reports that allows us to achieve the promise of performance auditing.

    • No report issued = no change (or no result from the audit work).


    Efficient audit reviews l.jpg

    Efficient Audit Reviews

    • “Three drafts or it’s out!”

    • Focus report on key objectives, defer other issues to future reports (can mention this in current report)

    • Use visuals – photos, charts, maps, sketches to explain


    Efficient audit reporting l.jpg

    Efficient Audit Reporting

    • Die Hard-style reporting (no opening credits to slow reader)

    • Movie-making and audit-making

    • Star Wars trilogy vs. Lord of the Rings – timing of fieldwork vs. timing and packaging of reports


    Department of redundancy department l.jpg

    Department of Redundancy Department

    • Avoid recycling editing comments to the point that a second editorial change shifts the text back to be the way it was before the first change!

    • Distinguish between line edit / formatting comments and content suggestions.


    Activate audit findings l.jpg

    “Activate” Audit Findings

    • Consider “action” titles for reports that describe major finding --

      • “Audit of Police Bureau”

      • “Police: Crimes Unsolved”

      • “Police: Crimesolving Could be Improved with Policy Changes”

    • Which do you prefer?


    Presenting results l.jpg

    Presenting Results

    • Think about the “elevator speech” – your main points of the audit distilled into 30 seconds

    • Consider use of summary with elected officials, media, and public

    • Focus on findings


    Remember your audience l.jpg

    Remember Your Audience

    • Audit readers tend to be busy

    • State main point in title, in summary, and throughout the report

    • Write at 8th grade level with short sentences (conversational, focused, and for the public)


    Hemingway not michener l.jpg

    Hemingway, not Michener

    • Keep sentences short


    Long reports may not be read l.jpg

    Long Reports May Not Be Read!

    • Nothing is wrong with a long novel for your summer reading –

    • Audit reports, though, should be reasonably easy to digest!

    • Brevity is the soul of an audit report that’s used!


    Release delays are shakespearean l.jpg

    Release Delays are Shakespearean!

    • “And how his audit stands, who knows, save Heaven?” – Shakespeare, Hamlet (Act III, Scene 3)

    • We should clearly communicate where our audits stand and the status of recommendations!

    • Update auditees and the public.


    Inspiring action with auditing l.jpg

    Inspiring Action with Auditing

    • “I’ll be an auditor – an actor too, perhaps, if I have cause.” – Shakespeare, A Midsummer Night’s Dream (Act III, Scene 1)

    • Audits can inspire action and change by management.

    • Audits help change happen through well-supported, published recommendations.


    Two kinds of audit reports l.jpg

    Two Kinds of Audit Reports

    • The kind that is perfect in every respect, with absolutely nothing that could ever improve it.

      OR

    • The kind that gets finished.


    Summary of tools discussed l.jpg

    Summary of Tools Discussed

    • Set and stick to a clear scope

    • Limit fieldwork to that scope

    • When objectives are met (or it’s determined that they can’t be), start writing

    • Use high-yield OSM sections

    • Place the focus on the auditee, not on the auditor, when data is limited


    Summary of tools discussed66 l.jpg

    Summary of Tools Discussed

    • Write more, shorter reports (plate appearances)

    • Focus on “quick response” by limiting scope, fieldwork, and reporting to a specific issue

    • Avoid audit birthdays

    • Consider low-cost, high-frequency audits rather than high-cost, low-frequency to maximize contacts and efficiency/yield


    Summary of tools discussed67 l.jpg

    Summary of Tools Discussed

    • Be visual

    • Cap draft reviews at three

    • Die Hard, Star Wars vs. Lord of the Rings: How to best package a body of audit work

    • Use “action” titles for reports and in your short descriptions of your work

    • Know thy audience, simplify

    • Write the report that gets finished


    The promise of auditing l.jpg

    The Promise of Auditing

    • Improving government with recommendations

    • Improving public accountability

    • Enhancing decision-making

    • Showing a “footprint” of change and of the value auditing adds


    What s not to like l.jpg

    What’s Not to Like?

    • A track record of audit recommendations implemented and of government improving!

    • Auditors suggest improvement, and management improves!


    Questions and contact information l.jpg

    Questions and Contact Information

    Drummond Kahn

    Director of Audit Services

    Office of the City Auditor

    City of Portland

    1221 SW Fourth Avenue

    Room 310

    Portland, Oregon 97204

    (503) 823-3536

    [email protected]

    www.portlandoregon.gov/auditor/auditservices


  • Login