It’s poor economy to use bank notes to boil water for tea, not very efficient to boil five litres of water to make only one cup of tea, and certainly not effective to produce a cup of tea, when the customer wants a cool drink.
Sometime in the late 1960s,
I became fascinated with what was then called “Operational Auditing”. By the seventies, it had been renamed “Management Audit”, and by the eighties, “Value for Money Auditing”. By the nineties, it had broadened its title to “Performance Audit” and “Program Audits”.
This article relates the author’s experiences in completing such audits in both the private and public sectors now over five decades, and ending with the Performance Audit of Knowledge in the know-how sector.
Performance Audits – The Concept
If an enterprise has a fleet of trucks, the financial auditors’ role is to satisfy themselves only that the trucks exist; their client is the legal owner, that the fair value has been correctly established and the vehicles have been properly disclosed at the reporting date.
The performance auditor’s role is, by examination of the existence of management systems and decision processes before, during and at the end of this economic value activity that:
- the trucks were required in the first place,
- the purchase price and financing was the best value that could have been obtained.
- the trucks are being used and maintained economically, efficiently and effectively, and
- they were worth the investment made
in them.
1960s – Operational Auditing on the High Seas
My first run at an operational audit was to evaluate and report on the performance of tanker fleets for a major UK oil company. These massive ships, when the Suez was closed, transported crude oil between the Gulf and European ports. At this time, the two basic elements of performance – economy and efficiency – were intended to produce the final result, effectiveness.
- Economy relates to the spending of money, in my case fuel, salaries and ships’ provisions .
- Efficiency was making best use of the resources (ships, voyage planning, crew numbers and, most importantly, time).
Thus the audit was about costs and the saving of time.
Even in these early days, we recognised that measuring performance required something to measure and standards to which to measure against.
Nautical miles per 100 gallons of fuel used, required having available these two actual numbers (miles travelled and fuel used), as workload indicators. It also required information systems to collect the data. Standards used were either some past performance of the same tanker, those of similar ships, or some generally accepted performance benchmarks.
There were three people in the audit team, one expert in tanker management, one explosives expert and one expert in finance. Fire was at that time a very high risk – thus the explosives man. Even in those days, we were focusing our audit selection on high risk system issues and the related potential loss.
Unlike financial audits, we were expected to follow the evaluation with a conclusion and, if requested, make recommendations for improvements. It was during these operational audits, that we developed the following six stage process;
- Statement of the Condition – What we found or observed.
- Criteria – (What should be!) Standard or Policy that should have been followed
- Cause – (Why did it happen?)
- Effect – in terms of money – (does it matter?)
- Statement of Conclusion – (Summary of good and less-than-satisfactory performance.)
- Recommendation – (What should be done to improve performance?)
Poor performance was often caused by lack of a clear management policy or guidance, not by any inefficiency in the operations. Evaluating managers was yet to come.
1970s Management Audits
The next decade was indeed all about management. Universities were expected to have MBA programmes in place and managers were mandated to run everything. Management of health services was taken away from doctors and given to managers. Mayors were dispensed with and city managers took their place. To get promotion you had to give up what you were trained to do, (engineer, architect, accountant) and become a manager.
Management Audit was developed to evaluate and conclude on how well these new managers were performing against generally accepted management standards.
Effectiveness became the third key evaluation criterion, joining economy and efficiency, Effectiveness means producing profits, outputs and results of an appropriate quality, as a result of being economical and efficient.
Indeed the words “audit” and “auditor” came to be used by a range of non-accounting experts. This was a great concern of the accounting profession, which assumed that it had the proprietary rights over the word ‘audit.’
Effectiveness in Extreme Situations
My most testing effectiveness audit assignment was to conclude on the effectiveness of international earthquake relief operations in Turkey. We had the earthquake; we had the survivors and six months of assistance to audit. Effectiveness in this situation was assessed by the following four sets of indictors:
- Results — Numbers of people assisted, medical operations performed, meals served and money provided and spent.
- Acceptability — The views of those assisted, and those of the experts in aid provision.
- Benefits — Indicators of how much the recipients benefited, compared with the situations where no assistance had
been given. - Impact – indicators of the long term positive effect of the aid for the region,
Effectiveness in this context has all to do with quality of service. The cost of one life saved was very high. However economy is not a major criterion in natural disaster situations.
As with economy and efficiency, standards should have been set to evaluate performance. However, in aid relief type of situations:
- standards must be set in relation to the resources actually made available; and
- they must be interpreted in the light of what was possible always under very difficult circumstances.
All these indicators and the standards were part of the objective of developing a formal effectiveness reporting system for this type of operation.
1980s – Value for Money Audits
Mrs Margaret Thatcher, the then Prime Minister of the United Kingdom, was very keen on public sector performance standards and had indicators for just about everything. Value for Money audits were soon built into budget and execution processes, and made good money for the auditors.
Certain of the indicators for satisfactory performance were measurable by hard numbers; such as money and time spent, and results. However, other softer indicators relating to the quality of the services had to be devised. We used the following six criteria to evaluate quality of any service:
- Appropriateness – Service is designed to meet the needs of the users or the organisation.
- Awareness – Users are provided with the knowledge of what services are offered
- Availability – The service is there when it is needed or can be used.
- Accessibility – The service is available at place convenient to the users.
- Extensiveness – The volume of service is in relation to the needs and expectancies of the users. Not too much, not too little.
- Equity – Service offered is balanced to provide it to all those entitled to use it.
I once completed the performance audit of a major library. Using the six criteria we concluded that it was situated in the wrong place, open at times when most potential users were at work, and had books that were not what the population would read,
having being selected by the library staff from their own reading habits. Only children with bicycles could make it to the library when it was open. Sadly, the children’s section had little to make it worth the journey. Although the library was very economical and run on a very efficient basis, this library failed all six effectiveness criteria.
1990s – Programme Performance Audits
By this decade many public sector organisations had moved from departmental line budgets to Programme Performance Budgets. These focused on why money was to be spent, rather than what was spent, wages and salaries did not appear in a hospital or university programme performance budget. The activity on which the money was used to produce an output replaced these. Standards and related indicators of achievement were defined at the start of the programme and the evaluation (audit) was performed at the following three stages;
- Before the Programme began, to determine whether the standards and the indicators set for the outputs were appropriate, and information systems were in place to collect the performance information.
- During the Programme to evaluate and conclude how well things were going and recommend changes.
- At the close to evaluate how well or otherwise the Programme performed in meeting its objectives.
Even the Church of England undertook a spiritual audit, following these divine principles.
The objective of the conclusions at the end of these audits, was not to find errors, place fault and recommend penalties. Rather they were intended to identify what was good and what could have been better, thus to help design better programmes in the future.
2000s – Performance Audit of Organisational Knowledge
The 21st century, my fifth decade, saw knowledge as a new major asset in all enterprises. Unlike the trucks referred to earlier, knowledge is often invisible, being in the minds of people, and unless controlled is easily lost or transferable. It’s not in the books at any value, so the financial auditors (readers of this journal) do not concern themselves much with it.
The Knowledge Performance and Knowledge Management Audits required us to evaluate the existence, ownership, use and maintenance of knowledge (information) and know-how (skills and experience) at three levels:
- – Institutional or Organisational Level — strategic and managerial know-how
- – Operational level — departmental and processes know-how
- – Individual level — individual worker knowledge and skills know-how
In some organisations, such as accounting firms, the knowledge flow runs from A to C. In more creative organisations, it runs in the opposite direction. Evidence to audit of the existence and use of knowledge was available from records such as procedure descriptions in operational manuals, service or job profiles, and research finding documents, client information and recorded knowledge acquired on assignments.
This is the easy part of identification, as it is tangible and explicit. However, know-how (experience), in the memory of the individual employee or organisation, is implicit and unless captured is not identified as an asset. Also, implicit knowledge (experience) is not easy to articulate, nor is it often observable in use. To identify experience as an asset required, the use of such techniques is as critical as incident analysis and knowledge mapping.
Some of our conclusions on Knowledge Performance Audits were that organisations:
- did not have systems to capture knowledge and know-how into a knowledge data-base;
- did not have effective systems and controls to maintain and protect knowledge;
- failed to identify blocked or obsolete knowledge;
- could improve the overall productivity in the use of knowledge; and
- missed opportunities to franchise or sell their know-how.
Now most major organisations have “Knowledge Managers” as key members of their management team. Over the next ten years, access to global performance standards, and immediate availability of best practice via the internet, has made knowledge an even more important subject for Programme Performance Auditing.
Performance Audit – The Next Decade
In South Africa, Performance Audit is currently being practised to some extent in the public sector and by internal audit functions. Their aim, we hope, is to provide independent assurance as to:
- the economic, efficient and effective implementation of policy;
- the reliability of the indicators and the published statements about performance; and
- to identify examples of best practice as standards.
Performance Audits such as:
- Environmental Management Audits;
- Energy Management Audits;
- Security Management Audits; and
- Information Management Audits.
may become required, as each of these four resources becomes more valuable and vulnerable. The accounting profession has developed a very reputable process called ‘audit’. All the techniques, (planning, control evaluation, evidencing, sampling and impartial reporting), are used in this wider scope of Performance Audit.
Over the five decades it is has provided me with exciting audits and real contact with the operations and activities at the centre of enterprises.
Colin Sutherland CA(SA), CA, CA(Z), is an international independent consultant.
