This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.
Introduction
The chief advantage of surveys is that they allow the auditor to make aggregate statements on a topic relevant to audit objectives; for example, "Overall, there has been a loss of resources to ..." or, "A quarter of managers had difficulty distinguishing between..."
Two key preconditions for making general statements are the following:
- the use of a sample that allows an unbiased estimate of the situation about which the general statement will be made; and
- the use of a standardized approach and questions to all members of the sample surveyed.
Generalizing to a population
One of the most common purposes of a survey is to allow the auditor to reach a general conclusion (generalize) about a population on a topic relevant to audit objectives. This is a useful approach when the significance of a condition depends upon how widespread the condition is, for example on how frequently certain deficiencies in human resource management practices occur in an organization. When individual instances may in and of themselves be significant, other approaches, such as case studies may be more appropriate. For example, in examining controls over operations in a given program area, there may be a few operations that involve significant physical hazards for the public or employees, warranting individual attention.
In generalizing to a population, it is vital that the survey uses a representative sample of the entire population. Surveys of this type are only feasible where the relevant population of interest can be clearly identified and a means exists to identify individual members of the population for the purposes of sampling. In the case of a small population, it may be feasible to get an unbiased measure by accessing the entire population (census), if identifiable.
When is a survey appropriate?
Surveys of individuals may request a variety of information, such as the following:
- the characteristics of the respondent (e.g. age, position within an organization, education or other qualifications, etc.);
- prior or current actions (what they have done in the past);
- the reasons for actions;
- factual information related to the program (expenditures, management practices);
- satisfaction with policies, programs and practices; and
- views regarding the effectiveness of programs or practices.
The data collection techniques employed in surveys of individuals are similar to audit interviews or management representations (when information is obtained from managers) in that they represent respondents' views regarding facts. Therefore, they may require corroboration to confirm the existence of factual situations. It may be difficult to ask corroborating questions as part of questionnaires and structured interviews. Therefore, additional techniques may be required to obtain corroboration. Where independent corroboration is considered essential, other techniques, such as case studies or document review, may be preferable. In some cases it may be possible to combine a survey approach with other approaches. For example, a survey questionnaire may ask respondents to provide corroborating documentation for review by the auditor.
Public opinion polls and marketing surveys conducted by others have asked respondents about their response to hypothetical or future situations, e.g. what they would buy under given circumstances or how they will vote. It is very difficult to corroborate individual responses to hypothetical situations. Therefore, great caution must be exercised in the use of hypothetical questions in surveys.
Many of the considerations that apply to the design and use of questionnaires, structured interviews and samples in surveying individuals apply to the use of these techniques for other purposes as well. Consideration should be given to consulting the FRL for surveys in these circumstances.
For instance, questionnaires and structured interviews may be used for other purposes than generalizing to a larger population. They may be used to obtain information from program staff, management or clientele in a particular location as part of a case study or series of case studies. Or, they may be used to identify suitable recommendations regarding deficiencies revealed by an audit.
Rather than obtaining information from individuals, survey techniques may also be used to make general statements based on examination of documents, direct observations and items on data bases. For example, a survey of all program evaluation reports produced by departments and agencies over a seven-year period resulted in general statements about the types of evaluation issues addressed within that time frame. The Office frequently surveys information about transactions, such as loans, contracts, payments to personnel, contained in department and agency data bases.
Practical considerations
It is not possible for surveys to provide the breadth and depth of information available from case studies and audit interviews. Consequently, use of a survey to generalize to a larger population requires careful planning and specification of the information to be collected. This can only be accomplished when sufficient knowledge exists to develop a standard set of questions that can be administered in a standard fashion. For these reasons, surveys are best suited when matters of potential significance have already been well elaborated and the survey can be well focused.
Because of the amount of planning involved, surveys require several months from the time planning is started until final results are available, and they can be resource intensive. The development cost will be greater the less well understood the issue area, the greater the difficulty in identifying individual population members for sampling, and the more complex the issue area to be examined.
When to use a survey
Surveys are most useful and efficient when
- significance depends upon the frequency and extent to which a condition occurs in the population being studied
- the population of interest can be clearly defined and individual members identified for the purposes of sampling
- matters of potential significance have been well elaborated so that the survey can be well focused
- independent corroboration of each individual's responses is not required
- several months are available to plan, conduct and analyze the survey
Other approaches, such as audit interviews, case studies, and document review are often better suited when
- significance can be demonstrated by identification of deficiencies in a few important cases
- the issue area is less well understood
- identifying matters of potential significance
- independent corroboration of each individual's responses is required
- time frame for the audit is extremely short (less than five to six months)
Representative Sampling
