Aboriginal Justice Strategy Evaluation, Final Report

3. Methodology

Several lines of evidence have been used to address the evaluation issues and questions, as follows.

3.1 Document Review

A review of publicly available documentation on the AJS was conducted in spring 2011. These documents included reports from Statistics Canada (the Canadian Centre for Justice Statistics, Juristat) and the Department of Justice. Data of significance to the background, rationale and relevance of the AJS were included in the report.

The potential limitations of this information are its completeness and timeliness. Statistical information relies on accurate data input, and some studies cited only included data from a sample of provinces and territories. As there is a delay in the production of statistical reports, most data was not available beyond 2009-10.

A complete list of documents reviewed is included in Appendix E.

3.2 Case Studies

The case studies provided descriptive information as to the relevance, impacts, lessons learned and inspiring practices of the AJS through inquiry with triangulated sources: offenders, victims, AJS program staff, justice-related personnel and community members (such as Elders), in addition to a statistical analysis of reported crime data.

Case studies of AJS-funded programs were conducted as a means of gathering in-depth data on community-based justice programs, including their operations, the challenges they face, and the impact they have on the communities they serve. Many of the impacts of the AJS are not measurable through quantitative data; the case studies provided the opportunity to assess program impact and learn of best practices or lessons learned directly from those involved.

In total, 13 community-based justice programs participated in the case studies in 2010-11, and crime data was obtained for 11 of the corresponding communities. The case studies included a diverse mixture of programs, such as diversion measures, community sentencing, and family and civil mediation. Data collection for the case studies comprised a document review; interviews with key informants, including program managers and staff, program participants, program partners and related organizations, such as justice coordinators, police officers, victim workers, court staff, Justice Committee members, city officials, Elders, prosecutors, legal aid officers and probation officers; and group sessions with program participants, Elders and program staff.

The group sessions used culturally based methods. The Waawiyeyaa Evaluation Tool, developed by Johnston Research Inc., is a self-evaluation method grounded in traditional knowledge and ways of being. The Ojibway word Waawiyeyaa refers to a circular process that can lead to rebirth and transformation. The second method involved the use of culturally relevant iconic images to generate answers to evaluation questions.

The major limitation of the case study methodology is that results from one case study represent unique findings that are not generalizable to all community-based justice programs. This limitation was mitigated by conducting 13 case studies at sites selected to be representative of the diversity of community-based justice programs across all provinces and territories. As well, effectiveness data collected through the analysis of rates of re-offending, key informant interviews and document review all provide means of triangulating case study findings.

3.3 Key Informant Interviews

A total of 25 interviews were conducted by telephone in June and July 2011 as part of the evaluation of the AJS. These interviews were semi-structured and addressed issues of relevance, performance and implementation of recommendations from the 2010 mid-term evaluation of the AJS.

Semi-structured interviews were selected as a methodology for this evaluation to provide support to the findings raised by the case studies and an opportunity for AJS stakeholders to present their perspective on the relevance and performance of the AJS.

To ensure consistency in reporting findings, all interview results were coded and those results that appeared consistently across interviews are reported as noteworthy findings. When possible, responses were triangulated with data from the document review, case studies, and analysis on rates of re-offending.

To mitigate potential methodological weaknesses inherent to semi-structured interviews, all interviews were conducted by a single evaluator familiar with the AJS, who was able to probe for detailed answers. The questions asked during the interviews (see Appendix B) were reviewed to ensure they were not leading, and rather than infer findings from a small sample of interviews, all AJS FPT WG members were invited to participate, as were several key staff from the Department of Justice. Table 2 presents a breakdown of the key informants interviewed as part of this evaluation.

Table 2: Key Informants Interviewed as part of the AJS Evaluation
Category of Key Informant Number
Department of Justice Director General, Programs Branch 1
Aboriginal Justice Directorate Staff: Headquarters 3
Aboriginal Justice Directorate Staff: Regional Office 1
Aboriginal Justice Directorate Staff: Regional Coordinators 6
Aboriginal Law and Strategic Policy Staff 1
Provincial and Territorial Partners Members of the AJS FPT WG 13
Total 25

3.4 Review of Administrative Files and Program Data

A review of AJD administrative files, as well as of files sent from some community programs to supplement the case studies, was performed to support data collected through the evaluation.

3.5 Recidivism Study

The approach taken for the recidivism study was to compare recidivism rates of individuals who participated in an AJS-funded program with those who were referred but did not participate in any aspect of the program. A total of 5,141 cases were referred from the 25 AJS programs that provided participant data formed the sample for this study. The data was collected for the period 1998 to 2007, which allowed for a comparison of recidivism rates over time.

For the analysis, the time elapsed after completing the program until receiving a criminal conviction was statistically modeled as a function of age, sex, number of pre-program convictions (the intervening variables), and participation or not in an AJS program. Survival analysis, specifically the Cox Proportional Hazards Model, was the statistical approach used to model the likelihood of re-offending. This is the same method that was used in the 2006 and 2000 recidivism studies, allowing for comparison over time.

The major limitation of the analysis of rates of re-offending was the lack of true experimental design, as practical and ethical constraints precluded the random assignment of persons to participant and control groups. Thus, pre-existing differences between the participant and comparison groups could lead to differential outcomes with respect to re-offending. To mitigate this strategy, a statistical approach that could control underlying differences between the participant and comparison groups was utilized.

Another limitation is the non-random nature of the data provided for the study. Both provincial and program agreements were reached in order to access program data, which precluded a study of all AJS-funded program participants or of a random sample. To mitigate this limitation, a large sample size was used, and records analyzed as part of this study were randomly selected from the sample provided by programs (using a random number generator), while ensuring representation of all programs that had submitted data.

3.6 Cost Analysis

A cost analysis was undertaken to explore the cost implications of the AJS. To this end, 17 AJS programs located across Canada were asked to provide data on federal and provincial program funding, as well as number of clients served, for the fiscal year 2008-09[6].

This analysis was conducted as a means of estimating the cost efficiency of AJS-funded programs. The mainstream justice system was used for comparison as it is the only alternative for the majority of AJS-funded program participants. The cost analysis was designed to measure both immediate and longer-term (through impacts on recidivism) cost efficiency of the AJS.

Selection bias is a potential limitation to this analysis, as offenders who participate in AJS-funded community-based justice programs are generally referred for relatively minor offences. These types of charges might have led to low mainstream court costs compared to the average, as they are not complex. Although it was not possible to control case complexity in the cost analysis, this bias is countered by other factors not taken into consideration. Especially, the relatively high costs to the mainstream justice system compared to community-based justice programs of serving remote communities could not be measured.

Another methodological limitation is the inability to pinpoint which mainstream justice costs are not incurred as a result of participation in AJS-funded programs. For example, in some cases referrals are made to AJS programs prior to police involvement, which would further reduce costs when compared to the mainstream justice system, while in other cases policing costs would be comparable. Since it is impossible to determine the true value of cost savings in this area, it was excluded from study as a means of mitigating this limitation.

A final limitation is that the program data might not be representative of all AJS-funded community-based justice programs. To mitigate this potential limitation, programs were selected to share data based on their representativeness of the variety of AJS-funded programs. The selected programs are located across Canada, in urban, rural and remote locations, and provide a variety of services to their communities.

3.7 Survey of Police and Crown Representatives

A web-based survey was sent to all RCMP members nationwide, via a link provided in a RCMP electronic bulletin in September 2009, as part of the 2010 mid-term evaluation of the AJS. Crown counsel that work with the communities delivering a community-based justice program were identified by the provincial/territorial partners and were also invited to participate.

At the outset of the survey, respondents were screened for participation based on whether they worked or had worked since 2007 in or near a community delivering a community-based justice program. Those who did not satisfy these criteria were excluded from survey participation.

The aim of this survey was to gauge the level of awareness of mainstream justice partners in AJS-funded programs. As it was not possible to know which RCMP officers had worked in or near which communities and target invitations, the exact number of eligible participants is not known. Therefore, response rates and sampling error cannot be calculated. In spite of this limitation, the methodology of sending an invitation to all RCMP officers was seen as the most effective and efficient means of reaching the full cohort.[7] The survey is presented in Appendix D.

Table 3 summarizes the number of respondents to the police and Crown surveys respectively.

Table 3: Number of Police and Crown Survey Respondents
Key Informant Group Number of Individuals Consulted
RCMP/Police 35
Crown 10
Unknown 1
Total 46

  • [6] One program provided data from 2007-08 as 2008-09 data was not available at the time of evaluation.

  • [7] Response rates and sampling error cannot be determined as the size of the sample frame and population are not known.

Date modified: