Aboriginal Courtwork Program Evaluation
1.1. History of the Aboriginal Courtwork Program
Research in the early 1960s identified particular challenges faced by Aboriginal persons before the court, including a sense of alienation from the administration of justice in Canada, a feeling of futility or apathy, and limited awareness of their rights, their obligations, court procedures, and the resources available to them. Judicial and court officials often failed to understand Aboriginal people. It was noted that simply improving access to legal services did not fully serve the needs of Aboriginal people in the justice system.
In response to these challenges, Native Friendship Centres began to operate Courtworker programs in Winnipeg and Edmonton that provided non-legal advice and support to Aboriginal persons appearing before the court. Federal financial support for these programs was first provided to the Native Friendship Centres in 1969. In 1972, the Department of Justice undertook pilot projects that provided guidance and information to Aboriginal people involved in the criminal justice system. By 1978, the pilot projects were expanded and became the Native Courtworker Program, which was recognized as a permanent federal-provincial/territorial (FPT) cost-shared program. In 1987, the mandate of the Program was revised to include provision of services to Aboriginal youth, following the adoption of the Young Offenders Act.
The Aboriginal Courtwork (ACW) Program currently operates in all territories and provinces, except Prince Edward Island and New Brunswick. The main purpose of the Program is to ensure that Aboriginal people charged with criminal offences receive fair, equitable and culturally sensitive treatment by the criminal justice system. Towards this end, Courtworkers assist Aboriginal persons before the court to understand their rights, responsibilities and obligations, and help them to better understand the nature of the charges against them and the philosophy and functioning of the criminal justice system. Furthermore, Courtworkers work with judicial and court officials on enhancing the awareness and appreciation of the values, customs, languages and socio-economic conditions of Aboriginal people. Lastly, they respond to problems and special needs caused by communication barriers that exist between Aboriginal people and those who are involved in the administration of the criminal justice system.
A formative evaluation of the ACW Program was undertaken in 2007, and a summative evaluation was completed in 2008. Prior to that, no federal government evaluation of the Program had been completed since 1985.
1.2. Purpose of the Evaluation
The national evaluation of the ACW Program was conducted to meet the requirements of the Treasury Board of Canada Policy on Evaluation and the Federal Accountability Act. The study examined issues of continued relevance, performance, and program design and governance. The specific evaluation issues and questions addressed in this evaluation are listed below.
Evaluation Issues and Questions
- Issue #1: Continued Need for the Program
- Is there a continued need for the ACW Program?
- To what extent is the ACW Program responsive to the needs of Aboriginal persons before the court?
- Issue #2: Alignment with Government Priorities
- Are the ACW Program objectives consistent with the priorities of the federal government?
- Are the ACW Program objectives consistent with the Department of Justice strategic outcomes?
- Issue #3: Consistency with Federal Roles & Responsibilities
- Does the ACW Program duplicate or overlap with other programs, policies or initiatives delivered by other stakeholders?
- Is there a role for the federal government with respect to the ACW Program?
- Issue #4: Achievement of Expected Outcomes
- To what extent are the clients aware of their rights and obligations as a result of their interaction with the Courtworkers?
- To what extent do Courtworkers help their clients make informed decisions with respect to their circumstances before the court?
- To what extent do the Courtworkers advise clients of the legal and community/social resources available to them in their community to address their needs?
- To what extent are the clients able to make informed decisions about pursuing alternative measures/restorative justice programs and services as a result of the Program?
- To what extent does the ACW Program assist clients in receiving fair and equitable treatment before the court?
- To what extent do justice officials receive information from the Courtworkers relating to:
- the circumstances of the clients;
- legal and community/social resources available to the clients in their community;
- alternative/restorative justice programs and services available to the clients in their community; and
- cultural traditions and social needs pertaining to the clients?
- How and to what extent has the information provided by the Courtworkers on the circumstances of the clients (question 6) been used by the justice officials?
- How and to what extent have the Courtworkers developed linkages between themselves and the communities they serve; and linkages between Courtworkers and the community-based justice programs?
- To what extent have communication and collaboration among the different service providers that serve Aboriginal clients within the justice system changed? In what ways?
- Has the role of the Courtworker changed over the past five years?
- If yes, what is the nature of these changes?
- Have these changes affected the Courtworkers' capacity to do their job?
- What is working well in the ACW Program?
- To what extent has the Four-Year ACW Project Fund impacted the ACW Program?
- What is not working so well in the ACW Program? What needs to be changed?
- Are there any unintended impacts arising from the ACW Program? If yes, what are they?
- Issue #5: Program Design and Governance
- Are there any differences in Courtworker services needs by Aboriginal men and women before the court?
- To what extent are the performance indicators clearly and consistently reported upon by each jurisdiction?
- To what extent has the TWG had input into the development of the ACW Program performance indicators?
- What are the challenges associated with collecting and reporting standardized performance measures across all jurisdictions?
- Have the changes to the TWG governance in terms of addition of a third co-chair and the creation of a FPT Working Group for the ACW Program increased the efficiency and effectiveness of the TWG collaboration?
- To what extent is the TWG working collaboratively?
- Issue #6: Demonstration of Efficiency and Economy
- Are the most appropriate and efficient means being used to achieve the ACW Program outcomes?
- Does the ACW Program have the resources it needs to achieve its objectives?
- Have all the resources for the ACW Program been used?
1.3. Method of Study
The evaluation utilizes multiple lines of evidence including both primary data sources (interviews with key informants and judicial and court officials as well as surveys of Courtworkers and clients) and secondary data sources (document and file review, and budget and performance data). In total, nearly 1,500 representatives from various groups participated in the evaluation through interviews and surveys. Primary data collection occurred between April 2011 and September 2012.
1.3.1. Interviews with Key Informants
Interviews were conducted with 50 key informants including Justice Canada representatives, provincial/territorial representatives, service delivery agency (SDA) representatives, and other stakeholders including Aboriginal Community Justice Program representatives. Of those, 27 participate in the Tripartite Working Group (TWG). The interviews were completed in August and September of 2012. Sixty-five people were approached to participate; thus, the response rate was 77% (50/65). The number of interviews completed by key informant groups is summarized in Table 1.
|Key Informant Groups||TWG||Other||Total|
|Federal Department of Justice Officials||4||5||9|
The other stakeholders included representatives from Aboriginal Justice Strategy (AJS) and community justice programs. The input of the key informants, particularly those who are closely involved in the design and delivery of the ACW Program, are crucial in addressing evaluation questions, including those related to program alignment with government priorities, effectiveness of program design, and efficiency and economy.
1.3.2. Interviews with Judicial and Court Officials
Telephone interviews were completed with 116 judicial and court officials across all jurisdictions where the Program was in operation at the time of the interviews (April to June 2011). The number of officials interviewed by jurisdiction is shown in Table 2.
|OthersTable note *||Total|
|Newfoundland and Labrador||--||--||1||1||--||2||4|
- Table note *
Others include one acting sheriff, one provincial representative, one Director of Court Services, and one RCMP Liaison Officer.
The interviews obtained feedback on the level of familiarity with the ACW Program amongst judicial and court officials, the need for the Program, impacts and effects, relationship to other justice initiatives, and opportunities for improvement. Those interviewed were drawn from contact lists of 326 potential respondents, providing a response rate of 36%.
1.3.3. Survey of Aboriginal Courtworkers
A web-based survey of Courtworkers was undertaken between April and May 2012. The purpose of the survey was to obtain feedback on the Courtworkers' perceptions of, and involvement with, the ACW Program. Of the then total of 185 Courtworkers, 161 responded to the survey (a response rate of 87%). The number of Courtworkers surveyed by jurisdiction is shown in Table 3. Of those who responded to the survey 70% were female and 27% were male (2% of Courtworkers surveyed did not respond to the gender question).
1.3.4. Survey of Clients
The survey of clients took place between July and September 2011. A total of 19 Aboriginal local interviewers Footnote 4 were hired to collect the Client Survey. Although some of the interviewers were employees of the SDAs, they were not Courtworkers. Rather, they were Courtworker trainers, administrators, communication officers or consultants. The interviewers were familiar with the ACW Program and some spoke the local language. None of the interviewers contacted/interviewed their own clients. The interviewers collected the data by having the clients complete the survey on their own or interviewing the client in person or by telephone. The data were collected from three types of sites: urban areas and remote areas with a resident Courtworker, and remote areas with a non-resident Courtworker. As indicated below, 1,166 clients were surveyed, representing an overall response rate of 82%. The number of clients surveyed by jurisdiction is illustrated in Table 4.
|Jurisdiction||Targeted (#)||Actual Surveyed|
The purpose of the Client Survey, currently conducted every five years, was to assess the clients’ level of satisfaction with respect to the following: Courtworker services, the way their case turned out, and referrals. Clients were asked about their awareness of the ACW Program and their understanding of the information they received from the Courtworkers. The survey also delved into whether the clients needed additional help, and their perceptions of the justice system.
1.3.5. Document and File Review
An extensive review of documents and administrative files was conducted to analyze information on the evaluation issues pertaining to relevance and performance of the Program. The document and file review focused primarily on foundational policy and program documents and files that are pertinent to understanding the Program environment and delivery context. More specifically, the document and file review included but were not limited to:
- Government of Canada and Departmental Documents: Speeches from the Throne, Department of Justice Canada Report on Plans and Priorities, Departmental Performance Reports;
- Program Files: ACW Program annual reports, budget reports, financial expenditure statements, brochures, evaluation reports, and other documentation describing the Program and its objectives, activities and outputs;
- Provincial and Territorial Documents: annual work plans and performance reports from provinces and territories, as well as other administrative files;
- FPT Working Group and TWG Files: work plans, meeting minutes, meeting decisions, list of projects, policies, and research activities;
- SDA Data Reports: aggregate data and files from the SDAs in the provinces, which provided quantitative evidence on the level of demand for ACW Program services and profile of the clients;
- Other files, websites, and reports related to the Department of Justice, Government of Canada, provincial governments, and non-profit organizations.
1.4. Evaluation Strengths and Limitations
The 2009 Treasury Board Policy on Evaluation requires that all direct program spending by the federal government be evaluated every five years. The Department of Justice Evaluation Division has assessed all direct expenditures according to six risk factors: complexity, the extent to which the program or service is complex in nature; materiality, the level of resources involved in the delivery of the program; skills and expertise, the ability of the Department to recruit and retain the necessary work force to fully delivery on their mandate; time since the last evaluation; and the quality of the information to support evaluation. Based on an analysis of these risk factors, the ACW Program is deemed to be a low-risk program.
The strengths of the current evaluation include the use of multiple lines of evidence in order to triangulate findings and increase the data reliability, significant sample sizes incorporating the perspective of all key stakeholder groups involved with the Program, and the use of both quantitative and qualitative data. The methodology for the evaluation, including the data collection instruments, was developed in consultation with federal and provincial partners of the ACW Program and the Evaluation Advisory Committee.
The evaluation encountered several challenges and limitations, including a reliance on qualitative data, differences in how client data is reported across jurisdictions and over time, and jurisdictional differences in design and delivery. Availability of quantitative data on the activities and outcomes of the Program is limited and the methodology did not allow for direct observation of service delivery across jurisdictions. Several measures were taken to reduce the effect of response bias and validate interview results, including (i) the use of multiple lines of evidence, particularly validating findings through other primary and secondary research; (ii) interviewers clearly communicated the purpose of this evaluation, its design and methodology, and strict confidentiality of responses to participants; and (iii) key informants, including judicial and court officials, were asked to provide a rationale for their ratings including a description of specific activities which contributed to the outcomes reported.
All but one jurisdiction collect information on the number of clients served. The remaining jurisdiction collects information on the number of cases. This has made it difficult to assess the Program utilization, and its economy and efficiency. To respond to this limitation, qualitative questions on the Program efficiency were included in the key informant interviews.
The Program is designed and delivered differently across the jurisdictions, reflecting differences in program scope, the demand for services, the roles of the Courtworkers, and availability of other programs and resources. Thus, some observations about the Program (e.g., pressures to expand the scope, service coverage) may apply to some jurisdictions and not to others. Although the Program has been designed to be delivered in a flexible way, methodologically this creates challenges for the evaluation by not measuring entities equally across jurisdictions. As a national evaluation, it has focused primarily on broad issues and trends rather than on jurisdictional differences.
1.5. Structure of the Report
The evaluation report is divided into four chapters. The next chapter provides a brief overview of the ACW Program in terms of its purpose, delivery model, budget and intended outcomes. Chapter 3 provides a summary of evaluation findings regarding Program relevance and performance. Chapter 4 presents the major conclusions and recommendations emerging from the evaluation as well as the management responses.
- Date modified: