Legal Aid Program Evaluation, Final Report
This section of the report describes the methodology used to conduct the evaluation of the LAP.
The evaluation framework was based on the 2009 Legal Aid Program Impact Evaluation Design and the 2007 ARAF for the LAP. The approach was developed in consultation with an Evaluation Advisory Committee consisting of federal and provincial representatives that served as a technical reference group for the evaluation.
The evaluation methodology consisted of a document and file review, key informant interviews, and site visits with four jurisdictions. Data collection instruments used for the evaluation are included in Appendix A. Triangulation was used to verify and validate the findings obtained through these methods and to arrive at the overall evaluation findings.
3.2.1. Document and File Review
The document and file review provided contextual information on the LAP and responded to the evaluation questions on relevance, effectiveness, and efficiency and economy. The review began with a scan of documents on the LAP, legal aid in Canada, and the legal aid plans. The evaluation benefitted from the availability of administrative data, such as data found in annual reports of the legal aid plans as well as secondary sources. An on-site review of FPT PWG and LAP files was conducted early in the evaluation. The document and file review included the following categories of documents:
- federal program documents, including FPT PWG files from 2006-07 to 2010-11, LAP terms and conditions, and contribution agreements/access to justice agreements;
- documents from legal aid plans, such as annual reports, business plans, and evaluations/research studies;
- Justice research and evaluation reports; and
- statistics from secondary sources from 2005-06 to 2009-10, including the Canadian Centre for Justice Statistics' (CCJS) Legal Aid Survey, the Uniform Crime Reporting Survey, and the Integrated Criminal Court Survey.
3.2.2. Key Informant Interviews
In-depth key informant interviews provided insights into evaluation questions on the relevance of the LAP and its performance, particularly its effectiveness and efficiency in achieving its intended outcomes of enhanced capacity to provide legal aid. For this evaluation, interviews were the main line of evidence and were used to obtain the opinions of those directly involved in the components of the LAP and other relevant stakeholders on the relevance and performance of the LAP, as well as context for understanding the quantitative data on legal aid.
A total of 36 individuals were interviewed, including:
- P/T representatives on the FPT PWG (n=7);
- Legal aid plan representatives on the FPT PWG (n=10);
- Justice representatives, including members of the Legal Aid Directorate and Programs Branch (n=12); and
- Other federal representatives: CIC (n=5), IRB (n=1), and Public Prosecution Service of Canada (PPSC) (n=1).
The interviews were conducted using semi-structured interview guides that included pre-determined, open-ended questions. The guides are attached in Appendix A. Key informants received the interview questions in advance, so they could provide considered responses. All members of the FPT PWG were interviewed either as a key informant or during the site visits (see Section 3.4).
Most key informants could not respond to questions concerning all five components of the LAP; for example, only the jurisdictions/legal aid plans that provide I&R legal aid or have had experience with PSAT cases could respond to those questions. Across the key informants, all components were covered.
The Legal Aid Directorate identified an initial list of potential interviewees. All individuals on the list received an initial communication from the Directorate that explained the purpose of the evaluation and invited them to participate in an interview. The interviews were conducted by telephone in the preferred official language of interviewees.
To respect the anonymity of respondents, interview results are usually reported in aggregate rather than by category of respondent.
3.2.3. Site Visits
The evaluation included site visits in Alberta, Manitoba, Ontario and Nova Scotia. From these visits, additional information on these legal aid plans was collected, thereby enabling the evaluation to provide examples using legal aid plan data for some key evaluation indicators. The site visits also included in-person interviews, additional to those described in Section 3.3, which provided information on the relevance of the LAP and its performance, particularly its effectiveness and efficiency in achieving its intended outcomes of enhanced capacity to provide legal aid.
The site visit locations were chosen based on the consent of the participating legal aid plans and provincial representatives and in consultation with Justice's Legal Aid Directorate. The sites provide geographic representation by including two eastern and two western jurisdictions as well as both large and smaller legal aid plans. They also represent a variety of delivery models: Alberta uses mostly a judicare model, where legal aid is provided by private lawyers on a certificate; Manitoba is a mixed system, with both judicare and staff lawyers; Ontario uses private lawyers as well as a legal aid clinic model; and Nova Scotia uses mostly staff lawyers.
Each site visit included interviews and a data review and lasted one to two days. To minimize the burden on legal aid plans, the data review involved discussing a list of potential data needs based on indicators in the evaluation matrix. The plans then provided information in the format most convenient for them. Interviews were held with representatives of legal aid plans and the provincial government, including the FPT PWG members from that jurisdiction. FPT PWG members were asked to identify additional legal aid plan and provincial representatives for interviews. Some formal interviews were conducted, as well as informal consultations on data availability and access. The distribution of interviews is listed below:
- Management staff of legal aid plans (including FPT PWG members, for a total of n=17); and
- Provincial representatives (including FPT PWG members, for a total of n=5).
The interviews were conducted using the same semi-structured guides that were developed for the key informant interviews. For those interviewees who could only address data needs, interviews were less formal and focused on issues related to data availability. For reporting purposes, the results of site visit and key informant interviews are combined.
3.2.4. Interviews with Criminal Justice Professionals
The evaluation also included interviews with criminal justice professionals (Provincial Court judges, prosecutors and defence counsel) in the four site visit jurisdictions. Interviews with criminal justice professionals were conducted to obtain information from those directly involved in criminal cases on whether there have been changes in the proportion of unrepresented accused, the causes for any changes, and the impact of unrepresented accused on the criminal justice system. The interviews gathered qualitative information on the role of legal aid funding in the accessibility of legal aid, which is a key outcome measure for the LAP.
The evaluation targeted four individuals for each category, for a total of 48 interviews. In one jurisdiction, Provincial Court judges declined to participate, while a fifth defence counsel interview was conducted in another jurisdiction. As a result, 45 interviews with justice professionals were completed as part of the evaluation. Table 5 presents the distribution of the completed interviews.
|Category||Province A||Province B||Province C||Province D||Total|
Protocols were developed for each group and included an initial letter from the Legal Aid Directorate describing the study and seeking any necessary permission. Chief Judges identified Provincial Court colleagues willing to be interviewed, provincial justice ministries and/or Head Crowns identified prosecutors, and legal aid plans identified defence counsel. The interviews were conducted by telephone and used the semi-structured interview guides that are in Appendix A. Interviews held with criminal justice professionals are reported separately from site visit and key informant interviews.
The evaluation faced some methodological limitations. These are listed below, including any mitigation strategies undertaken.
Few lines of evidence were available. The evaluation relies on findings from available documents and the interviews. The possibility of obtaining legal aid plan and court data to support comparison of costs and outcomes of criminal cases with a legal aid certificate versus criminal cases, for which applicants have been denied a certificate based on financial ineligibility, was explored but was not feasible within the budget and timelines of the evaluation. As a result, qualitative evidence remains the primary source of information on legal aid outcomes.
To mitigate this limitation, although still qualitative data, the evaluation incorporated interviews with criminal justice professionals (judges, Crown prosecutors and defence counsel) to provide information on frontline experience with unrepresented accused. In addition, the evaluation conducted a comprehensive review of legal aid annual reports and the national compilation of legal aid data, incorporating this quantitative information where relevant.
Consistent national legal aid data are not available. The data in the national Legal Aid Survey compiled by Statistics Canada have several limitations. Most significantly, jurisdictions report on some information using different measures, and others cannot support the data collection necessary for inclusion in the national reports.
The site visits included efforts to obtain additional information that would be comparable across the site visit legal aid plans. However, it was beyond the scope of the evaluation to require plans to produce information that was not readily available. As a result, the information obtained was not entirely consistent between sites. Given the variety in the delivery methods across the country, consistent legal aid data may not be possible in every circumstance, but some issues could be addressed by developing agreed upon performance measures that can be tracked over time. This related but separate issue is addressed next.
Performance data are not being consistently recorded and some indicators are not well defined. In the 2007 ARAF, Justice concluded that "improved performance reporting is necessary to enable a better assessment of the outcomes of the legal aid program and the adequacy of the federal resources allocated to it." Steps identified in the ARAF for improving performance measurement included developing performance measures agreed to by the FPT PWG, and federal funding to support ongoing data collection and analysis. These steps should enable future evaluations to have more consistent and available quantitative data.
Currently, the identified performance measures seem fairly straightforward (e.g., number of people served) but can be complicated by different methods of measuring service. For example, some plans track number of individuals served, while others track units of service, which may mean that one individual receives multiple units of service. Other measures are not defined or are unclear, such as "cost avoidance", "quality of agreements", or capacity to deliver. Some measures are not well aligned with the intended outcomes. For example, an intermediate outcome that legal aid is provided to eligible persons includes indicators related to satisfaction and quality of service. These indicators go beyond the basic provision of legal aid and would mean evaluating the legal aid plans' service delivery, which is outside the scope of this evaluation.
Finally, potentially useful measures that could provide a more complete picture of whether the federal contribution is enhancing service capacity were not readily available at the time of the evaluation, such as the complexity of legal aid cases or the effects of federal legislation on the demand for and cost of legal aid. Making these measures available would benefit future evaluations, and any review of the LAP performance measurement strategy should be mindful of this limitation.
Identifying provincial contacts beyond the FPT PWG members. The interviews relied extensively on FPT PWG members, which limited the range of opinion to stakeholders with a direct interest in the evaluation findings. The site visits were the mitigating strategy to obtain feedback from additional individuals with knowledge of legal aid in their jurisdiction. Although these visits were successful in obtaining information from additional contacts within legal aid plans, additional provincial representatives with sufficient knowledge to address the evaluation questions could not be identified. Therefore, the range of opinion for key informant interviews is potentially limited.
- Date modified: