Tax Law Services Portfolio Evaluation

3. Methodology

In accordance with Treasury Board evaluation policies, the Department of Justice used a risk-based approach in planning evaluations to ensure the efficient use of evaluation resources. The Department of Justice Evaluation Division assessed the risk level (low, medium or high) of planned evaluations, taking into consideration six risk factors: the contingent nature of program funding; the complexity of the program or service; materiality (i.e., the level of resources involved in program delivery); skills and expertise (i.e., business risks facing the Department regarding recruitment and retention, and the need for specialized skill sets); time since the last evaluation; and information challenges (i.e., whether program information is available and accessible to fully support an evaluation). The TLS Portfolio was selected as the first legal services evaluation due to its overall high risk rating. The methodology developed also responded to the level of risk by ensuring multiple lines of evidence that would support robust findings.

The evaluation of the TLS Portfolio draws on five lines of evidence, i.e. a document and data review, key informant interviews, a survey of legal counsel, case studies, and focus groups. Each of these methods is described more fully below. This section also includes a brief discussion of methodological challenges.

The methodology was developed with the TLS Portfolio evaluation working group. The data collection methods and instruments were all reviewed and approved by the working group.

The evaluation matrix, which lists the evaluation questions, indicators and lines of evidence and is used to guide the study, is included in Appendix A. The data collection instruments developed to respond to the evaluation matrix are in Appendix B.

3.1. Document and Data Review

The document and data review was conducted both to inform the development of data collection instruments and to address the majority of the evaluation questions.

Documents reviewed were obtained from internal, external and publically available sources. Departmental documents reviewed included Departmental Performance Reports; Reports on Plans and Priorities; departmental audit reports; and Client Feedback Survey results from 2008 and 2011 Footnote 5. Internal TLS Portfolio documents and relevant CRA documentation were also reviewed, as well as publically available information, such as Budget speeches and Speeches from the Throne.

In addition to documents, the evaluation involved the review of iCase data from fiscal years 2007/08 to 2011/12. The iCase is the Department's integrated case management, timekeeping and billing, document management and reporting system.

3.2. Key Informant Interviews

The key informant interviews conducted for this evaluation addressed the majority of evaluation questions, and were a key line of evidence in gathering information on the need for the TLS Portfolio, as well as the effectiveness of Portfolio activities. A list of potential key informants was prepared, and interview guides tailored to each key informant group were developed, in consultation with the evaluation working group. Interviews were conducted with a total of 43 key informants representing the TLS Portfolio (n=22) and the CRA (n=19). Representatives from the PLS and the Department of Finance were also interviewed. All interviews were conducted with personnel in managerial-level positions.

Potential interviewees received an invitation to participate in an interview. Key informants who agreed to participate were provided with a copy of the interview guide (in the official language of their choice) prior to the interview. Each interview was conducted in the respondents' preferred official language, and key informants were assured of the confidentiality and anonymity of their responses. This evaluation included a mix of telephone and in-person interviews.

3.3. Survey of Tax Law Services Portfolio Counsel

To gather the input of all TLS Portfolio counsel at headquarters, the LSUs, and in the regions, the evaluation included an anonymous and confidential web-based survey that was delivered through the Department of Justice's intranet site to all TLS Portfolio counsel. In order to encourage responses, the Department sent each potential survey participant an invitation by email describing the purpose of the survey and the importance of their participation. Email invitations included an electronic link that participants could click on to access and complete the survey.

Prior to the survey going "live", it was pretested with 14 counsel to ensure that questions were clear and that it functioned properly (with working skip logic, etc.). The survey was revised based on the feedback provided by those involved in the pretest.

The survey was online for a total of three weeks — from June 15 to July 6, 2012. During this period, two reminders were sent to potential participants to increase the response rate. Invitations were sent to 357 counsel. Fourteen of these potential participants were not available during the entire time the survey was online. In total, 159 counsel completed the survey for a response rate of 45%. Footnote 6 Once the survey was finished, open-ended questions were coded and the survey data was analyzed using SPSS, a statistical software package.

Table 3 provides a profile of survey respondents and shows that generally, respondents were representative of the population of TLS Portfolio counsel. Although the information on the types of files most often worked on is not available for all of the counsel in the TLS Portfolio, the sample appears representative.

Table 3: Comparison of TLS Portfolio and Legal Counsel Survey Respondent Profiles
Characteristics TLS Portfolio Footnote 7 Survey Respondents
What is the classification level of the position you currently occupy? (N=336) Footnote 8 (n=159)
LA – 01 26% 25%
LA – 2A 43% 43%
LA – 2B 14% 16%
LA – 3A 8% 6%
LA – 3B 1% 1%
Senior Management 7% 8%
Total   100%
When did you first join the Department? (n=369)  
Less than one year ago -- --
Between 1 and 5 years ago 28% 31%
Between 6 and 10 years ago 20% 15%
More than 10 years ago 51% 54%
Total 100% 100%
Where do you currently work? (n=336)  
Headquarters (excluding LSU) 13% 12%
Regional office 78% 75%
LSU 9% 13%
Total 100% 100%
What kind of files do you work on most often?Table note * Not available  
Litigation   82%
Advisory   18%
Total   100%

Note: Some totals do not sum to 100% due to rounding.

Table note *

Information not available for all TLS Portfolio counsel.

Return to table note * referrer

3.4. File Review

A review of a selection of closed litigation and advisory files was conducted to allow for a more in-depth understanding of the life of a file in relation to the performance measures for the TLS Portfolio. This method also allowed the evaluation to explore whether the information obtained from key informants on how the Portfolio conducted its work was supported by a review of selected case files.

The file review involved the examination of 45 files, including 32 litigation files and 13 advisory files. Multiple court levels were reviewed for 16 of the litigation files, so the evaluation could consider changes in risk and complexity rating, counsel assigned to the files, and level of consultation as the case moved through the levels of appeal.

The sample of files was chosen with the input of the evaluation working group and was considered to provide a good representation of the broad spectrum of the TLS Portfolio's work. As files were not chosen by random selection and as the sample is not large (considering the thousands of files worked on by Portfolio counsel during the time period covered by the evaluation), the file review sample is not a strictly representative one. Rather, the file review was intended to be illustrative of the Portfolio's approach to its work.

Two counsel from the Portfolio conducted the file review in order to protect confidential taxpayer information as well as solicitor–client privilege. To ensure that comparable information was collected from the files, counsel completed a standard file review template that was developed for the study (see Appendix B). The template collected information to respond to the evaluation matrix and focused on factual information available in the files.

3.5. Case Studies

Three case studies (two litigation cases and one advisory case) were conducted to allow for an exploration of best practices and the TLS Portfolio's national approach. As such, cases selected were higher-profile files (i.e., those involving complex or high-risk issues, and those involving a high degree of collaboration among Portfolio offices and/or between Portfolio counsel and CRA colleagues).

For each case study, file review templates (completed as part of the file review conducted for the evaluation) were reviewed. In addition, telephone interviews were conducted to supplement documented information, to provide context for the work, and to allow for a more in-depth assessment of how the file was handled and the effectiveness of the working relationship between the Portfolio and the CRA. Interviews involved TLS Portfolio counsel and CRA representatives who worked on the file that was the subject of the case study (including both headquarters and regional representatives from each organization).

Interviews were conducted with a total of 16 stakeholders. Interviewees took part in either an individual or small group interview. The approach used to schedule and conduct the interviews with case study participants was the same as the approach (described above) for scheduling and conducting key informant interviews.

3.6. Focus Groups

Focus groups were the final line of evidence used in this evaluation. Two focus groups were conducted (one with CRA regional representatives and one with TLS Portfolio regional representatives) after the other lines of evidence were completed. These focus groups were used to follow up on emerging findings and obtain additional details and insights about issues identified by the other lines of evidence.

The focus groups each included seven participants representing four regional offices (British Columbia, Ontario, Quebec and Atlantic). The CRA focus group participants were litigation officers or litigation managers. The TLS Portfolio group involved TLS Section counsel. Focus group participants did not occupy management-level positions; rather, the participants' primary role was handling tax law litigation files.

The CRA focus group was conducted by teleconference. The TLS Portfolio focus group was conducted by videoconference, using the videoconferencing capabilities of the Portfolio. Both focus groups were recorded to ensure accuracy of the notes, but participants were assured of the confidentiality of their responses.

3.7. Limitations

The evaluation faced a few methodological limitations. These are listed below by line of evidence.

Document and data review: iCase limitations.

Overall, iCase was a useful source of information for the evaluation. There were, however, some limitations.

  • The rating attached to legal risk and complexity changed in 2009/10. Prior to 2009, lead files would have been assessed and given an appropriate risk and complexity rating. All related files were given a risk level of 1 (low) and complexity level of low. After 2008/09, the TLS Portfolio, to be consistent with other portfolios, changed the risk and complexity levels of all related files to match those of the lead file — a process that impacted approximately 5000 files in both 2009/10 and 2010/11. This change means that, as of 2009/10, both the lead file and related files carry the "true" rating, thereby limiting the evaluation to three years of comparable data (2009/10, 2010/11, and 2011/12). For this reason, an analysis of the trends for legal risk and complexity is premature.
  • As iCase is an information management system that is not constructed with evaluation needs in mind, its data on dispute resolution (DR) strategies is limited to indicating the use of DR and does not support an analysis of its success in resolving disputes.
  • Currently, data to support cost-benefit analyses, such as the cost of legal services relative to the tax recovered on a file, are not collected.

Survey of legal counsel.

As with all surveys, the survey with legal counsel will be affected by self-selection bias, which is bias based on who responds and who chooses not to respond. Despite conducting a pretest to ensure relevance, clarity and ease of response, as well as best efforts to build a response rate, less than half (45%) of counsel completed the survey. However, a comparison of the respondents to all of the TLS Portfolio legal counsel on a few key characteristics (LA classification, years in Department, location of work, type of files handled) shows that the respondents were fairly representative of the Portfolio's legal counsel.

Survey of legal counsel, interviews and focus groups.

The survey, interviews and focus groups with key informants and case study participants have the possibilities of self-reported response bias, which occurs when individuals are reporting on their own activities and so may want to portray themselves in the best light, and strategic response bias, whereby the participants answer questions with the desire to affect outcomes.

File review.

In any given year during the evaluation (2008/09 to 2011/12), the TLS Portfolio actively managed more than 13,000 files. To obtain a random sample with a reasonable error level would require reviewing hundreds of files, which was not feasible. Instead, the evaluation relied on the opinion of TLS Portfolio members to select files that they believe reasonably represented their work.

Mitigation strategy.

The mitigation strategy for the above methodological limitations was to use multiple lines of evidence that seek information from the TLS Portfolio and CRA, management and "front line" staff, file review and more comprehensive administrative data review (iCase). The mitigation strategy also included using both quantitative and qualitative data collection methods to answer evaluation questions. By using triangulation of findings from these different sources, the evaluation was able to strengthen its conclusions.

Date modified: