Evaluation of Public Legal Education And Information: An Annotated Bibliography

2. Academic Articles

2. Academic Articles

2.1 Articles on PLEI

The following articles, drawn from Canadian and American legal journals, span a wide range of initiatives and commentary and come from a broad-based conception of PLEI.

Barry, M. M. (1999). Accessing justice: Are pro se clinics a reasonable response to the lack of pro bono legal services and should law school clinics conduct them? Fordham Law Review: Special Issue, 67, 1879-1926.

General Overview

This article lies slightly beyond the scope of PLEI delivery and its evaluation in Canadian organizations and clinics. The object of the paper is to examine the gap that exists for individuals who are acting on their own behalf in a legal dispute and address the question of whether the needs of this group can be served by clinical programs established through US law schools. Similar to the configuration in various provinces, many of these clinics offer a blend of legal representation and dissemination of legal information, while some are directed exclusively at one or the other. Based on the presupposition that self-represented clients, most often from a marginalized or disadvantaged position in society, require access to both legal substantive and procedural materials as well as opportunities for legal representation, the definition of "pro se clinics" used by the author is any organization whose mandate is to provide "general information about the law, procedure, and practice to a group of litigants or prospective litigants who share a common category of legal issues. The idea is to provide sufficient information to allow participants to understand and access the type of pleadings required, basic rules such as service of process, basic information that the court will require to render a decision, and a sense of the range of remedies available. The term…has been used to describe programs that provide some or all of this information" (p.1883). Within this framework, community legal information is credited with the potential to "create a rising tide of litigants who will gain insight sufficient to demand that their needs be served more effectively" (p.1926). The author walks through a description of types of services offered by these clinics across the United States (some of which are very innovative) to assist prospective litigants. This includes information and form-based kiosks located at courts, law student programs, and various roles filled by paralegals and attorneys. The author looks specifically at two types of educational programs. However, the discussion of evaluation is centered on what law students stand to gain, as opposed to the clients to whom services are directed. The impact and benefit identified for participants is addressed on a very basic level.

The Challenges of PLEI Evaluation
  • One recurring challenge for PLEI is the barrier of language. This includes consideration for non-English speaking clients as well as the legalistic nature of how and what is being conveyed. Aside from ensuring that legal information was available in the native languages of clients, lawyers and staff who are making presentations to community groups must be encouraged and reminded to speak in general terms. This challenge extends to PLEI evaluation. If participants are unfamiliar with the terminology used, and therefore unable to distil information that is relevant to their particular situation, they will be unable to assess its usefulness. Moreover, where feedback is not offered in the participant's first language, a common result is one-word descriptors that do not sufficiently address the content or format of the program.
  • When information is provided in a "mixed model" context it may be difficult to distil a meaningful assessment from clients. The line between information and legal advice is a hard one to draw and there is a greater risk that feedback will be purely outcome driven. Although this is a valid response from clients who were seeking a precise legal remedy, this type of feedback will not be as useful for improving the type or content of material being provided.
Evaluation Methodologies
  • In one clinical program in Florida written questionnaires were distributed after training sessions and phone interviews were conducted three to six months after completion. The primary aim was to collect ratings about convenience and format of the sessions, as well as the amount of social support that was made available. In addition to clarity and overall satisfaction participants were invited to comment on any shortfalls of the sessions. Concerns expressed related to complexity of court proceedings and general intimidation of the process.
  • Most of the other programs appraised in the article measured their success using activities indicators i.e. how many people attend the clinic sessions, how many calls are received/processed annually, and how many clients obtained a divorce (or any other result they were seeking).

Bowal, P. (1998). A study of lay knowledge of law in Canada. Indiana International and Comparative Law Review, 9, 121-141.

Not annotated.

This statistical analysis is relevant reference material for access to justice issues in general as it is not limited to PLEI for low-income persons. The study illustrates the lack of legal knowledge among university-educated Canadian citizens and points to the media, and predominantly American media, as the greatest source of (mis)information.

Brustin, S. (1993). Expanding our vision of legal services representation: The hermanas unidas project. American University Journal of Gender and the Law, 1, 39-59.

General Overview

This article describes the creation and subsequent growth of the Hermanas Unidas project in Washington D.C. The project is not a typical "legal" community legal education ("CLE") program, but more of a community empowerment project that was introduced by the author and modeled on similar projects in Mexico and throughout Latin America. The project primarily involves immigrant Latina women who have enlisted the services of lawyers at AYUDA, a traditional legal aid clinic, regarding domestic, immigration and employment matters. The core group of women who were initially involved in the project have now become "leaders" or "directors" who shape the direction of the project and help other Latina women in the community "move ahead" and overcome systemic barriers that lead to legal actions. This type of community empowerment project seeks to address shortcomings of the traditional legal system by providing support networks though which disenfranchised individuals can share experiences. The method emphasized for achieving this goal is education, both between members and through outreach programs, so that common problems do not escalate to the point of litigation. By looking at the range of concerns – economic, emotional and legal – the members recognize that many problems faced by this community are collective, socio-political problems as opposed to discrete legal issues. The legal system, including legal aid clinics that are positioned to address the needs of disadvantaged groups, generally overlooks issues of intersectionality, whereas this new model for community legal education actively includes this analysis in defining goals and objectives. Empowering these women though education and information-sharing has proved to be an effective and successful way to build the confidence necessary to seek resolutions without total dependence on professionals. The holistic approach of the project simultaneously addresses economic and emotional needs that are beyond the scope of the legal system, while alleviating, if only marginally, the constant demand that plagues the legal aid system. Given the nature of the project, referred to by the author as "grassroots legal education," the lawyers involved play a facilitative, rather than an advisory role to the group. Accordingly, these individuals do not seek to evaluate the progress, content or benefits of the programs. Moreover, if members of the project ceased to derive any benefit from its activities, new directions and ideas would likely be canvassed by both lawyers and community participants.

Connections between Goals of PLEI and its Evaluation
  • The self-generated nature of the project means that needs assessment and evaluation are seen as organic to the process of the organization. Any external evaluation, even by the facilitators, is ideologically opposed to the central objective of empowerment, as its goals would be governed by "outsider" perspectives. Therefore, it is essential to allow members to create agendas and resolve conflicts, which inevitably involve self-evaluation.
Challenges of Funders' Evaluation Requirements
  • By situating this CLE project outside the ambit of general ‘legal services,' they are removed from competitions for government funding that is earmarked for that purpose. This allows the project to avoid the notion of attorney as authoritative problem-solver and decision-maker.
Evaluation Methodologies
  • Although it would be possible to assess the numbers served and obstacles to access in the same way as other CLE programs, any such quantification would bypass the non-legal concerns that participants are attempting to address. Once again this runs counter to the holistic approach which presupposes a direct link between the non-legal concerns of individuals and the impact these have on legal decision-making.

Eagly, I.V. (1998). Community education: Creating a new vision of legal services practice. Clinical Law Review, 4(4), 433-484.

General Overview

Part I of this paper explores the history of legal services and the role of community legal education. Starting with the well-known article by Stephen Wexler which advocated "rights" education for the poor, the author traces various theories and practices that have been developed to address legal needs of low-income citizens. This culminates in an exposition and analysis of the "new poverty scholarship," in which PLEI (CLE as it is referred to in the United States) plays a significant role. Part II describes one community legal education program based in Chicago that centers on legal employment issues faced by immigrant women. Five different education models were used: (i) media campaigns to publicize workplace laws; (ii) educational materials (written and video); (iii) training for social service providers; (iv) workshops for community members; and (v) a workplace rights course. The article then evaluates the community legal education approach to legal services by analyzing the programs delivered and assessing the effect on participants. The different educational models called for different types of evaluation. Although this observation is somewhat obvious, the article clearly illustrates why the nature and the objective of each type of CLE project requires varied methods of evaluation.

The Challenges of PLEI Evaluation
  • Differences in class, race, educational levels and social contexts between PLEI providers and participants increases the difficulty of evaluating whether programs for targeted populations are effective.
  • A recurring issue with PLEI evaluation is the diversity of benefits gained, many of which cannot be measured by traditional quantified research. Yet, it is important not to underestimate the value of these immeasurable benefits. For example, facilitators found that hearing about other women's experiences who had faced similar challenges and suffered similar violations was extremely significant, especially for populations like domestic workers who tend to be isolated. Therefore, the challenge remains regarding how to document these immeasurable benefits.
Connections between Goals of PLEI and its Evaluation
  • Defining specific goals for legal education projects directly impacts the mode of delivery and the orientation of the project, including how it is to be evaluated. For example, it is crucial to know exactly who is being targeted so that evaluation can concentrate on whether an appropriate method of delivery was utilized for reaching that particular group. (e.g. if the program is for women who cannot read or speak English, an assessment should identify whether an appropriate mode of communication is being used to accommodate that characteristic).
Evaluation Methodologies
  • Assessing a program's effectiveness often comes down to stories of individuals and how they themselves identify changes in their approach or transformations in actual situations. Volunteering this feedback usually includes how participants have been able to assist others either through peer education or by making connections so others can get assistance or seek it out themselves. One informative method is to see what is done with the acquired knowledge and how skills have been employed i.e., do they form a group on their own and/or share their knowledge with others, or have they taken steps to transform a problematic situation.
  • Getting people to participate in "active learning" during the program can also help to assess what has been gained. This exercise is especially useful during longer educational programs. Relatedly, mini-evaluations were found to be beneficial when administered part way through a longer program as the participants still had a stake in the course and could participate it making it relevant to their needs.
  • Effects can also be evaluated by asking participants to take part in future activities. By watching what experiences are shared with others, project managers can see the impact directly and make adjustments to programs where necessary.
  • At the completion of a program "action plans" were created by participants where they outlined how they intended to use acquired information. By these means coordinators can determine what knowledge and skills participants identify as being most useful.
Useful Materials for PLEI Evaluation

Once appropriate adjustments are made for methods and objectives, the following headings used in Eagly's study could be used to guide thinking about PLEI evaluation. In this project, individuals' stories were used by coordinators to analyze major effects that programs had on the lives of participants.

  1. Reaches populations not served by traditional legal services
    1. Does the program take place in a location that is known and trusted
    2. Should ask directly about: (i) prior engagement with the law; (ii) knowledge acquired in program; (iii) plans to use in future situations/how will this change your course of action
  2. Provides support for individuals involved in litigation
    1. Is there a noticeable increase in interest in concurrent or subsequent litigation.
  3. Responds to concerns that cannot be redressed by the legal system
  4. Allows individuals to develop their own problem-solving skills
  5. Develops leadership skills
  6. Raises consciousness

Hunter, R. & Genovese, A. (2000). Qualitative aspects of quality: An Australian case study. University of British Columbia Law Review, 33(2), 319-342.

General Overview

The focus of this study centers on the provision of family law services in the legal aid context. The quality of services provided is examined from the view of legal aid lawyers based on their understandings of quality. In approaching this assessment the authors make two significant observations. First, they acknowledge that in the era of significant cost-cutting quality assessment may be "too fussy" since it is generally agreed that some representation is better than none at all. This attitude is also prevalent among PLEI providers. As noted in the introduction, the lack of sufficient resources often means evaluation is relegated to a low priority, especially as against the provision of PLEI programming and materials. Second, the authors denote that "reliance on cultural norms [may be] a more efficient and effective means of quality assurance than reliance on externally imposed, bureaucratic quality criteria and procedures" (p.325). This sentiment also applies to the relationship between funders and core PLEI providers. Although there are standard concerns about objectivity, PLEI providers like lawyers are in the best position to set realistic quality standards and are no less fit to define and assess quality than various funding agencies.

The Challenges of PLEI Evaluation
  • At multiple points in the article, the authors note the importance of considering evaluation criteria "locally." This is emphasized due to the differences found from one practice area to another vis-à-vis quality standards. This observation may be relevant in the Canadian PLEI context as between criminal and civil streams and in accounting for different audiences.
  • The main obstacle in undertaking this evaluation of quality assessments in general was the lack of consensus regarding 'adequate' versus 'good' services among lawyers. In an attempt to address this issue a number of tools were suggested. While this failure to agree on consistent standards is very likely to emerge on the national or provincial PLEI stage, the tools discussed in this paper would not provide much assistance given their strong origins in the institutions of legal aid. For example, the accreditation system was regarded as a possible source of external standard for quality assurance. However, PLEI organizations are not, and arguably should not be subject to a system of accreditation. Also, unlike traditional representation where the court can provide an external, knowledgeable assessment, there is no reciprocal institution by which the merits of a PLEI program may be measured. Finally, unlike legal aid lawyers who are often engaged in an adversarial relationship with other lawyers, PLEI providers do not need to work with other organizations (though they often benefit from doing so), and therefore there is no apparent need for peer review as there is with practitioners.

Connections between Goals of PLEI and its Evaluation

  • In attempting to correlate the objectives of legal aid with the assessment of quality of services provided, it is important to adjust the methods of evaluation to match the mode of services provided.
  • The article notes that evaluation must always consider the substance of what is provided as well as client satisfaction, but that emphasis will vary depending on who is asking the questions i.e., is it the program providers themselves or a funding agency.
  • In any evaluation there is value in exploring what is being done elsewhere i.e., in the same legal areas and among different community groups in the same location.
Evaluation Methodologies
  • Client surveys
  • Interviews with lawyers, both specific to family legal aid and otherwise
    • The questions in the interviews were formulated around four topics: structure, process, input and output
  • File review/analysis

McDonald, S. (2000). Beyond caselaw: Public legal education in Ontario legal clinics. Windsor Yearbook of Access to Justice, 18, 3-59.

General Overview

The first part of this paper provides a synopsis of the historical and current state of PLEI in Ontario and explains the need for PLEI in the context of the liberal legal model. The author then canvasses the emerging "new poverty law scholarship", which advocates a shift away from the traditional lawyer/client model. After explaining the rise of the plain language movement and setting out the basic goals of legal literacy, she argues that PLEI must go beyond these tenets to challenge the substance and procedures of the legal system. The Ontario PLEI projects are placed within a framework of three general PLEI models and it is posited that within the province PLEI still falls under the liberal legal model, especially insofar as it's tied to state funding and aligned with accessibility of the traditional legal system. Current methods of PLEI are comprised of information dissemination that employ a didactic approach. The information tends to be content and process oriented, with the absence of challenge or critique of that information and little or no audience participation. In examining the shortcomings of this approach, the author advocates for pedagogically appropriate education that will achieve legal literacy by "developing capacities to use law and rights as a political resource and to gain skill and power needed to effect positive change" (p.35). An analysis of PLEI activities was undertaken by way of document review (funding applications) and interviews with PLEI providers. The discussion involving evaluation was tied to general goals of PLEI, which was the primary characteristic that differentiated among the three classifications/models. Synthesis of the data collected clearly demonstrated the importance of coordinating PLEI evaluation given the limited resources available and the opportunity to learn from the identified mistakes of others. The author found that most providers included assessment or review of effectiveness of PLEI activities in the stated goals of the projects.

The Challenges of PLEI Evaluation
  • Where the primary goal of PLEI is defined as expanding accessibility to legal information the task of measuring these increases proves to be very difficult.
  • Whereas short-term goals such as timely dissemination of information relating to a piece of legislation may be easier to assess, longer-term goals such as empowerment, improving the legal system and assisting people with the complexities of the legal process are often immeasurable.
Connections between Goals of PLEI and its Evaluation
  • If education is viewed as a two-way street, part of evaluation can concentrate on what the "teacher/coordinator" is learning about the community being served.
  • Insofar as PLEI (and especially self-help materials) is understood to be integral to the provision of legal services, and not opposed to or as "filler" for representation, evaluation should be done as part of a larger "strategy" assessment.

Challenge of Funders' Evaluation Requirements

  • The funder's perspective on PLEI evaluation, as derived from the funding applications, was largely focused on activities indicators such as time spent on PLEI activities.
  • Accountability was sought through requisition of goals, though it was left up to PLEI providers to indicate what significant activities were produced and there was no requirement that these included PLEI projects.

McDonald, S. with Cross, P. (2001). Women's voices being heard: Responsive lawyering. Journal of Law and Social Policy, 16, 207-239.

General Overview

Within a theoretical framework, this paper describes an extensive PLEI project that targeted immigrant women who had experienced domestic violence. The project brought together various community groups and academics to engage in a collaborative process of client-centered PLEI. Although the multi-layered needs of this target group were recognized, the emphasis on education accommodated the objective of sustained empowerment. The project offers a good example of the proactive and creative approach to PLEI design and delivery that was proposed in the background papers to the McCamus Report (see Charendoff, Leach & Levy in McCamus, 1997). The need for PLEI evaluation decreases, and the type of evaluation required changes if the model developed and employed in this study were to be adopted. This is because the participants are integrated into the process of creating and defining their PLEI needs. As long as "clients" are involved at this level, evaluation processes are built into the programs and will shift over time ensuring that objectives and goals are being met in the various manifestations of PLEI provision. It is important to note that this potential result requires the use of methods such as feminist participatory research and the collaborative efforts of multiple groups of "experts" as well as participants. The authors raise a salient point about gaps in evaluation in emphasizing the need for evaluation to consider how a program is delivered in order to assess the totality of effectiveness. Although a few exceptions are noted, generally needs assessment is purely content-focused, which overlooks the crucial factors of audience and context. Finally, the authors suggest that evaluation should consider whether a PLEI initiative has increased the "options" that are available to participants. The mandate of most PLEI projects is to provide information that clients need or would find useful, and is thus inherently client-centered. Accordingly, it is indicative of success to assess whether the student/client/participant has been introduced to more tools and resources than they were aware of prior to program delivery.

The Challenges to PLEI Evaluation
  • Authors acknowledge the difficulty in quantifying the benefits of education noting "learning can help address systemic barriers to equality and social change, as well as prevention, the results of which are never immediate and often cannot be evaluated" (p. 236).

Report of the working group on assessment of systems for delivering legal services. (1999). Fordham Law Review: Special Issue, 67, 1869-1878.

General Overview

This paper compiles a list of recommendations for assessing and evaluating legal services, and outlines the discussions that led to them. The recommendations are categorized under eight headings and seek to answer a number of key questions. No clear distinction is drawn in most of the recommendations between direct, one-on-one legal aid services and broader "volume-oriented" approaches such as hotlines and community legal education. The consensual starting point among this diverse group was that assessment and evaluation are both important and necessary in order to ensure the dissemination of quality legal services and information. The intention of the recommendations is to offer an overarching approach to assessment as well as some specific guidance that will be useful to a variety of organizations that fund or provide legal services. The paper presents a thorough synopsis of general areas that should be addressed in PLEI evaluation as well as some of the challenges involved. Whether funders are looking for accountability or organizations are seeking to ensure that programs are meeting articulated goals, these recommendations may be appropriated and tailored to specific programs to conform with whatever type of resources are being devoted to evaluation and assessment.

Useful Materials for PLEI Assessment

The report is divided into the eight questions reproduced below. The recommendations and commentary from the working group are paraphrased and edited to highlight the key points discussed.

  1. What is the purpose of assessment?
    1. Assessment is value-laden and as such, values must be articulated up front.
    2. Evaluation is necessary for achieving quality and looking at the efficacy of the efforts expended.
    3. Needs of communities must be considered for representation as well as information.
    4. The purpose of the evaluation must be clearly articulated to ensure that funders are not using it narrowly as a punitive tool. Therefore, the evaluation must be directed to supporting and improving delivery of programs/information, but may also be used to articulate and publicize the value of PLEI.
    5. Evaluation, although necessary, should not be mandatory. Accordingly, no recommendations were given as to frequency or whether it should be a requirement for funding. The working group recognized that assessment is often informal and ongoing. Given limited resources, this is believed to be reasonable.
  2. What are the goals of the assessment?
    1. Evaluation should address concerns in the following general areas (specific examples of questions are provided in the paper): (i) National, provincial, and local system and program design for serving client communities; (ii) quality of client services; (iii) leadership, management and administration.
    2. Under each of the headings above, there should be an assessment as to whether overarching goals are being achieved and to what extent resources are being used effectively to address the identified need.
    3. All evaluations should look at what is not being done in addition to determining the effectiveness of the activities of the program.
    4. Depending on the context it may be important to compare approaches i.e., traditional client representation and the alternatives such as PLEI initiatives. This is important where the provision of legal aid and PLEI occur within the same association and therefore are financially accountable to the same agencies.
  3. What data should be collected?
    1. Data should be collected and analyzed further to the goals listed under the same headings as above [see 2(a)]. Useful areas include:
      • Administrative data; existing government and organizational statistics; surveys of clients, staff and community addressing client satisfaction, understanding of information and outcomes; focus groups of clients, staff and community members; information regarding legal, social, economic and political contexts (i.e., changes in entitlements and law affecting clients; literature (legal needs surveys, cost-effectiveness studies, etc).
    2. Evaluators at the government level should recognize the use of broad empirical data for understanding the identified need and developing activities to address it. This recognition would help to legitimize efforts expended to address these needs.
    3. The group rejected the option to insert ethical requirements into data collection.
    4. No final decision was reached on specific criteria to be used. However, the recommendation implicitly suggests a range of evaluative criteria.
  4. Who should be involved in assessments?
    1. A wide range of stakeholders should be included:
      • Clients; legal services program staff; management and governing bodies; advocacy groups; community based organizations; social service providers; private lawyers and bar associations; courts; law schools; social scientists; public officials.
    2. Without the participation of a broad range of stakeholders and clients, evaluation could be excessively staff-oriented. The danger is that assessment of programs could be defined around particular interests and priorities.
  5. Methodology: How should assessment be undertaken?
    1. Model data collection instruments and methods should be formulated by organizations at all levels as well as researchers and academics.
    2. Community programs should collaborate with others to set research agendas and undertake research.
    3. Programs should collect data as part of the ongoing operation.
    4. Due to a preference for local, community-based decision-making, the group rejected the idea of establishing a specific set of standards.
  6. Who should provide funding and resources for data collection and assessment?
    1. Independent entitiesi.e., academic institutions, foundations and non-profit research institutes should be encouraged to conduct this research.
    2. Federal government
    3. Major funders should form a consortium to fund the development of model assessment methodologies. They should also create a research grant program to assess the effectiveness of different approaches to delivery of PLEI.
    4. The group recognized that individual programs should not be expected to expend significant resources on self-assessment, but that government and other institutions should fund assessment in order to improve delivery of services.
  7. How do we share information?
    1. Establish a national clearinghouse that gathers information about evaluation methods and findings, including assessment, evaluation, methodology and research instruments. The use of technology would be essential to ensure that information is linked and widely accessible.
    2. Establish a library.
    3. Standardize data on community, provincial and national providers and make it available for research.
    4. Encourage projects to share methodologies as well as official assessments.
    5. Publish findings in journals.
    6. Issue reports at the national, provincial or local level.
    7. The group noted some significant drawbacks to sharing evaluations. However, these are to be weighed against the benefits in knowing what is working in other locations and precludes the unnecessary duplication of efforts.
  8. What are the challenges of assessment?
    1. Programs and researchers should:
      • Assure client confidentiality; Respect integrity of judgments involved in local decision-making; Avoid undervaluing what is difficult to assess i.e., quality of justice; Minimize burden on clients and programs; Determine priority to be given to assessment; Avoid undervaluing informal assessments.
    2. Assessment should be sensitive to external factors such as funding, political culture and demographics of the community.
    3. Goals and values should be clearly stated.
    4. Evaluation should not divert resources away from program delivery. Therefore, in order to prevent a chilling effect on creativity of programs, time spent must be balanced against the drain on resources that can be burdensome and overwhelming.
    5. Assessment must support, as opposed to undermine, the mission of providing meaningful access to legal information for low-income people.
    6. The group recognized the difficulty in valuing legal assistance on a social level.
Date modified: