Evaluation of Public Legal Education And Information: An Annotated Bibliography


1. Introduction

1.1 What is PLEI and Why an Annotated Bibliography on Evaluation?

Law-related education beyond what is taught in university faculties throughout the country has come to be known as "public legal education and information" or PLEI. The proliferation of PLEI materials, services and programs has occurred over the past three decades in Canada, in step with other common law jurisdictions, and very likely with other legal systems around the world. Each province and the territories now have organizations that are solely dedicated to PLEI provision. Some of these core PLEI providers function as an umbrella for various associations, schools and community organizations that offer PLEI services and programs. In other provinces, there is less focus on coordinating PLEI i.e., by performing needs assessments and they are one of a number of organizations exclusively devoted to PLEI. Furthermore, in some provinces such as Ontario and British Columbia many PLEI initiatives are enveloped in the provincial legal aid institutions and managed under the Office of the Attorney General. Regardless of how PLEI is administered and delivered across Canada, its mandate seems to be generally consistent. Providing information about legal rights and obligations to the Canadian public is one practical and effective way to ensure greater access to justice.

The purpose of this project was to review the available literature, government, non-government and program material on the evaluation of PLEI initiatives. The emphasis was on Canadian materials produced within the past five years, however, it will be noted that many of the evaluations pre-date this timeframe. This is largely due to the shortage of PLEI evaluation and discussion that has been formally documented. The objective in looking at these sources was to determine what types of approaches are being taken to evaluating PLEI and what barriers exist for evaluators. Many authors have maintained that an overarching problem to accessing and conducting PLEI evaluation is the result of imprecise definitions. The lack of precision and concurrence regarding the terms "public" "legal" "education" and "information" poses ongoing difficulties for those who engage in any form of comparison and/or assessment. Consequently, the 1986 Survey of Canadian PLEI Providers discovered that many organizations that include PLEI in their activities do not self-identify as PLEI organizations. This is still true today.

1.2 Methodologies

Bibliographic materials for this report were gathered from several sources. The three prominent legal databases, Quicklaw, Lexis-Nexis, and Westlaw, were searched for academic articles in addition to multi-library searches for relevant legal literature. PLEI documents developed by and for the Department of Justice were forwarded directly from the contract supervisor in the Research and Statistics Division. The Public Legal Education pages of ACJNet were searched and a number of materials were also sent from the Legal Resource Centre, an affiliate of the Legal Studies Program at the University of Alberta. The balance of the materials collected was forwarded in response to a nation-wide information/solicitation notice sent to over 50 organizations connected with PLEI. In addition to these sources, a number of Internet searches were conducted to find evaluation materials. As the bibliography was concentrated on the Canadian PLEI context, Internet research in other jurisdictions was purposefully limited.

The challenges faced in the bibliography were mostly related to time and lack of materials. Most of the electronic mail correspondence and material gathering occurred in the month of December. This is a particularly difficult time to contact people in both governmental and non-governmental organizations. With more time, it is likely that a few more evaluation reports may have been located. Despite this constraint the materials annotated represent the types of evaluations that have been performed by and for PLEI providers and their funders. The government documents and academic articles were not impacted by time limitations. Lack of information was the other primary challenge. Most of the academic articles found pertained to legal aid programs in the United States and only addressed public or community education peripherally. Furthermore, papers on public legal education almost universally advocate for the use of PLEI or discuss in detail specific PLEI initiatives. Few articles ever mention the need or methods for evaluating programs. Relatedly, many of the organizations that did forward evaluation material supplied one or two-page "feedback forms" or questionnaires which they distribute after a particular program or along with educational materials. These evaluation efforts are unquestionably valuable for needs assessment and content/format revision, however, they do not provide useful information for this bibliography.

Although the bibliography cannot claim to be exhaustive of all available PLEI materials in Canada, the use of multiple approaches to finding materials as well as the variation of document type reviewed ensures that the collection is comprehensive.

1.3 The Continued Case for PLEI

The scope of PLEI programs and materials is extensive and varies between organizations and regions depending on the surveyed and perceived public need. Under Canadian law all citizens are entitled and even expected to know their legal rights and responsibilities. However, given the language used in statutes and regulations, among other systemic barriers, access to justice demands more than making these laws publicly available. To a greater or lesser degree federal, provincial and municipal laws affect all people. PLEI offers a constructive and transformative method for achieving understanding of the content, process and impact of laws on our lives. Independent of discrete legal disputes or specific compliance requirements where individuals encounter the legal system directly, every person stands to gain from understanding the political-legal institutions in which they function. For these reasons PLEI is often aimed at elementary and secondary students as well as new immigrant populations.

PLEI is also one way to empower low-income individuals or other marginalized groups who are in need of information. In addition to the general desire to inform and educate persons about their rights and responsibilities under the law, PLEI can be used for purposes of self-representation or to indicate when it is necessary to retain a lawyer. There is ongoing debate about whether self-representation is to be encouraged as a form of empowerment and a realistic response to legal aid gaps/reductions or whether increased access to legal aid is unquestionably the preferred route of under-served populations. Recent studies in the United States found that 50% of low and middle income clientele can be served though community legal education materials and that the limited access to lawyers was largely the result of lack of information about when to seek legal advice (see Powers, 1997). Regardless of the outcome of this discussion, PLEI provides knowledge required by individuals to make informed decisions. PLEI is also a way for the public to become informed and educated about the laws that affect daily life. Going beyond the government mandate of securing access to justice, it gives meaning to the "penchant" that policy makers have for public consultations. In the cases of referendums or political polling about proposed legislation little attention is paid to whether the "public" in fact understands the impact of the laws on which they are voting (see Bowal, 1998). PLEI could play a role in legitimating this aspect of the political process by dispelling myths and correcting mistaken assumptions about the legal system in Canada, which is largely informed by the American media, including entertainment and news programming.

Although television, video, radio, computers and phone lines have all been used successfully in PLEI initiatives, some writers express concern regarding the potentially disadvantaging impact of increased use of technology in the provision of PLEI on low-income people. The use of technology must be assessed and monitored to ensure that clients are finding information, accessing assistance and obtaining favorable outcomes. If the universal starting point is focused on access, the provision of services should be equally available to those with limited access to computers and limited education. Projects that opt for these modes of communication and dissemination must be alert to unintended consequences, and must also ensure that low-income individuals have access to computer training and computers through social service agencies and public libraries (see Houseman, 1998). Once these concerns are addressed it is commonly believed that selective use of computer and Internet technologies can be vastly beneficial to PLEI organizations and ultimately to the populations they serve. Province-wide and inter-provincial coordination could be exponentially improved through suggested developments. The range of activities include: province-wide email access for all community based organizations and institutional providers, province-wide websites for up-to-date info about legislation, regulation and policy developments that impact various constituencies, electronic libraries of relevant materials which are accessible to all PLEI providers, development of a coordinated research strategy integrating internet usage, on-line services, proprietary sources and other resources, and development of coordinated data-management systems to facilitate information sharing (see Houseman, 1998: 422-427).

1.4 PLEI Evaluation: Approaches and Ideas

The following notes are drawn from various reports and articles included in the bibliography. Combined with the list below, they provide some overarching guidelines to bear in mind as the annotations are reviewed.

  • Assessment must be audience-driven in that its design and objectives are sensitive to how knowledge/information will be used by participants and the learning contexts in which they operate. The greater the extent to which these factors are examined and built into program design, the more effective an evaluation will be at determining if needs are met and the extent of impact.
    • Authors of VIOLET: Learning on the Net identified six characteristics of the particular target audience and related assumptions were accounted for in the evaluation process.
  • Tools for Moving Forward noted that two opposing approaches persist regarding what "level" of administration is better positioned to develop and implement PLEI evaluation. While some providers believe that an umbrella organization such as PLEAC or the Department of Justice should use their resources to develop evaluation tools for PLEI providers, other suggest that core PLEI organizations as well as intermediaries are in the best position to design and implement evaluations of their programs and client impact.

Below are further ideas and observations that the author has gathered from various reports. These are not directly articulated in the materials.

  • In developing an evaluation framework for PLEI the latest evaluation research in the fields of education and human/community services should be obtained and incorporated.
  • Findings of evaluation studies and conclusions drawn from assessments can and should be used by funders in setting and guiding priorities.
  • It is important to have a comprehensive list of community organizations that are involved in PLEI delivery as well as inventories of materials that exist in topical areas. This need is being addressed in part by ACJNet, however, as it was found in the Survey of Canadian PLEI Providers in 1986, "the survival of PLEI activities is dependent upon active coordination and communication among organizations who provide PLEI" (p.108). This massive undertaking can only be accomplished in a fashion similar to the work of Godin, Gill & Alderson, Shariff and others. Projects such as these have gathered all PLEI materials that relate to a specific topic and have summarized the contents. Making this type of inventory available in all areas of law will facilitate evaluation by identifying gaps and allowing providers to compare their programs and materials.
  • Given the reality of budgetary constraints, it is recommended that PLEI providers and funders incorporate an assessment of flexibility and adaptability of their programs and services as a matter of course in all evaluations. This element will ensure that programs are maximally responsive to the unique characteristics of audiences and additionally, it will facilitate creative exploration of how materials may be widely applicable across PLEI areas and organizations.
  • As methods and tools used in assessment tend to vary with the model of program delivery and audience more than they do over time, preparation of a comprehensive evaluation framework should include some examination of the older non-PLEI evaluations.
  • Both internal and external evaluations should always make recommendations for ongoing evaluation/quality control/assessment of programs and services. Given the prohibitive nature of full-scale evaluations, this is a critical component of any formal evaluation process as it is directly beneficial to those in the organizations and may be implemented immediately. Whereas staff may not have skills or time to contemplate the best methods for ongoing assessment, an evaluator is ideally situated to consider this component of PLEI.
Date modified: