Evaluation of Public Legal Education And Information: An Annotated Bibliography

Executive Summary

This annotated bibliography attempts to amass the current available materials on evaluating public legal education and information (PLEI) initiatives. The provision of PLEI manifests in various forms including distribution of pamphlets or information packages, classroom-format teaching, telephone hotlines, audio-visual products and multi-day seminars or workshops. There is growing recognition among many jurisdictions that acquisition and "ownership" of legal knowledge resides in an exclusive domain insofar as it is confined to university-operated law schools. Moreover, the Canadian government continues to expand and explore the dimensions of access to justice. Accordingly, momentum has been building for enhancing traditional approaches to access, which include more solution-oriented and participatory forms of justice.

The literature and materials reviewed in this project are divided into sections based on document type. The contractor was asked to review a range of documents on PLEI evaluation with a focus on Canadian materials produced over the last five years. The types of documents reviewed include academic articles that discuss various PLEI initiatives, government materials on PLEI and actual evaluation reports of PLEI projects. Where practical and useful materials are included in the document, they have been identified and explained. The annotations have been organized around four prevalent themes that emerged in earlier reports of PLEI evaluation. Although many of the findings evade concrete classification, issues were partitioned into thematic headings in anticipation of the bibliography's intended use. Two overarching purposes for which these themes were developed are utilization, in terms of designing evaluation, and identification or prevention of common difficulties. The four themes that run throughout the bibliography are: (i) challenges to PLEI evaluation; (ii) connections between goals of PLEI and its evaluation; (iii) challenges of funders evaluation requirements; (iv) evaluation methodologies. A section on research and knowledge gaps is included in the Overview of Findings. The themes that directly identify challenges may be used preemptively. In identifying and documenting challenges this bibliography recognizes the potential for benefit through the experience of others, which was also a recurring thread in the articles. Ideas concerning the connection between stated goals of PLEI programs and evaluation objectives were singled out, as articulation of these connections can often dictate what evaluation methods should be used. Evaluation methodologies were included to illustrate trends and creativity in both external and internal evaluations. Finally, research knowledge and gaps were summarized in the Overview of Findings section to highlight areas for future research and consideration.

Itis apparent from the materials reviewed that challenges facing PLEI evaluation are multi-faceted. Therefore, it is not surprising that despite the extensive list of PLEI organizations and initiatives across Canada, there are very few reports that directly evaluate PLEI programs, services or materials. There are a small number of well-organized evaluations that have been produced in the last five years in addition to a larger number of evaluations for other legal-related programming such as legal aid (see Brantingham and Brantingham, 1984). The most comprehensive and functional document available is the Evaluation Resource Book for PLEI Organizations, written by Focus Consultants in 1986. This detailed guide was the result of earlier research in which prominent issues and obstacles faced by PLEI providers were addressed. Most of these issues are still very much alive today and therefore are reiterated in the annotations.

Recurring Themes in Evaluation

The themes as described above are set out below along with a few of the prevalent findings.

  1. The Challenges of PLEI Evaluation
    • Lack of skills, resources, money
    • Insufficient definitions of PLEI and evaluation
  2. Connections between Goals of PLEI and its Evaluation
    • Many of the goals and objectives of PLEI projects cannot be measured by quantitative means
  3. Challenges of funders' evaluation requirements
    • Without stable funding PLEI providers are dissuaded from engaging in extensive evaluation efforts
    • Different emphasis and goals between funders and PLEI providers leads to distinct methods of evaluation and misinterpretation of findings
  4. Evaluation Methodologies
    • Most common forms: questionnaires and surveys, semi-structured interviews, call back/follow up; pre/post-testing; focus groups
  5. Research and Knowledge Gaps
    • No mention is made as to how evaluations, assessments or quality reports should be used by the program or organization under evaluation
    • Similarly, there is no discussion about how findings or recommendations of evaluations conducted and transcribed can be used by other PLEI agencies and organizations
Date modified: