The Effectiveness of Using Mediation in Selected Civil Law Disputes: A Meta-Analysis

2. Method

2.1 Design of the Meta-Analysis

Meta-analytic techniques are a quantitative method for aggregating knowledge, which have been used extensively in fields such as education and medicine, only more recently being adopted within the social sciences (Lipsey and Wilson 1993).  Similar to many standard quantitative research methods, the meta-analytic process has three basic steps:

  1. literature review – to identify and gather relevant research studies;
  2. data collection – to extract data through a pre-determined coding procedure; and,
  3. data analysis – to analyse the aggregated data using statistical techniques.

Proponents of meta-analysis, such as Rosenthal (1991) claim that “meta-analytic reviews go beyond the traditional reviews [of the literature] in the degree to which they are more systematic, more explicit, more exhaustive, and more quantitative. Because of these features, meta-analytic reviews are more likely to lead to summary statements of greater thoroughness, greater precision, and greater intersubjectivity or objectivity” (p.17).

Essentially, a meta-analysis aggregates the magnitude of a relationship between two or more variables for collections of studies (Glass, McGaw & Smith, 1981).  The studies under examination in a meta-analysis often differ on a number of important characteristics, such as the operationalisation of dependant and independent variables, sample selection methods, sample size and the quality of research design.  A meta-analysis describes the typical strength of the relationships under investigation, the variability and the degree of statistical significance.  A meta-analysis also provides a researcher with the opportunity to identify and explore variables that affect the strength of relationships between variables.  The outcome of a meta-analysis is an ‘effect size estimate’(ESE), which can be described as an estimate of the effect the independent variable has on the dependant variable.  For instance, an average effect size of + 0.05 for an independent variable (e.g. evaluative mediation style) compared to a dependant variable (e.g. cost savings) could be described as “an evaluative mediation style accounted for a 5% improvement in cost savings”.

2.2 Sample: Study Identification Criteria

Studies were gathered for the meta-analysis through a comprehensive literature search.  The search strategy started with on-line databases[1] and an Internet search, using search terms such as ‘dispute resolution’,‘alternative dispute resolution’,‘mediation’, and ‘conciliation’, as well a ‘mandatory mediation’,‘voluntary mediation’,‘mediation evaluations’ and related short forms.  Further references were obtained through the examination of the bibliographies of sources that met the study criteria.  In addition, in order to elicit unpublished or un-catalogued studies, 85 individuals or organizations with expertise and experience evaluating mediation programs were contacted, 50 responded.  Using this method over 250 sources were identified.

A specific set of criteria were established in order for a study to be included in the meta-analysis:

  1. the study evaluated a mediation program concerned with tort or contract cases (not criminal law, Aboriginal law or family law cases);
  2. the study evaluated a use of mediation whose style was either evaluative, facilitative or transformative (‘arbitration’ and ‘negotiation’ were not included);[2]
  3. the study used a control or comparison group that did not use mediation or used mediation of a different type;
  4. sufficient statistical information was reported in order to calculate an effect size;
  5. at least one of the following four outcomes was reported for both the mediation group and comparison group —
    • plaintiff satisfaction;
    • defendant satisfaction;
    • procedural fairness; and/or,
    • cost effectiveness.
  6. the study was conducted between 1980 and 2006.

After applying the selection criteria, only 26 publications remained.  Of these publications, some examined multiple mediation programs, while others examined multiple sites.  In all, a total of 37 unique programs or sites were available for analysis.  (See Table 1 for details.)

2.3 Coding Procedures

Standardized information was drawn from each research study using a coding manual.  Following the conventional standards of meta-analyses, multiple definitions of each of the outcomes of interest were accepted.  For instance, the term ‘fairness’ covered related concepts such as ‘equity’ and ‘justice’ that were measured by some studies.

Several studies reported on the results of multiple, clearly identified comparisons between a mediation group and a comparison group.  In these cases the different comparisons were coded separately. 

The variables included in the coding manual were developed from the areas of analytic interest to the Department of Justice, but were also constrained to those variables which were commonly reported across a range of studies.  For instance, although it would have been interesting to examine the impact of gender and minority status on the effectiveness of mediation, not many studies examined these variables.  While an analysis of gender is common in studies of family law or criminal law mediations, it is not in studies of mediation in tort and contract cases.  While limited examples of such studies do exist (Hermann 1993) they were not numerous enough to allow for a robust analysis.

In order to generate sufficient data for analysis, a number of coding techniques were employed.  For example, if 70% or more of the study sample were real estate cases, we coded it as a “primarily real estate program” and if 70% or more of the study sample used evaluative mediation techniques, we coded it as a “primarily evaluative program”.  We also accepted textual indications that the programs were primarily of a certain type.  In addition, a number of variables were coded only if the authors made an explicit statement.  For instance, the program was coded “yes” as a pilot project only if the authors directly stated this to be true.  Furthermore, for variables indicating agreement or satisfaction, responses above neutral were tabulated as positive responses.  Therefore, the comparisons made in this report are subject to these limitations.  It should be noted that these are general issues with all meta-analytic research studies.

All values involving monetary amounts denominated in foreign currency were converted to Canadian dollars for comparison purposes.  The average annual exchange rate (Antweiler 2005) for the last year in which data was collected for the study was used to make the calculation.

2.4 Effect Size Calculations

Following the meta-analytic techniques of Rosenthal (1991),the phi coefficient (Pearson’s r product moment correlation applied to dichotomous data) was used as the effect size estimate.  In studies where one mediation group was compared against multiple comparison groups, the comparison groups were aggregated for the overall analysis and disaggregated for the analysis of comparison group effect.[3]  This allowed us to see if the type of comparison group was associated with different outcomes in the other variables.  Also, where multiple follow-up periods were reported in a single study, the longer period was selected. 

After the effect size from each study was calculated, a number of analyses were conducted across each of the outcome measures.  First, the overall mean effect size and corresponding confidence intervals were calculated.  Additional analyses were conducted to explore whether particular variables had a moderating effect on the magnitude of the effect size.  For instance, if the information was available, the area of law targeted by the mediation program, if the mediation program was voluntary or mandated in some way and how mediators were trained and paid were examined to determine possible effects on program success.  Thus, specific program characteristics were isolated for further examination.

2.5 Limitations

We found that many discussions or evaluations of mediation programs were largely qualitative, often conducted by people with law or legal studies backgrounds who may have had little training in quantitative research methods.  In addition, a surprising number of studies that attempt to rationally utilize numbers in the evaluation of the mediation program did not adequately analyse the mediation program in comparison to any other group.  Thus, although many studies were found and examined, most did not meet our study criteria.  As this study did not include a qualitative review of the excluded studies, it is unknown if the findings from these qualitative studies of mediation programs or sites exhibited different characteristics or trends from the quantitative ones that are analysed here.  It is conceivable that those mediation programs or sites designed with the primary goal of achieving time or cost savings might be more likely to be measured using quantitative methods, while those more focused on fairness and satisfaction might be more appropriately evaluated using qualitative methodologies and thus be excluded from this meta-analysis.


Date modified: