​

This paper was prepared and settled jointly by the members of ADRAC.

 

Effectiveness in ADR: Key Issues

​

Although ADR has a well-established reputation for being an effective (or successful) method for resolving disputes, that reputation is not without challenge and controversy.[1]
 

There are unresolved issues around what “effectiveness” means in the context of ADR, and around how such effectiveness is measured.  There are a number of approaches to conceptualising effectiveness in ADR, each of which considers particular variables, including: point of view,[2] interpretations of ADR constructs,[3] and approaches to measuring effectiveness.[4]   In addition, many factors influence ADR effectiveness and it is difficult to establish which aspects of any ADR process could be called “predictors” of effectiveness.

What is effectiveness in ADR?

Effectiveness in ADR is most commonly accepted as being the achievement of settlement; in other words, a successful ADR process gets the disputants to an agreement.[5]  For many (commentators and practitioners alike) this is seen to be a narrow – even crude – definition that excludes many other dimensions of effectiveness.
 

The ADR literature describes other definitions/interpretations of effectiveness in ADR that take into account (but not limited to):
 

• Levels of disputant (or participant)[6] satisfaction – with the process, with the practitioner, and/or with the outcome/s;[7] the perceived fairness of an ADR process underlies the levels of disputant satisfaction

• The nature of the terms of agreement (e.g., are they more personalised than a basic exchange of resources?)

• The perceived durability of the agreement[8]

• Perceived improvements in the relationship between the disputants

Effectiveness can include the achievement of settlement plus any, or a combination of any, of the above factors.  It can also include a partial agreement and/or the narrowing of the issues in dispute, as well as one-off definitions that are specific to the presenting dispute.
 

Generally, structured ADR programs tend to define effectiveness in terms of settlement.  While this approach may simplify program evaluations, as well as their various reporting mechanisms, it may also promulgate ADR practices and approaches that are focused only on settlement.
 

It has been suggested that programs with a high settlement rate may not be necessarily either fair or effective,[9] and that focus on settlement might mask disputant dissatisfaction with the process or with the practitioner.  Some argue even further that the achievement of settlement is not synonymous with “resolution” because the latter implies a deeper, more complex addressing of the whole dispute situation, whereas the former represents merely a “quick fix” of the major symptoms of the dispute.

Two interpretations of “effectiveness” in ADR

In 1998, The Australian Law Reform Commission published its five objectives for the federal justice system, including associated ADR processes: the system and its processes should be: just, accessible, efficient,[10] timely, and effective.[11]
 

Regarding “effective”, the ALRC proposed that:

• ‘the process should ensure, or at least, encourage a high degree of compliance with the outcome’ [synonymous with durability]

• ‘at the conclusion of the process, there should be no need to resort to another forum or process in order to finalise the dispute’ [synonymous with reaching settlement]

• ‘the process should promote certainty in the law.’[12]
   

In the National Mediator Accreditation System, the definition of mediation (a specific ADR process), does not include the achievement of settlement:
 

‘Mediation is a process that promotes the self-determination of participants and in which participants, with the support of a mediator:

(a) Communicate with each other, exchange information and seek understanding

(b) Identify, clarify and explore interests, issues and underlying needs

(c) Consider their alternatives

(d) Generate and evaluate options

(e) Negotiate with each other; and

(f)  Reach and make their own decisions.’[13]
 

How is effectiveness in ADR measured?

On the whole, in any given situation, the preferred definition of effectiveness tends to dictate the way in which the effectiveness is likely to be measured.  ADR program evaluations tend to measure settlement rates, although some do include measures of disputant satisfaction. 
 

In particular, court-connected ADR programs[14] measure settlement rates – although they can also measure whether a matter is no longer in the court lists.  The latter may not necessarily reflect a settlement having been reached; it may also indicate that one or more disputants chose not to continue with court proceedings – which can be for many reasons other than having achieved settlement.
 

Disputant (and other participant) perceptions of an ADR process/practitioner/outcome tend to be measured by post-process surveys (or interviews) of the subject disputants/participants; in some instances, ADR practitioners have been surveyed about what they thought the disputants thought about the process.  Such surveys can encompass: the nature of the process, disputant participation levels, fairness, the nature of the agreement, the behaviour of the practitioner, the behaviour of the disputants/participants, general/specific satisfaction, disputant relationship improvements, and the durability of any agreement reached.  Such research approaches are fraught with methodological problems which remain unresolved; one commentator has recommended recently: ‘… ADR research questions should look beyond party satisfaction and settlement rates.’[15]
 

Generally, settlement rates are established through examination of program – or court/tribunal – files.  As noted above, finalisation of a court file may not necessarily be the result of achieving settlement.  Where a court refers a matter to a private ADR process, it is most likely that no further records of the matter are retained by the court. There is a significant proportion of private ADR practitioners and they produce very limited independent data about their processes.  In addition, the data from government-funded programs (e.g., court-connected mediation, diversionary/restorative justice, and/or statutory conciliation programs) are often restricted and have quite limited public availability.

Many studies of effectiveness in ADR focus on only one ADR process (most commonly mediation, conciliation, or restorative justice processes)[16].  The findings from a study of one style of process are not necessarily transferable to other ADR processes – let alone to all ADR processes.  It is important for appropriate comparative research designs to be developed that enable cross-process investigations.
 

Many factors can impede any investigation, or measurement, of effectiveness in ADR.  One is the lack of consistency in definitions and in practice of ADR processes.[17]  Another is confidentiality and privacy: all ADR processes are protected by significant levels of confidentiality or privacy, limiting their availability for rigorous, independent research scrutiny. 
 

Which factors influence effectiveness in ADR?
 

Although it has been suggested that the ADR practitioner strongly influences effectiveness in ADR processes,[18] it has also been suggested that the characteristics of the disputants and other participants are also very influential.[19]  In fact, many factors have been reported to influence effectiveness in mediation.  They include:
 

• The characteristics and background of the disputants and other participants

• The expectations of the disputants and other participants – and disputant expectations created by their advisers

• The nature and extent of any intake and pre-intake process/es

• The nature of the presenting dispute

• The context of the dispute, the disputants, and the mediation process

• The institutional context of the dispute, the mediation process, and the mediator

• The timing of the mediation process

• Whether attendance at the mediation is voluntary or mandated (noting that reaching an agreement is not mandatory)

• The skills, approach, style, background, and philosophy of the mediator

While it is reasonable to assume that any of these factors could be readily applied in other ADR processes, it is difficult to design research approaches that can properly investigate the extent of these influences.

There are no consistent research data confirming that any model of mediation practice is more effective than any other model of mediation practice.
 

Additional factors that are likely to influence the reliability of effectiveness data include:
 

• Private mediators select matters that are more likely to reach agreement – thus skewing any settlement data[20]

• In most court-connected restorative justice programs, offenders are required to admit the offence in order to gain access to the program; it has been suggested that this amounts to selection bias – again potentially skewing any settlement, or other effectiveness, data[21]

• In court-connected mediation programs, disputants may self-select into the program because they are already amenable to a negotiated settlement – again, potentially skewing any settlement, or effectiveness, data

• Researchers’ personal preferences can influence definitions and measurements of effectiveness that are applied in any ADR research project.[22]
   

Conclusions
 

Although ADRAC acknowledges the many unresolved issues that confound investigations into the effectiveness of ADR, ADRAC is convinced that ADR processes are socially and individually beneficial and effective.

ADRAC welcomes rigorous research into all aspects of ADR, including the measurement of effectiveness and success in ADR.
 

Cross references to other ADR Mapping Papers, ADRAC and other documents:
 

An overview of ADR forms, methods and practices appears here: http://www.adrac.org.au/adr-mapping/adr-forms-methods-and-practices 
 

More detailed information concerning various forms of ADR can be found in individual ‘ADR Mapping’ papers (such as those dealing with mediation, conciliation, expert determination, facilitation, online dispute resolution, and family dispute resolution).
 

Design aspects of mediation, particularly initial intake sessions, are discussed in a separate paper found here: http://www.adrac.org.au/adr-mapping/initial-separate-sessions.
 

Although an oft-recited tenet of ADR is the importance of ‘fitting the forum to the fuss’, problems arising from the lack of evidence-based data concerning the different forms of ADR are discussed here:
 

http://www.adrac.org.au/adr-mapping/evidence-based-data-collection-for-adr-research

http://www.adrac.org.au/adr-mapping/the-evidentiary-black-hole-of-mediation

http://www.adrac.org.au/adr-mapping/dr-knowledge-getting-beyond-belief   

 

[1] For an overview of the persistant misperceptions in this area, and the challenges facing researchers, see: American Bar Association Section on Dispute Resolution, Report of the Task Force on Research on Mediator Techniques (2017); available at: https://www.adrac.org.au

[2] Point of view: of each disputant, of the practitioner, of professional support people and advisers, of constituents, and of others – including any organisational point of view for ADR programs.

[3] Interpretations of ADR constructs: for example, durability of outcome can be perceived in various ways; for some, a durable agreement is a rigid agreement, while for others a durable agreement is a reliable agreement, and so on. Generally, measures of durability include levels of compliance with the terms of any outcome.

[4] Measuring effectiveness: measurement of effectiveness can be via parametric or nonparametric, and other, methods.

[5] In arbitration, any final arbitrator decision is binding, as are the terms of the arbitrator’s award; for the purposes of this paper, that decision and its attendant award are accepted as being synonymous with “achieving settlement”.

[6] In this paper, the term “disputant” is intended to apply to the direct parties to the dispute; the term “participant” is intended to apply to any other participant, other than the practitioner (e.g., legal, or other, advisers).

[7] It has been noted that, where no agreement has been reached, but the disputants are satisfied with the process, the process should be considered to have been effective.

[8] As noted earlier, the durability of any agreement can vary according to perspective. In brief, durability can be a relatively linear measure for a self-executing agreement, but a less definitive measure of the inherent adaptability of a non-self-executing agreement.

[9] L. Tinder and J. Kellett, ‘Fairness, Efficiency and Effectiveness in Court-Based Dispute Resolution Schemes’ (2007) 21 International Journal of Law, Policy and the Family 322.

[10] The issue of efficiency – and cost savings – that has traditionally been associated with ADR has proven difficult to confirm; overall, the research findings in this area are mixed, and, in one report, the findings could be based only on the practitioners’ and disputants’ perceptions of cost savings, rather than on objective data.

[11] Australian Law Reform Commission, Rethinking the Federal Civil Litigation System (Issues Paper 20, ALRC, 1998).

[12] Australian Law Reform Commission, Rethinking the Federal Civil Litigation System (Issues Paper 20, ALRC, 1998), 3.16.

[13] National Mediator Accreditation System (NMAS), 2015, p 2.

[14] Court-connected ADR programs can include: mediation, conciliation, restorative justice processes, conferencing, and victim-offender mediation; private ADR processes available for court referral can include any of the above, plus arbitration.

[15] D. T. Eisenberg, ‘What We Know and Need to Know about Court-Annexed Dispute Resolution’ (2016) 67 South Carolina Law Review, p 247.

[16] On the whole arbitrations are conducted privately.

[17] See: L. Boulle, Mediation Principles Process Practice (3rd ed, LexisNexis Butterworths, Australia, 2011); T. Sourdin, Alternative Dispute Resolution (5th Ed, Thompson Reuters, Australia, 2016).

[18] K. Mack, Court Referral to ADR: Criteria and Research (National ADR Advisory Council and Australian Institute of Judicial Administration, 2003).

[19] R. L. Wissler, ‘Mediation and Adjudication in the Small Claims Court: the Effects of Process and Case Characteristics’ (1995) 29(2) Law and Society Review 323.

[20] For example, see J. A. Wall, Jr, and D. E. Rude, ‘The Judge as Mediator’ (1991) 76(1) Journal of Applied Psychology 54.

[21] D. T. Eisenberg, ‘What We Know and Need to Know about Court-Annexed Dispute Resolution’ (2016) 67 South Carolina Law Review 245. 2016.

[22] E. P. McDermott, ‘Discovering the Importance of Mediator Style – An Interdisciplinary Challenge’ (2012) 5(4) Negotiation and Conflict Management Research 340.

​

​