Assessing the Effectiveness of Organized Crime Control Strategies: A Review of the Literature

3. Definitional Issues, Measurement and Other Concerns in Organized Crime Research (cont'd)

3.3 Crude and Varied Performance Measures

3. Definitional Issues, Measurement and Other Concerns in Organized Crime Research (cont'd)

3.3 Crude and Varied Performance Measures

A variety of measures have been proposed in assessing the impact of OC control strategies (Peterson, 1994:381; Maltz, 1990). A study may, of course, adopt several measures. Some of the key measures used or proposed have been:

  • Body-count measures (e.g., the number and nature of cases opened, arrests, prosecutions, convictions, and prison terms imposed);
  • Size of the illicit market (e.g., changes in the volume of drugs trafficked/sold, number of usurious loans);
  • Volume and scope of organized crime groups (e.g., the impact of control efforts on criminal organizations or their leadership);
  • Efficiency of criminal justice units (e.g., strike forces);
  • Harms produced by OC (e.g., the impact of control efforts on the physical, economic, and other harms occasioned by OC);
  • Seized and forfeited assets and recovery of unpaid taxes.

Body-Count Measures

Among the most common measures of the impact of OC control strategies are tallies of arrests and convictions achieved. The Canadian Association of Chiefs of Police (1993) for example, provide a breakdown, by province, of the number of charges and arrests for various activities in which criminal organizations are thought to be involved. However, such reports provide no indication as to the proportion of acts of prostitution, illicit gambling, drug trafficking, and related activity that are linked to OC. Also, police data of this kind tend to seriously underestimate the volume of such activities, as there is a massive "dark figure" in relation to consensual crimes, extortion, and many other activities associated with OC.

Furthermore, simple body-counts of this type provide no indication of the justice system inputs required to generate them (i.e., no cost-benefit analysis) and no information on the impact of the law enforcement activity on the relevant criminal organizations. Maltz (1990:39) argues that counting convictions and computing average sentence lengths is tantamount to determining the battles won with no indication of how the war is going. Since all convictions are given the same weight, such data provide no information on the significance of the battles won. A strike force can conceivably run up an impressive tally of convictions by pursuing the easiest cases. In some contexts, this may even assist more powerful criminal organizations, as more vulnerable competitors are put out of circulation.

Moreover, what is one to make of a reduction in the number of charges following the introduction of a new enforcement initiative? Is such a reduction an indication of failure or an indication of the deterrent effect of the new measures (Marx, 1988:109)? Many police reports trumpeting a large arrest tally, or the conviction of some high profile offenders, imply that more charges, arrests, or convictions are indications of success. However, such increases may reflect an increase in criminal activity or simply that the relevant agency has benefited from additional resources.

Conviction and sentence-based measures are also problematic because they discount the ancillary benefits of prosecution where a conviction or lengthy sentence is not achieved. Prosecution in itself can have a general deterrent effect, put an end to the illicit behaviour under consideration, bring the public’s attention to those who may victimize them, and free sectors of the economy from infiltration by OC groups (Maltz, 1990: 9).

Measures such as the conviction rate may also be counterproductive in terms of the operations of OC strike forces. Maltz (1990:9) argues that these agencies should undertake risky cases and concern for a high conviction rate would lead them to pursue less risky alternatives. Also, prosecutors could inflate their conviction rates by accepting a guilty plea on a lesser charge.

In themselves, therefore, such body-counts provide just a starting point, but little in the way of a useful indication of an initiative’s efficacy or efficiency. It is important to note that the Homicide Survey, maintained by the Policing Services Programme at the Canadian Centre for Justice Statistics, collects data on "gang-related" homicides.

Size of the Illicit Market

Attempts have been made in Canada (Porteous, 1998) and elsewhere to ascertain the volume of OC. Beare and Schneider (1990:2) assert that, "There is no verifiable method for determining the size of the illicit economy. Estimated figures in this area of illicit proceeds, however carefully calculated, are only guesses. Once stated they take on a reality they do not deserve."

There are many unknowns in estimating the size of the illicit market in relation to any product or service. With regard to prohibited drugs, the size of the market is a function of the quantities purchased and the cost, neither of which has been accurately measured (Kleiman, 1994). Drug seizures are an inadequate indicator of quantity, as only a small percentage is seized. A 10 percent seizure rate is considered highly speculative and variable across drugs and time (van Duyne, 1996:128). ). Furthermore, the estimates of profit margins that form the basis for the calculation of proceeds from the drug trade are also controversial (van Duyne, 1996:129). Drug prices, sources, and amounts bought are difficult to ascertain as they are usually not probed in existing drug use surveys (Kleiman, 1994).

Much of the United States’ data on drug trafficking is collected by the National Narcotics Intelligence Consumer Committee. The Committee concedes that there is a high degree of uncertainty in the estimates of the amounts of drugs produced and entering the US, trafficking patterns, and prices and purity of drugs (President’s Commission on Organized Crime, 1986:342-343). The President’s Commission has added that there is no centralized data collected system on drug seizures, resulting in the frequent double and triple reporting of the same drug bust by participating agencies.

In discussing the diversion of controlled chemicals for the purpose of manufacturing illicit substances, the United States’ General Accounting Office (1993) noted that the international community has no centralized data exchange for monitoring controlled chemicals.

Aside from official sources, information on the size of the illicit market can be obtained from perpetrators themselves. Reuter and Haaga (1989), for example, acquired detailed information about the drug market from incarcerated high-level drug dealers. Studies of incarcerated persons are always exposed to the criticism that those contacted do not constitute a representative sample, as the most intelligent and sophisticated offenders are less likely to be caught and convicted. The issue of whether those interviewed for this study did represent all high-level drug dealers was also underscored by the fact that just 40 percent of those contacted consented to giving interviews.

Another concern with the use of a particular form of criminal activity, such as the amount of illicit drugs sold/smuggled, as a performance measure is that criminal organizations may compensate by a crackdown on one activity by increasing their involvement in another (Maltz, 1990:14). It would serve as little consolation to society if OC groups shifted their activities from gambling to drug dealing. An intensification of law enforcement efforts in one area is also likely to generate more seizures of contraband, underscoring the need to develop measures that are not influenced by enforcement activity.

Maltz (1990:15) adds:

We may not be able to estimate the magnitude of an activity, but must be content with knowing whether it is increasing or decreasing, or what its targets are, such as which businesses are being infiltrated. Furthermore, we may have to resort to sources of information that cannot be cross-checked—for example, the comment of the target of an investigation in a tapped telephone investigation that he has had to alter his activities due to enforcement efforts would be a definite indicator of impact, but could not be disclosed in an evaluation report.

Volume and Scope of Organized Crime

The United States’ General Accounting Office (1977:17) has reported that "complete and reliable data is not available on the number of organized crime figures in particular areas, their position within the organization, and the extent of their criminal activity."

Measures that attempt to ascertain the number of criminal enterprises may be limited in their validity. Van Duyne (1996:207) argues that OC may not be quantifiable in the sense that the number of illicit enterprises is not static:

Organized crime is not a quantifiable phenomenon. In the first place any operational definition for counting the number of crime-enterprises is bound to be invalidated by the fluidity of reality. During my research it sometimes happened that I thought to analyse one enterprise which split into two or more while doing my research. The reverse also happened: different crime-enterprises cooperated so closely that the police thought they were investigating one organization. Counting organized crime is like counting sandbanks in the North Sea…there is a constant shift in composition, shape, and size.

Apart from the difficulty involved in documenting their number, van Duyne notes that the determination of the threat to society posed by these enterprises is quite subjective and a questionable measure, as most of the entrepreneurs running them have no intention of challenging the existing social order. In any event, any measure of this type would need to consider the quality of the threat and extent of harm posed by each criminal organization, rather than a simple tally of these enterprises. Furthermore, quantifying the number and activities of criminal enterprises would still fail to take into consideration those activities of OC groups that are undertaken through legitimate businesses. Moreover, as Maltz (1990:25) notes, it is difficult to ascertain the extent to which activities usually associated with OC, such as gambling or prostitution, are run by criminal syndicates as opposed to individual entrepreneurs.

A related measure of program success is its impact on OC groups. In other words, has a given control effort disrupted a criminal organization? Measuring disruption entails detailed and accurate knowledge of an organization. Even where disruption can be documented, one must attend to the possibility that other groups have emerged to fill the void left by the weakened organization.

Efficiency of Criminal Justice Units

Blakey, Goldstock, and Rogovin (1978:55) take the view that fairly modest goals should be set in evaluating efforts to control OC. They acknowledge that attributing changes in crime patterns to modifications in the functioning of the justice system may be overly ambitious, as too many other factors may play an important role in those changes. They therefore propose that the efficiency of criminal justice agencies/units, rather than their impact, should be the focus of evaluative efforts. One can determine, for example, the relation between the inputs (resources) and outputs (arrests, convictions) of the unit. Modest goals can be established for parts of the system and efforts can be undertaken to ascertain how well the unit is working to achieve these goals. Examples include the ability of a unit to gather evidence, the amount of information developed on OC, the number of requests for assistance by local prosecutors, and the number of cases handled by specialized units that other law enforcement agencies could not handle (Maltz, 1990:7-8). Blakey and his colleagues also recommend that OC control units should undergo continuous internal review, as well as periodic external evaluation.

The need for external evaluations based on clear agency objectives has also been emphasized by Beare (1996:185). She cites the conclusion of reports by the United States’ General Accounting Office and various task forces that most evaluations of OC control efforts have been self-evaluations by agencies. These evaluations have tended to focus on administrative data rather than on the impact of agency activities. Furthermore, she adds that even where agencies set clear objectives, they seldom create oversight committees that can steer agency activities toward the objectives.

While the measures of efficiency recommended by Blakey and his colleagues set more modest (and perhaps more realistic) objectives for evaluations, they are plagued by all the above-mentioned shortcomings of administrative body-counts. One would think that the ultimate objective is impact, rather than mere efficiency, however difficult the measure of impact may be.

Harms Produced by Organized Crime

Maltz (1990) calls for harm-based measures of OC. Thus, assessments of OC control efforts should be based on the extent to which physical, economic, psychological, community, and societal harms, engendered by OC, have been mitigated. According to Maltz, evaluations might measure the volume of harm occurring in a jurisdiction on an annual basis. Before a reduction in the overall harm is attributed to a specific agency’s control efforts, the difficult task of determining the contribution of other factors must be undertaken. Other factors that may play a role in harm reduction include: the efforts of other agencies; natural changes in supply or demand (e.g., changing drug preferences, cocaine crop failure); non-enforcement phenomena (e.g., the death of a key OC figure); measurement error (e.g., less information available on a given activity than in the previous year); unknown sources; and random fluctuations (Maltz, 1990:20).

Measuring the harms produced by OC is more complex than in the case of conventional crimes such as robbery. Victims frequently report such conventional crimes to the police or mention them in victimization surveys. By contrast, the recipients of the illicit goods and services furnished by criminal organizations act voluntarily and, hence, do not view themselves as victims. Furthermore, the victims of non-consensual OC activities (e.g., extortion) often stay quiet due to intimidation.

This harm-based approach is intuitively compelling as the ultimate goal of more broadly based OC control efforts is to minimize the adverse effects of criminal organizations. Whether the aim of a particular program is to "decapitate" a criminal organization or to reduce the supply of an illicit substance or service, achieving these aims is of limited value if the harms produced by the organization or substance are unabated. Measuring harms, however, is no simple task.

The first category of harm defined by Maltz (1990) is physical harm. Even this, the most concrete of all harms, is difficult to measure with accuracy. The counting of acts of violence is impeded by a substantial "dark figure". Even homicide, purportedly the most accurate barometer of violence, is counted with a significant degree of error (Gabor, Hung, Mihorean, and St. Onge, 2002). Questions arise as to whether a violent death was due to an attack, was self-inflicted deliberately, or was unintentional. Also, the assessment of whether a homicide was a "gangland slaying" is not always clear.

Measuring economic harms incurred by victims is also complex. The direct damage done to a commercial building that has been burned down can be quantified quite readily. Calculating the secondary effects--the loss of a business, need for added security, higher insurance premiums, loss of wages, and other, more enduring effects—is a more difficult undertaking. Also, the victim in this case would include the owners and occupants of the building, employees, as well as the surrounding neighbourhood. Also, do we classify those who voluntarily purchase illicit goods and services as victims in tallying the amount of economic harm produced by OC?

Intimidation and fear are staples of OC. However, quantifying the number of people experiencing psychological harm, as well as its varying intensity, is daunting. There is a substantial "dark figure" of business people, witnesses, and offenders who face such intimidation in any jurisdiction.

OC can engender considerable community harm. Arson can blight an area, protection rackets can damage the business community, and drug dealing or prostitution can contribute to the community’s disintegration. Maltz (1990:45) recommends surveying community residents and business people to quantify the extent of this harm. However, people may have no idea as to the extent to which the behaviour in question is attributable to OC.

Societal harm involves threats to the governance of a country or region. This might be ascertained through public surveys probing the corruption of public officials and law enforcement. However, public perceptions may be inaccurate and unduly influenced by disclosures in the media. They may also vary significantly based on local conditions. Ironically, those neighbourhoods that are most affected by OC may be least likely to provide information, leading to the erroneous conclusion that the most affected areas are the least affected (Maltz, 1990:46).

Other Measures

Other measures proposed in assessments of OC control efforts have included seized assets, changes in the number of businesses or industries infiltrated, and changes in the level of public support for OC. The amount of seized assets is likely to constitute a very small proportion of a criminal organization’s total assets and reflects enforcement activities themselves. Money-laundering practices, too, give us little indication of the proceeds of OC, as this type of concealment may be a manoeuvre used by private citizens or legitimate enterprises to evade taxes, rather than an activity of OC (van Duyne, 1994).

With regard to measures of infiltration, a similar paradoxical situation may prevail as in the case of quantifying societal harm. That is, surveys of heavily infiltrated industries might yield less information on OC than those of sectors that are less affected. Finally, measures of changes in the public’s tolerance of OC can be useful; however, such changes may be based more on erroneous impressions of OC than on actual changes in OC’s activities.

Date modified: