Below, I discuss the units of observation used within existing research, the sources generally employed for data and then provide information on the datasets themselves. Please report inactive links.
Units of Observation - To investigate state repression and human rights violation, repression scholars have generally taken two approaches to data collection. Some compile information on discrete events, identifying how many times authorities arrest or execute citizens and impose curfews (Ball et al. 1999, Davenport & Stam 2003, Taylor & Jodice 1983). Other researchers place events into discrete categories that represent some measure of repressive lethality and/or scope (Gurr 1993, Poe & Tate 1994, Shellman 2006, Valentino et al. 2004). Rather than trying to identify exactly how many times an event occurred, these scholars identify how much something has taken place as well as other characteristics that could not simultaneously be addressed within an event count. They ask, for example, how many individuals were sanctioned and by how many different repressive techniques, how lethal were the different activities applied, and how long did the after-effect endure? The former approach is useful because it is more closely attuned to the dynamic elements of state-dissident interactions (i.e., how repression and dissent are actually experienced). The latter approach is useful because it allows a more holistic/encompassing approach to relevant behavior incorporating occurrence, scope and severity at the same time.
Sources - The information used to create datasets on repression can be divided by the categories identified above. For example, those interested in events have typically used newspapers. In many respects, the use of these sources makes sense, for it is commonly believed that they pay attention to the activities undertaken by authorities and especially those actions that were newsworthy (i.e., controversial, large-scale, bizarre, or violent). The selection also makes sense because many of the researchers who use newspapers are specifically interested in behavior that targeted protestors engaged in antistate activity—a controversial and newsworthy topic (Taylor&Jodice 1983). However, the perceptions about newspaper-generated data are not quite accurate (Davenport & Ball 2002, Jackman & Boyd 1979, Sommer & Scarritt 1999). Subjecting these data to a battery of statistical examinations,
researchers have identified three problems. First, there are “threshold” effects: Only events above a certain level of significance receive coverage, because they appeal to a larger market and cost less to cover than smaller events (McCarthy et al. 1996, Taylor & Jodice 1983). Second, there are “fatigue” effects: Only events of short duration receive coverage, in an effort to hold costs down (Gerner & Schrodt 1996). Third, there are “newshole” effects: Coverage is determined by the amount of available space within the pages of a newspaper (Honig et al. 1991). The use of newswires (Bond et al. 1997, Francisco 2000) and local sources (Oliver & Myers 1999) overcomes many of these problems, but several remain.
In an effort to more completely identify the level of human rights violation, other scholars rely on non-governmental organizations (NGOs) that are more specifically focused on repressive behavior and that use a different approach to discover “who did what to whom.” Specifically, they use human rights reports and government records compiled from eyewitness accounts (Ball et al. 1999, Davenport & Stam 2003, Gibney & Dalton 1996, Poe & Tate 1994, Restrepo et al. 2004). The information contained in these sources has proven extremely useful because unlike news organizations, they are directly and exclusively interested in state coercion. Toward these ends, relevant organizations have well-developed networks of communication both within and across countries. This directly aids them in overcoming many of the difficulties identified above.
However, the use of human rights NGO and government records has unexpectedly led to a different problem. These sources provide no information about the behavior of ordinary citizens or those specifically challenging the state. This is problematic because one of the principal explanations for repressive behavior concerns nonstate political conflict (i.e., protest behavior and civil unrest). Measures for these activities end up still relying on newspaper accounts. What is needed is something equivalent to the human rights–oriented NGO but with an interest in dissent and insurgency. Such information is frequently provided by governments (like the U.S. records on Iraqi dissident activity), but the question remains whether these event catalogs are accurate and to what extent they are used for purposes beyond identifying who did what to whom.
An alternative source is surveys of former insurgent combatants (Humphreys & Weinstein 2004, 2006) and government agents (Davenport & Stam 2003, Kalyvas 2006). Here, individuals who have been victimized by or fighting against repression, as well as those directly engaged in state coercive behavior, are asked questions about who did what to whom (and why). Though useful for generating information about what authorities did, the utility of this approach is limited for many of the issues pursued by repression scholars. For example, surveys are not likely to generate detailed event sequences across time and space, hindering the ability to gauge change in the approach to political order. They are more likely to be concentrated on specific times and places. Even if individuals can remember what occurred, it is unclear why they would talk about what they did, especially in the case of government actions. Additionally, to verify the testimonies, one would need to develop a very sophisticated method of fact-checking and cross-referencing (Ball 1996)—something that few scholars are prepared to do or interested in doing.
Datasets - Drawing on these different concepts of relevant activity and source material, numerous databases on state repression have been created. Essentially, three distinct types exist related to the geographic scope of the effort.
In the dominant orientation, the focus is global (Bollen 1986, Bond et al. 1997,
Cingranelli & Richards 2004, Fein 1995, Gibney & Dalton 1996, Gurr 1993, Harff 2003, Hathaway 2002, Henderson 1991, Karatnycky 1999, Krain 1997, Mitchell & McCormick 1988, Poe & Tate 1994, Taylor & Jodice 1983, Valentino et al. 2004). Here, effort is extended to identify and explain state repressive behavior within all or most countries of the world, generally by the nation-year. There are some differences with regard to the type of behavior highlighted. Some researchers focus on only one type of repression; e.g., Taylor & Jodice focus on negative sanctions, Harff on genocide/politicide, the Uppsala Conflict Data Program on “one-sided” state violence, Hathaway on torture. Some focus on a wide variety of activities placed together in some index: e.g., Gibney & Tate, Poe & Tate, Mitchell & McCormick, and Cingranelli & Richards. Other researchers focus on a wide variety of activities but disaggregate the components so that individuals could combine them as they see fit (Cingranelli & Richards 2004). Still others do not focus on political violence explicitly but allow the occurrence of this behavior to influence their coding of other repressive activity (e.g., Freedom House).
Several researchers focus on specific geographic regions, e.g., Duff et al. (1976) on Latin America and Francisco (2005) on Europe, or a subset of countries based on some criteria (e.g., Davis & Moore 1995). By reducing the number of countries under examination, these scholars have endeavored to disaggregate space (identifying states, cities, and villages), time (identifying year, month, and day), actors (identifying perpetrators and victims), and actions (identifying categories of action, frequency, objectives, outcomes, and so forth).
A growing number of researchers focus on single countries: e.g., Colombia (Restrepo et al. 2004), El Salvador (Ball 2006), France (Tilly 1995), Germany (Koopmans 1995), Greece (Kalyvas 2006), Guatemala (Ball et al. 1999, Davenport & Ball 2002), Kosovo (Ball 2006), Rwanda (Davenport & Stam 2003), Sri Lanka (Ball 2006), and the United States (Davenport 2005, Earl et al. 2004, Gibson 1988). These efforts have provided the greatest degree of disaggregation across both space and time, allowing analyses of who did what to whom by community, neighborhood, village, and city as well as by quarter, month, week, day, and hour. Because these efforts are the most recent, they have not yet influenced existing research (i.e., how we think about as well as study the topic of interest), but they do represent the next generation of repression research.
Data Links
(Please Report Any Problems and inactive links)
Below, I have attempted to be as comprehensive as possible regarding those efforts to systematically identify state repression and human rights violation. While there are several other projects/databases that exist (e.g., compilations held at the Inter-University Consortium for Political and Social Research - which tends to house older efforts), I have only listed those projects/databases that have been employed by the scholarly community in peer-reviewed publications as well as those that can be directly downloaded by those interested in the topic. If it has not been used in a venue where the quality has not been assessed and it cannot be downloaded, it is not provided here. As new databases become available, I will provide more information. If you have any suggestions for inclusion or general comments, please send them to me. Let the buyer/consumer/curious observer beware. Care should be taken with all data and these are no exceptions; read carefully.
Global Coverage
CIRI Human Rights Data Project (1981-2012)
David Cingranelli and David Richards
Freedom in the World (1976-2012) - Kind of difficult to find the data download
Freedom House
Genocide & Politicide (1955-2012) - Not the easiest thing to track down
Barbara Harff/Political Instability Task Force/Center for Systemic Peace
Global Data on Events, Location and Tone (1979-2014)
Kalev Leetaru, Philip Schrodt, Patrick Brandt and John Beieler
Political Terror Scale (1976-2012)
Mark Gibney
UCDP One-Sided Violence Dataset (1989-2005)
UCDP
UCDP/PRIO Armed Conflict Dataset (1946-2009)
UCDP/PRIO
Worldwide Atrocities Dataset (1995-2007)
Political Instability Task Force/KEDS
World Freedom Atlas (1990-2006) - Mapping Program
Zachary Forest Johnson
Worldwide Press Freedom Index (2002-2008)
Reporters without Borders
Regional Coverage
Social Conflict in Africa Database (1990-2011)
Cullen S. Hendrix and Idean Salehyan
European Protest and Coercion (1980-1995)
Ron Francisco
African - New Violence Data (1970-1995)
Africa Research Program
Selective Coverage
Democide: Murder by Government (Historical)
Rudolph J. Rummel
Intranational Political Interactions Project (1979-1992)
David Davis and Will H. Moore
The Kansas Event Data System (KEDS)
Deborah “Misty” Gerner and Phill Schrodt
Minorities at Risk (1945-2006)
CIDCM
The Post-Internal War Accommodation and Repression (PIWAR)
Matthew Krain
Mass Killing During Wars (1945-2000)
Benjamin Valentino, Paul Huth and Dylan Balch-Lindsey
Draining the Sea Valentino et al
Single Country Coverage
Guatemala (1960-1996)
International Center for Human Rights Research
Kosovo (1999)
Human Rights Data Analysis Group - Benetech
Northern Ireland (1969-2007)
Malcolm Sutton Index of Deaths From the Conflict in Ireland
Rwanda (1994)
Christian Davenport and Allan Stam
Sierra Leone (1991-2000)
Human Rights Data Analysis Group - Benetech
Timor-Leste (1974-1999)
Human Rights Data Analysis Group - Benetech
United States
Dynamics of Collective Action, 1960-1995
Sarah Soule, Susan Olzak, John McCarthy and Doug McAdam
vs. the Black Panthers (1967-1973)
Christian Davenport
vs. the Republic of New Africa (1968-1974)
Christian Davenport
A New Initiative (Still Under Development)
Historically, information about diverse forms of socio-political behavior such as violence, conflict and crime have been collected and distributed by governments, journalists and more recently human rights organizations. This information has provided a large number of keen insights but these sources also involve a large number of limitations as well. Despite the reliance upon these sources, almost all information about social, political and economic life relies upon information provided by witnesses, victims and perpetrators of relevant phenomenon but these are frequently not collected in a rigorous manner nor is it always easy for these individuals to provide the information that they have.
The Illustrative Information Interface (or III) is a program created to facilitate an alternative way to collect, describe, archive and analyze relevant information. The first phase of the project concerns human rights violation/state repression (i.e., coercive activities directed by political authorities against those under their jurisdiction for the purposes of influencing behavior/attitudes and/or eliminating those perceived to be threatening). Other forms of social, political and economic behavior will be considered in the future (e.g., democracy, sexual violence and environmental pollution).
As designed, III allows individuals from all walks of life to provide as well as use relevant information in an easy, user-friendly manner on the internet or via some hand-held device. Entries take about 10-15 minutes to complete. In the program, victims, witnesses, refugees, scholars, students, travelers as well as ngos, journalists, and government personnel can give information about what they have heard, seen or read about. This information can then be used by others who are interested in the relevant topic to increase their awareness, conduct analyses, compare against other data sources, engage in advocacy or inform their conversations. Ready to go?
It should be clear the Illustrative Information Interface:
- is not simply concerned with the present or the recent but also what has taken place in the past (from 1900 to the present);
- does not attempt to take a position on the activities in question but merely to facilitate the compilation of this information in a rigorous manner;
- is as transparent as possible regarding the collection and distribution of all information;
- is protective of all those who submit information, never revealing sources but allowing individuals to select whom they wish to rely upon (e.g., specific age groups, genders, levels of education, time spent within the country of interest and so forth); and,
- is only as good as those who contribute to it.
With this I invite you to participate, use, learn, suggest, advocate and assist. The webpage here will provide some basic idea of how III works, answers some questions that have arisen about the project and provides some links to related concepts. The webpage for the Illustrative Information Interface is just for the program itself and does not contain most of what is provided on this page. If you have any questions, comments or concerns, please feel free to contact me.
In Memoriam: Steven Poe
Units of Observation - To investigate state repression and human rights violation, repression scholars have generally taken two approaches to data collection. Some compile information on discrete events, identifying how many times authorities arrest or execute citizens and impose curfews (Ball et al. 1999, Davenport & Stam 2003, Taylor & Jodice 1983). Other researchers place events into discrete categories that represent some measure of repressive lethality and/or scope (Gurr 1993, Poe & Tate 1994, Shellman 2006, Valentino et al. 2004). Rather than trying to identify exactly how many times an event occurred, these scholars identify how much something has taken place as well as other characteristics that could not simultaneously be addressed within an event count. They ask, for example, how many individuals were sanctioned and by how many different repressive techniques, how lethal were the different activities applied, and how long did the after-effect endure? The former approach is useful because it is more closely attuned to the dynamic elements of state-dissident interactions (i.e., how repression and dissent are actually experienced). The latter approach is useful because it allows a more holistic/encompassing approach to relevant behavior incorporating occurrence, scope and severity at the same time.
Sources - The information used to create datasets on repression can be divided by the categories identified above. For example, those interested in events have typically used newspapers. In many respects, the use of these sources makes sense, for it is commonly believed that they pay attention to the activities undertaken by authorities and especially those actions that were newsworthy (i.e., controversial, large-scale, bizarre, or violent). The selection also makes sense because many of the researchers who use newspapers are specifically interested in behavior that targeted protestors engaged in antistate activity—a controversial and newsworthy topic (Taylor&Jodice 1983). However, the perceptions about newspaper-generated data are not quite accurate (Davenport & Ball 2002, Jackman & Boyd 1979, Sommer & Scarritt 1999). Subjecting these data to a battery of statistical examinations,
researchers have identified three problems. First, there are “threshold” effects: Only events above a certain level of significance receive coverage, because they appeal to a larger market and cost less to cover than smaller events (McCarthy et al. 1996, Taylor & Jodice 1983). Second, there are “fatigue” effects: Only events of short duration receive coverage, in an effort to hold costs down (Gerner & Schrodt 1996). Third, there are “newshole” effects: Coverage is determined by the amount of available space within the pages of a newspaper (Honig et al. 1991). The use of newswires (Bond et al. 1997, Francisco 2000) and local sources (Oliver & Myers 1999) overcomes many of these problems, but several remain.
In an effort to more completely identify the level of human rights violation, other scholars rely on non-governmental organizations (NGOs) that are more specifically focused on repressive behavior and that use a different approach to discover “who did what to whom.” Specifically, they use human rights reports and government records compiled from eyewitness accounts (Ball et al. 1999, Davenport & Stam 2003, Gibney & Dalton 1996, Poe & Tate 1994, Restrepo et al. 2004). The information contained in these sources has proven extremely useful because unlike news organizations, they are directly and exclusively interested in state coercion. Toward these ends, relevant organizations have well-developed networks of communication both within and across countries. This directly aids them in overcoming many of the difficulties identified above.
However, the use of human rights NGO and government records has unexpectedly led to a different problem. These sources provide no information about the behavior of ordinary citizens or those specifically challenging the state. This is problematic because one of the principal explanations for repressive behavior concerns nonstate political conflict (i.e., protest behavior and civil unrest). Measures for these activities end up still relying on newspaper accounts. What is needed is something equivalent to the human rights–oriented NGO but with an interest in dissent and insurgency. Such information is frequently provided by governments (like the U.S. records on Iraqi dissident activity), but the question remains whether these event catalogs are accurate and to what extent they are used for purposes beyond identifying who did what to whom.
An alternative source is surveys of former insurgent combatants (Humphreys & Weinstein 2004, 2006) and government agents (Davenport & Stam 2003, Kalyvas 2006). Here, individuals who have been victimized by or fighting against repression, as well as those directly engaged in state coercive behavior, are asked questions about who did what to whom (and why). Though useful for generating information about what authorities did, the utility of this approach is limited for many of the issues pursued by repression scholars. For example, surveys are not likely to generate detailed event sequences across time and space, hindering the ability to gauge change in the approach to political order. They are more likely to be concentrated on specific times and places. Even if individuals can remember what occurred, it is unclear why they would talk about what they did, especially in the case of government actions. Additionally, to verify the testimonies, one would need to develop a very sophisticated method of fact-checking and cross-referencing (Ball 1996)—something that few scholars are prepared to do or interested in doing.
Datasets - Drawing on these different concepts of relevant activity and source material, numerous databases on state repression have been created. Essentially, three distinct types exist related to the geographic scope of the effort.
In the dominant orientation, the focus is global (Bollen 1986, Bond et al. 1997,
Cingranelli & Richards 2004, Fein 1995, Gibney & Dalton 1996, Gurr 1993, Harff 2003, Hathaway 2002, Henderson 1991, Karatnycky 1999, Krain 1997, Mitchell & McCormick 1988, Poe & Tate 1994, Taylor & Jodice 1983, Valentino et al. 2004). Here, effort is extended to identify and explain state repressive behavior within all or most countries of the world, generally by the nation-year. There are some differences with regard to the type of behavior highlighted. Some researchers focus on only one type of repression; e.g., Taylor & Jodice focus on negative sanctions, Harff on genocide/politicide, the Uppsala Conflict Data Program on “one-sided” state violence, Hathaway on torture. Some focus on a wide variety of activities placed together in some index: e.g., Gibney & Tate, Poe & Tate, Mitchell & McCormick, and Cingranelli & Richards. Other researchers focus on a wide variety of activities but disaggregate the components so that individuals could combine them as they see fit (Cingranelli & Richards 2004). Still others do not focus on political violence explicitly but allow the occurrence of this behavior to influence their coding of other repressive activity (e.g., Freedom House).
Several researchers focus on specific geographic regions, e.g., Duff et al. (1976) on Latin America and Francisco (2005) on Europe, or a subset of countries based on some criteria (e.g., Davis & Moore 1995). By reducing the number of countries under examination, these scholars have endeavored to disaggregate space (identifying states, cities, and villages), time (identifying year, month, and day), actors (identifying perpetrators and victims), and actions (identifying categories of action, frequency, objectives, outcomes, and so forth).
A growing number of researchers focus on single countries: e.g., Colombia (Restrepo et al. 2004), El Salvador (Ball 2006), France (Tilly 1995), Germany (Koopmans 1995), Greece (Kalyvas 2006), Guatemala (Ball et al. 1999, Davenport & Ball 2002), Kosovo (Ball 2006), Rwanda (Davenport & Stam 2003), Sri Lanka (Ball 2006), and the United States (Davenport 2005, Earl et al. 2004, Gibson 1988). These efforts have provided the greatest degree of disaggregation across both space and time, allowing analyses of who did what to whom by community, neighborhood, village, and city as well as by quarter, month, week, day, and hour. Because these efforts are the most recent, they have not yet influenced existing research (i.e., how we think about as well as study the topic of interest), but they do represent the next generation of repression research.
Data Links
(Please Report Any Problems and inactive links)
Below, I have attempted to be as comprehensive as possible regarding those efforts to systematically identify state repression and human rights violation. While there are several other projects/databases that exist (e.g., compilations held at the Inter-University Consortium for Political and Social Research - which tends to house older efforts), I have only listed those projects/databases that have been employed by the scholarly community in peer-reviewed publications as well as those that can be directly downloaded by those interested in the topic. If it has not been used in a venue where the quality has not been assessed and it cannot be downloaded, it is not provided here. As new databases become available, I will provide more information. If you have any suggestions for inclusion or general comments, please send them to me. Let the buyer/consumer/curious observer beware. Care should be taken with all data and these are no exceptions; read carefully.
Global Coverage
CIRI Human Rights Data Project (1981-2012)
David Cingranelli and David Richards
Freedom in the World (1976-2012) - Kind of difficult to find the data download
Freedom House
Genocide & Politicide (1955-2012) - Not the easiest thing to track down
Barbara Harff/Political Instability Task Force/Center for Systemic Peace
Global Data on Events, Location and Tone (1979-2014)
Kalev Leetaru, Philip Schrodt, Patrick Brandt and John Beieler
Political Terror Scale (1976-2012)
Mark Gibney
UCDP One-Sided Violence Dataset (1989-2005)
UCDP
UCDP/PRIO Armed Conflict Dataset (1946-2009)
UCDP/PRIO
Worldwide Atrocities Dataset (1995-2007)
Political Instability Task Force/KEDS
World Freedom Atlas (1990-2006) - Mapping Program
Zachary Forest Johnson
Worldwide Press Freedom Index (2002-2008)
Reporters without Borders
Regional Coverage
Social Conflict in Africa Database (1990-2011)
Cullen S. Hendrix and Idean Salehyan
European Protest and Coercion (1980-1995)
Ron Francisco
African - New Violence Data (1970-1995)
Africa Research Program
Selective Coverage
Democide: Murder by Government (Historical)
Rudolph J. Rummel
Intranational Political Interactions Project (1979-1992)
David Davis and Will H. Moore
The Kansas Event Data System (KEDS)
Deborah “Misty” Gerner and Phill Schrodt
Minorities at Risk (1945-2006)
CIDCM
The Post-Internal War Accommodation and Repression (PIWAR)
Matthew Krain
Mass Killing During Wars (1945-2000)
Benjamin Valentino, Paul Huth and Dylan Balch-Lindsey
Draining the Sea Valentino et al
Single Country Coverage
Guatemala (1960-1996)
International Center for Human Rights Research
Kosovo (1999)
Human Rights Data Analysis Group - Benetech
Northern Ireland (1969-2007)
Malcolm Sutton Index of Deaths From the Conflict in Ireland
Rwanda (1994)
Christian Davenport and Allan Stam
Sierra Leone (1991-2000)
Human Rights Data Analysis Group - Benetech
Timor-Leste (1974-1999)
Human Rights Data Analysis Group - Benetech
United States
Dynamics of Collective Action, 1960-1995
Sarah Soule, Susan Olzak, John McCarthy and Doug McAdam
vs. the Black Panthers (1967-1973)
Christian Davenport
vs. the Republic of New Africa (1968-1974)
Christian Davenport
A New Initiative (Still Under Development)
Historically, information about diverse forms of socio-political behavior such as violence, conflict and crime have been collected and distributed by governments, journalists and more recently human rights organizations. This information has provided a large number of keen insights but these sources also involve a large number of limitations as well. Despite the reliance upon these sources, almost all information about social, political and economic life relies upon information provided by witnesses, victims and perpetrators of relevant phenomenon but these are frequently not collected in a rigorous manner nor is it always easy for these individuals to provide the information that they have.
The Illustrative Information Interface (or III) is a program created to facilitate an alternative way to collect, describe, archive and analyze relevant information. The first phase of the project concerns human rights violation/state repression (i.e., coercive activities directed by political authorities against those under their jurisdiction for the purposes of influencing behavior/attitudes and/or eliminating those perceived to be threatening). Other forms of social, political and economic behavior will be considered in the future (e.g., democracy, sexual violence and environmental pollution).
As designed, III allows individuals from all walks of life to provide as well as use relevant information in an easy, user-friendly manner on the internet or via some hand-held device. Entries take about 10-15 minutes to complete. In the program, victims, witnesses, refugees, scholars, students, travelers as well as ngos, journalists, and government personnel can give information about what they have heard, seen or read about. This information can then be used by others who are interested in the relevant topic to increase their awareness, conduct analyses, compare against other data sources, engage in advocacy or inform their conversations. Ready to go?
It should be clear the Illustrative Information Interface:
- is not simply concerned with the present or the recent but also what has taken place in the past (from 1900 to the present);
- does not attempt to take a position on the activities in question but merely to facilitate the compilation of this information in a rigorous manner;
- is as transparent as possible regarding the collection and distribution of all information;
- is protective of all those who submit information, never revealing sources but allowing individuals to select whom they wish to rely upon (e.g., specific age groups, genders, levels of education, time spent within the country of interest and so forth); and,
- is only as good as those who contribute to it.
With this I invite you to participate, use, learn, suggest, advocate and assist. The webpage here will provide some basic idea of how III works, answers some questions that have arisen about the project and provides some links to related concepts. The webpage for the Illustrative Information Interface is just for the program itself and does not contain most of what is provided on this page. If you have any questions, comments or concerns, please feel free to contact me.
In Memoriam: Steven Poe
