Assessing the Validity of Primary-source Death Count Reporting: the Iraq Body Count and Benford’s Law
A Forensic Analysis of Incident-Based Tracking of Civilian Deaths in Conflicts
The following is a rough draft of a paper I am writing assessing the viability of Primary-source Death Count Reporting (PDCR) as a means to accurately count civilian deaths in conflicts. I utilize Benford’s Law to determine the validity of the Iraq Body Count project’s data.
Assessing the Validity of Primary-source Death Count Reporting: the Iraq Body Count and Benford’s Law
Benjamin Sylvia
Abstract
The availability of internet technologies has enabled the usage of Primary-source Death Count Reporting (PDCR) for tracking civilian casualties in conflicts. Information on civilian deaths gathered in this way have been critiqued for their inability to provide accurate total estimates, but have not yet had the data they produce on individual incidents of civilian casualties forensically examined to determine its validity. Ensuring that the data gathered from PDCR methods is reliable is critical if this data is to have any practical application. The use of the mathematical phenomenon known as Benford’s Law demonstrated that the data of the most well-known PDCR effort, the Iraq Body Count project, did not conform to Benford’s Law and is likely fraudulent or inaccurate.
Section 1: Introduction
The 2022 Russian invasion of Ukraine has been defined in large part by the targeting of Ukrainian civilians by Russian forces. The proliferation of real-time battlefield information available via the internet, especially over social media, has allowed for an unprecedented effort to track and document these civilian casualties. The Office of the UN High Commissioner for Human Rights’ (OHCHR) UN Human Rights Monitoring Mission in Ukraine (HRMMU) has attempted to utilize data collected from interviews with victims and their relatives, direct witnesses, official records, open source documents, photos, video materials, forensic records and reports, criminal investigation materials, court documents, reports from international and national non-governmental organizations, public reports by law enforcement and military actors, and data from medical facilities and local authorities to produce a running total of civilian casualties in this conflict. The OHCHR has claimed that it believes that the actual numbers of civilian casualties are higher and that their figures are likely an undercount due to a delay in the receipt of information from some areas of the conflict and the time delay associated with corroborating other reports. The HRMMU’s effort to track the civilian casualties of the 2022 Russia-Ukraine war represents the highest profile effort to track civilian casualty deaths in recent memory. The benefits of internet technologies to provide quick and accurate reports of incidents of civilian deaths is a relatively new phenomenon that has lent additional credence to the viability of the method of incident-based death count reporting, or what has sometimes been referred to as tally reporting or more simply ‘recording’, but will hereafter be referred to as Primary-source Death Count Reporting (PDCR).
1.1: Primary-source Death Count Reporting (PDCR)
PDCR and ‘estimation’ are the two methods most frequently utilized to determine the number of violent civilian deaths in a conflict. Estimation relies on statistical techniques to produce overall estimates of the number of civilian deaths in a conflict, and as such can result in a wide range of total death-counts depending on the methodology used and the organization performing the research. This stands in contrast to PDCR, a relatively new method that describes attempts to compile a list of all civilian casualties in a conflict by tracking those deaths as they appear in eyewitness accounts, media reports, hospital and morgue records, NGO accounts, official figures, etc. and then recording and verifying these accounts. One of the main benefits of this method of tracking civilian deaths is that it allows for the documentation of factual circumstances of civilian deaths in a way that is fully verifiable. One of the main criticisms of this approach is that it cannot provide an accurate total estimate of the number of civilian deaths in a conflict as some of these deaths may go unreported, resulting in what is generally accepted to be an undercount of the total deaths. The major purported benefit of this approach is its accuracy in producing factual, verified data. This assertion has to date not been substantively examined for two main reasons. First, scholars have argued that it is impossible to assess the degree of distortion present in only a single dataset. Second, the political focus on total death-counts in a conflict has directed criticism of PDCR towards the aforementioned critique, eschewing a focus on the validity of the data itself in favor of a focus on what has not been included in the data. While PDCR methods like that of the HRMMU have asserted that they are likely an undercount of the total, the totals they generate are still politically valuable as they appear to provide an accurate accounting of instances of civilian deaths, resulting in a seemingly reliable minimum number of deaths. Examining the validity of the actual data generated by PDCR methods and not their total counts is important if this methodology is to maintain its legitimacy as an accurate recording of factual, specific incidents of violent civilian deaths in a given conflict. To do so I will examine the most prominent example of this type of civilian death count reporting, the Iraq Body Count project (IBC).
1.2: The Iraq Body Count project (IBC)
The IBC is a non-profit organization under the British think tank Oxford Research Group that has tracked violent civilian casualty deaths in Iraq with PDCR methods since the beginning of the Iraq War in 2003. It draws its documentary evidence primarily from crosschecked media reports, but also utilizes data from hospitals, morgues, NGOs, and other official figures; this data is then reviewed by the IBC team and posted to their public database at iraqbodycount.org. Their stated priority was “to provide a robust baseline of verifiably recorded civilian deaths”, and the rationale they provide for utilizing PDCR methods to do so is that this methodology can help them provide the most accurate and detailed factual accounting of violent civilian deaths in Iraq. Their overall approach has been criticized most often for providing an inaccurate accounting of total civilian deaths; the IBC refutes this criticism by maintaining that they are not attempting to provide an estimated figure for total deaths but rather are curating “a record of actual, documented deaths”. As of yet, there has not been a forensic analysis of the validity of their data not on the premise that it is an undercount of total deaths but rather one assessing whether the data they present is itself accurate. The IBC asserts that the validity of their data can be tested by looking at other non-journalistic sources, as well as by comparing their data to survey samples and demographic data; however, this approach is insufficient as non-journalistic sources may also suffer from the same documentation problems as the IBC data and surveys and demographic data are useful only for evaluating the total figures generated by the IBC. Performing a forensic analysis of the IBC data to determine its validity is important if scholars and policymakers are to utilize this data and to establish the legitimacy of the PDCR methodology.
1.3: Rationale and Organization of this Paper
This paper proposes the use of Benford’s Law, a mathematical phenomenon, as a means to perform a forensic analysis of a single dataset of incidents of civilian deaths in a violent conflict that can independently assess the validity of the data without relying on fact-checking of individual incidents. To this end I will apply Benford’s Law (BL) to the IBC data to determine if it demonstrates conformity with the expected distributions of BL.
This paper is organized as follows: Section 2 provides an introduction to Benford’s Law and then demonstrates why Benford’s Law can be applied to the IBC data and civilian casualty reporting data more broadly. Section 3 describes the data profile and the types of tests performed on it. Section 4 provides the results of this analysis, which is then discussed in greater detail in Section 5. Section 6 contains the conclusions and implications for future PDCR efforts.
Section 2: Background
2.1: Introduction to Benford’s Law
Benford's Law describes a mathematical phenomenon whereby large sets of naturally-generated numerical data tend to have a greater proportion of numbers with a smaller leading digit than those with a larger one. This phenomenon also applies to the following digits as well as combinations of certain digits; however, Benford's Law of Leading Digits, which applies only to the first digit, is the most generally used and is often referred to simply as Benford’s Law. Benford’s Law of Leading Digits describes the expected distribution of a conforming set of numbers when the probability of observing a first digit d is approximately
From this, the expected frequencies of a certain digit, such as the first digit, can be derived from the following formula, where D1 represents the first digit and Prob indicates the probability of observing the event in parentheses:
This equation yields the expected proportions for each leading digit shown below in Table 1.
A data set should follow Benford’s Law if the data span several orders of magnitude, contain a significant number of entries, and are naturally occurring (where the records represent the sizes of facts or events with no built-in minimums or maximums). The exact amount of entries needed in a data set to evaluate Benford’s Law is disputed, with some applications demonstrating compliance with Benford’s Law in data sets of 50 to 100 entries; however, the general rule is that Benford’s Law works best in datasets larger than 100 entries.
Benford’s Law has gained prominence due its unique ability to detect anomalies in datasets. If a dataset demonstrates a weak level of conformity with Benford’s Law, there is a high risk that the data contains anomalies. In this way Benford’s Law provides for an objective ability to analyze a single dataset to look for anomalies that may be either intentional or unintentional. This principle has been used most famously to detect financial fraud but has also been applied to a number of other realms, some of which will be explored in Section 2.3.
2.2: IBC Data Profile
I utilized the Iraq Body Count project’s data for the purposes of this project, as it is the most well-known example of PDCR data and contains a large number of entries to analyze. It is also the first modern conflict with widespread internet and media coverage, making the applicability of its data gathering methods more relevant to the current moment.
The IBC data selected from the database contained 725 entries spanning from December 29, 2002 to December 25, 2016 that recorded the number of incidents of fatalities by week. This data was gathered from the IBC database and the following search qualifiers were used: The type of entry selected was “number of incidents”; the frequency selected was “per week”, the shortest frequency option; the time frame selected was “from 2003 to 2016”; the area of analysis selected was “All Iraq”; the perpetrators of the fatalities selected was “any perpetrators”, a category that included ‘US-led coalition, no Iraqi state forces, US-led coalition incl. Iraqi state forces, Iraqi state forces without coalition, anti-government/occupation forces, Islamic State of Iraq (ISIS, ISIL), and unknown actors”; the weapon(s) used by the perpetrators selected was “using any weapons”; and the amount of persons killed in an incident was set to “killing 1 or more”. The data set contains one incident outside of the 2003-2016 range (the 29DEC2002 incident) as this week encompasses the final week of 2002 as well as part of 2003. The time frame was set from 2003 to 2016 because the IBC data after this point has not been verified and are approximations. The data conditions specified above were chosen to provide the most representative sample that meets all of the qualifications of a data set that can be analyzed with Benford’s Law.
2.3: Applicability of Benford’s Law to PDCR
Theoretically, as long as a dataset meets the aforementioned requirements, it should be able to be analyzed with Benford’s Law. The IBC data meet the criteria because numbers of deaths is a naturally occurring phenomenon that represents the size of events and has no built-in minimums or maximums, the data span several orders of magnitude with the smallest entry at 1 civilian casualty and the largest at 2,330 civilian casualties, and the data contain 725 entries, well above the generally accepted minimum of 100 entries. Another condition to evaluate whether the IBC data on civilian casualties can be analyzed with Benford’s Law is to examine other cases that have shown Benford’s Law to work on similar types of data. Multiple researchers have utilized Benford’s Law to evaluate the death-counts of persons dying from Covid-19, while Daniels et. al demonstrated how manner of death-counts of persons in the U.S. and Canada conform to Benford’s Law. These four works demonstrate that number of deaths, regardless of cause, are a phenomenon that can be analyzed with Benford’s Law. The IBC PDCR-gathered data are an extension of this principle in that it examines civilian deaths that are the result of violent conflicts.
Section 3: Methods
The primary test for Benfordian analysis is the first digit test, which allows for the comparison of the distribution of first digits in a data set with the expected Benfordian distribution. However, this test by itself is often too high-level to produce a valid analysis, as the first digits may show close conformity to the expected Benford distribution but the second and later digits could deviate significantly from their expected distributions. To avoid this problem the first digit test was supplemented with the first-two digit test, which is a combination of the first and second digit tests. Each of these has an expected distribution that the data’s actual distribution can be compared against (see Figure 1 in Appendix for full list of expected Benfordian distributions for first-two digit test). Two second order tests were used to evaluate the null and alternate hypothesis listed below:
H0: The IBC data conforms to Benford’s Law.
H1: The IBC data does not conform to Benford’s Law.
The Chi-square (X2) test was used for the goodness-of-fit analysis, and the mean absolute deviation (MAD) was also used to evaluate the degree of conformity with Benford’s Law. The critical values used to analyze the MAD from Benford’s Law were taken from Nigrini (2012) and are presented in Table 2 below.
Section 4: Results
The analysis of the IBC data revealed that the null hypothesis can be rejected and that the IBC data demonstrate nonconformity with Benford’s Law. Both tests demonstrated a visible deviation from Benford’s Law as demonstrated in the graphs in Picture 1 and Picture 2 below; both the Chi-square and MAD tests confirmed this result. The results of the tests are displayed below in Table 3.
PICTURE 1: First-Digit
PICTURE 2: First-Two Digits
The p-values for both the First Digit and First-Two Digits tests were both below the 95% confidence interval threshold of 0.05, demonstrating a rejection of Benford’s Law for the data. The MAD values for both tests similarly were above the thresholds of 0.015 and 0.0022 respectively that indicate nonconformity.
Section 5: Discussion
The IBC’s data lack of conformity to Benford’s Law lends itself to two possible explanations. First is that it does not match the expected Benfordian distribution because the data is not applicable to Benford’s Law; certain types of data may seem to meet all of the qualifications of data that can be analyzed with Benford’s Law but still not conform. We can reject this hypothesis because the aforementioned studies demonstrate that number of deaths is a phenomenon that can be expected to conform to Benfordian proportions. The second is that the data does not match the expected distribution because it is fraudulent or inaccurate.
5.1 Possible Explanations for Lack of Benfordian Conformity
The IBC data could be fraudulent because the sources the IBC uses to get their data from or the IBC staff themselves are intentionally manipulating it to either create the perception that the number of deaths from specific incidents and/or the total number of deaths is either higher or lower than it actually is. This could be beneficial for sources on the ground who want to understate the extent of incidents of death or inflate it out of fear of reprisals, in the hopes of generating more attention for themselves as individuals or the organization they represent, or simply to persuade the public to believe that the war is either worse or better than it actually is. The IBC might also have the incentive to intentionally distort their data due to their stated anti-Iraq War stance; two founders of the project, John Sloboda and Hamit Dardagan, wrote in 2013 that “IBC’s specific, immediate aim has always been ending the violence of the Iraq War”. Inflating the death counts of certain incidents or intentionally misinterpreting incidents in favor of higher numbers may benefit their anti-war cause.
It is also possible that the data has not been manipulated and is just inaccurate. There are inherent difficulties in conducting effective PDCR; lack of frequency, coverage, and quality of data can frustrate PDCR efforts. The IBC thus could be suffering from a lack of accurate data, as the sources they receive their data from may have themselves reported inaccurate data. The founders of the project claimed that there is likely to be less coverage of incidents in which three or fewer deaths occurred; they claim that this problem has been mitigated by supplementing these sources with other ‘aggregate data’, such as data from morgues and hospitals, but this effort may be futile if the ‘aggregate data’ is also inaccurate.
Another explanation is that the IBC team has had difficulties in interpreting the data. In 2004, all project personnel were independent citizens of the UK and U.S., and in 2005 the IBC published a dossier in which they stated that they used exclusively English-language sources as English was the only language in which all of their team members were fluent. A lack of utilization of Arabic and other language sources may have limited the amount of data the IBC was able to gather, interpret, and use to validate other sources, skewing their data.
Section 6: Conclusions
Because of the increasing prevalence and importance of PDCR methods in tracking deaths in conflicts like the Ukraine, this research aimed to assess the validity of one of the most prominent PDCR methods, the Iraq Body Count project, using Benford’s Law. The analysis showed that the IBC data did not conform to Benford’s Law, indicating a high probability that the IBC data is either fraudulent or inaccurate.
6.1: Implications for Future PDCR Efforts
This analysis has implications for future PDCR efforts in that it demonstrates the likely non-validity of the data of one of the most notable PDCR projects, the IBC. This means that in the future, organizations and researchers utilizing PDCR methods will need to closely examine the means by which they obtain and validate their data. This also demonstrates the usefulness of the application of Benford’s Law to PDCR efforts that meet the Benfordian requirements to test their validity.
References
“About IBC :: Iraq Body Count.” Accessed May 22, 2022. https://www.iraqbodycount.org/about/.
Campolieti, Michele. “COVID-19 Deaths in the USA: Benford’s Law and under-Reporting.” Journal of Public Health, May 24, 2021, fdab161. https://doi.org/10.1093/pubmed/fdab161.
Carmo, Carlos Roberto Souza, Fernando de Lima Caneppele, and Fábio Caixeta Nunes. “ANALYSIS OF COVID-19 CONTAMINATION AND DEATHS CASES IN BRAZIL ACCORDING TO THE NEWCOMB-BENFORD.” Brazilian Journal of Biometrics 39, no. 4 (December 3, 2021): 522–35. https://doi.org/10.28951/rbb.v39i4.535.
Collins, J. Carlton. “Using Excel and Benford’s Law to Detect Fraud.” Journal of Accountancy, April 1, 2017. https://www.journalofaccountancy.com/issues/2017/apr/excel-and-benfords-law-to-detect-fraud.html.
Danchev, Alex, and John MacMillan. The Iraq War and Democratic Politics. London, UNITED KINGDOM: Taylor & Francis Group, 2004. http://ebookcentral.proquest.com/lib/gmu/detail.action?docID=199596.
Daniels, Jeremy, Samantha-Jo Caetano, Dirk Huyer, Andrew Stephen, John Fernandes, Alice Lytwyn, and Fred M. Hoppe. “Benford’s Law for Quality Assurance of Manner of Death Counts in Small and Large Databases.” Journal of Forensic Sciences 62, no. 5 (2017): 1326–31. https://doi.org/10.1111/1556-4029.13437.
Davenport, Christian, Hank Johnston, and Carol McClurg Mueller. Repression and Mobilization. Social Movements, Protest, and Contention ; v. 21. Minneapolis: University of Minnesota Press, 2005.
Fewster, R. M. “A Simple Explanation of Benford’s Law.” The American Statistician 63, no. 1 (February 1, 2009): 26–32. https://doi.org/10.1198/tast.2009.0005.
“Https://Reports.Iraqbodycount.Org.” Accessed May 24, 2022. https://reports.iraqbodycount.org/a_dossier_of_civilian_casualties_2003-2005.pdf.
Molitor, Nuoo-Ting, Nicky Best, Chris Jackson, and Sylvia Richardson. “Using Bayesian Graphical Models to Model Biases in Observational Studies and to Combine Multiple Sources of Data: Application to Low Birth Weight and Water Disinfection by-Products.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 172, no. 3 (June 2009): 615–37. https://doi.org/10.1111/j.1467-985X.2008.00582.x.
Morillas-Jurado, Francisco Gabriel, this link will open in a new window Link to external site, María Caballer-Tarazona, and Vicent Caballer-Tarazona. “Applying Benford’s Law to Monitor Death Registration Data: A Management Tool for the COVID-19 Pandemic.” Mathematics 10, no. 1 (2022): 46. https://doi.org/10.3390/math10010046.
Nigrini, Mark J., and Joseph T. Wells. Benford’s Law: Applications for Forensic Accounting, Auditing, and Fraud Detection. Hoboken, UNITED STATES: John Wiley & Sons, Incorporated, 2012. http://ebookcentral.proquest.com/lib/gmu/detail.action?docID=821998.
Seybolt, Taylor B., Jay D. Aronson, and Baruch Fischhoff. Counting Civilian Casualties: An Introduction to Recording and Estimating Nonmilitary Deaths in Conflict. Studies in Strategic Peacebuilding. Oxford: University Press, 2013.
OHCHR. “Ukraine: Civilian Casualty Update 19 May 2022.” Accessed May 20, 2022. https://www.ohchr.org/en/news/2022/05/ukraine-civilian-casualty-update-19-may-2022.