Over the past several months I have been blogging about the emergence of Hal 900 like experiences in eDiscovery and Web 3.0. As such, I have been following the work of Dr. Herb Roitblat regarding the utilization of semantic and related technology to reduce the cost of document review through computer aided document categorization.

In a Blog posting on the Orcatec Blog, "Information Discovery", on Monday January 11, 2009 titled, "Computer Assisted Document Categorization in eDiscovery ", Dr. Roitblat announces that the January issue of the Journal of the American Society for Information Science and Technology, 61(1):1–11, 2010, has an article by Roitblat, Kershaw, and Oot describing a study that compared computer classification of eDiscovery documents with manual review. It found that computer classification was at least as consistent as human review was at distinguishing responsive from non responsive documents. If having attorneys review documents is a reasonable approach to identifying responsive documents, then any system that does as well as human review should also be considered a reasonable approach.

Dr. Roitblat reports in his Blog posting that the study compared an original categorization, done by contract attorneys in response to a Department of Justice Second Request with one done by two new human teams and two computer systems.

He further reported that the two re-review teams were employees of a service provider specializing in conducting legal reviews of this sort. Each team consisted of 5 reviewers who were experienced in the subject matter of this collection. The two teams independently reviewed a random sample of 5,000 documents. The two computer systems were provided by experienced eDiscovery service providers, one in California, and one in Texas.

The documents used in the study were collected in response to a "Second Request" concerning Verizon's acquisition of MCI. The documents were collected from 83 employees in 10 US states. Together they consisted of 1.3 terabytes of electronic files in the form of 2,319,346 documents. The collection consisted of about 1.5 million email messages, 300,000 loose files, and 600,000 scanned documents. After eliminating duplicates, 1,600,047 items were submitted for review. The attorneys spent about four months, seven days a week, and 16 hours per day on the review at a total cost of $13,598,872.61 or about $8.50 per document. After review, a total of 176,440 items were produced to the Justice Department.Accuracy was measured as agreement with the decisions made by the original review team. The level of agreement between the two human review teams was also measured. The two re-review teams identified a greater proportion of the documents as responsive than did the original review. Overall, their decisions agree with the original review on 75.6% and 72.0% of the documents. The two teams agreed with one another on about 70% of the documents.About half of the documents that were identified as responsive by the original review were identified as responsive by either of the re-review teams. Conversely, about a quarter of the documents identified as nonresponsive by the original review were identified as responsive by the new teams. Although the original review and the re-reviews were conducted by comparable people with comparable skills, their level of agreement was only moderate.

Dr. Roitblat reported that they did not know whether this was due to variability in the original review, or was due to some other factor, but these results are comparable to those seen in other situations where people make independent judgments about the categorization of documents (for example, in the TREC studies). A senior attorney reclassified the documents on which the two teams disagreed. After this reclassification, the level of agreement between this adjudicated set and the original review rose to 80%. The two computer systems identified fewer documents as responsive than did the human review teams, but still a bit more than were identified by the original review.

One system agreed with the original classification on 83.2% of the documents and the other on 83.6%. Like the human review teams, about half of the documents identified as responsive by the original review were similarly classified by the computer systems.

Dr, Roitblat commented that, "As legal professionals search for ways to reduce the costs of eDiscovery, this study suggests that it may be reasonable to employ computer-based categorization. The two computer systems agreed with the original review at least as often as a human team did. "

If this is indeed true, the laborious and expensive task of document review could indeed be in the process of evolving to the next level of computer aided automation. Add in come simple computer aided voice technology and a Hal 900 experience may not be that far away.

For more information about this paper or to discuss computer aided document categorization, you may contact Dr. Roitblat at: herb@orcatec.com.