Manual disclosure review ordered due to concerns over party's computer assisted review

In a recent High Court decision, the court expressed concerns regarding the claimants' computer assisted disclosure review and ordered them to carry out a manual review of a substantial number of documents that had been excluded from review on the basis of a computer assisted sampling of the documents: Triumph Controls UK v Primus International Holding [2018] EWHC 176 (TCC).

In a number of judgments in recent years, the courts have given in principle approval of computer assisted reviews (CAR), including predictive coding (see here and here). The present decision illustrates that the court is prepared to delve into the detail of how such programmes have been set up and used in order to assess the adequacy of a party's disclosure.

It also serves as a reminder of the importance the courts place on transparency in such processes. Where a party fails to engage with its opposing party adequately in advance of designing and carrying out its disclosure searches, it risks being ordered to redo some or all of the review, even if that involves substantial additional time and costs.

Background

The underlying action was a US$65 million warranty claim arising from the sale of an aerospace business. The disclosure process had been, in the words of Mr Justice Coulson, "something of a running sore".

The defendants sought orders in respect of two aspects of the claimants' disclosure.

First, the claimants had limited the search for disclosable documents to about one-third of the folders and file paths in an extremely large shared drive (containing some 20 million documents in 860,000 folders and file paths). That subset was identified by way of a request to the claimants' custodians to identify the file paths likely to contain relevant documents, combined with the claimants' solicitors considering the full list of file paths as a cross-check.

The defendants sought an order requiring the release of the full list of 860,000 folders and file paths for their review.

Second, the claimants had identified 450,000 documents as responsive to the keywords agreed between the parties. Of those, the claimants searched approximately 230,000 manually, aided by CAR. As to the remaining 220,000 documents, the claimants had used CAR to take a 1% sample, which produced a prediction that only 0.38% of the documents would be relevant. On that basis, the claimants had decided not to undertake any further searches within the 220,000 documents.

The defendants sought an order that the claimants undertake a manual review of the full 220,000 documents.

Decision

The High Court (Coulson J) declined to make an order in respect of the list of folders and file paths in the shared drive, but did order a further manual review of a sample of one-quarter (55,000) of the 220,000 documents.

The shared drive

The court noted that the claimants had not consulted the defendants before adopting the approach they did to reduce the number of the shared drive file paths and folders searched, and observed that "it may have been better if they had done so".

However, the process was made clear in the claimants' original List of Documents and the defendants' complaint was therefore now very late. Further, the court was satisfied that the approach had been reasonable and proportionate, noting that only two further folders/file paths had needed be disclosed subsequently as a result of the document reviews and also that the defendants themselves had been unable to identify any missing file paths that were obviously potentially relevant (despite the fact that they had the assistance of the managing director of the claimants' business at the relevant time). The court therefore declined to make the order sought.

The 220,000 documents

The court had greater concerns regarding the process adopted in respect of the 450,000 keyword-responsive documents.

First, the court expressed concern as to the transparency of both the use of CAR and of the sampling exercise. It noted that what the claimants had done was not what they had said they would do in their Electronic Disclosure Questionnaire (which had indicated that there would be a manual search of all keyword-responsive documents) and was also not made clear in their List of Documents.

The court noted that adopting such an approach unilaterally, without any consultation with the defendants, ran counter to the approach clearly advocated in Practice Direction 31B (regarding electronic disclosure) and the e-disclosure protocols followed in the Technology and Construction Court (TeCSA/TECBAR), both of which provide for discussion and cooperation between the parties at the outset, to avoid misunderstandings and the risk of having to extend or repeat disclosure reviews.

In the present case, at no time had the claimants provided details as to how the CAR was set up or how it was operated. In circumstances where the decision to use the CAR was unilateral, the court regarded that lack of detail as "unsatisfactory". Similarly unsatisfactory was the lack of information as to how the sampling exercise operated, such as the stated tolerances and how many rounds of sampling were undertaken.

The court also expressed concern as to the number of people involved in the CAR process. In Pyrrho Investments Ltd v MWB Property Ltd [2016] EWHC 256 (Ch), the first English court decision to consider and give in-principle approval to the use of CAR, the court proposed a number of guidelines for its use. These included that, in "teaching" CAR technology about the relevant issues in a case "it is essential that the criteria for relevance be consistently applied ... So the best practice would be for a single, senior lawyer who has mastered the issues in the case to consider the whole sample". In the present case, the evidence suggested the involvement of 10 paralegals and four associates in the CAR process, and it was unclear whether there was any senior lawyer or other person undertaking the overseeing role advocated in Pyrrho. While Coulson J did note that the Pyrrho recommendation might be regarded as "a counsel of perfection", he thought that the sheer volume of those involved in the present case "may mean that it has not been 'educated' as well as it might have been".

Having concluded that the claimants' approach was neither transparent nor independently verifiable, Coulson J went on to consider whether it was reasonable and proportionate to require further reviews. He concluded that it was, based on various indicative factors including: the very modest number of documents that the reviews had identified as disclosable (19,500) as a proportion of the 450,000 keyword-responsive documents; the fact that 2,000 out of 3,000 documents subsequently disclosed in supplementary lists were responsive to the keywords, suggesting that they would have been disclosed initially if the process had been reliable; and the fact that at least some of the documents disclosed in the supplementary lists were documents that the claimants' own witnesses and experts had referred to or wished to rely on.

Given the absence of detail as to the CAR process that was used, so that it was difficult to formulate an alternative using that system, and that the trial was less than six months away, the court concluded that "only a manual review will do". However, given the tight timeframe, in lieu of a manual review of all 220,000 documents the court was prepared to order a review of a sample of one-quarter (55,000) of those documents, with the results to be reported back to the court in three weeks (apparently leaving open the possibility of further reviews being ordered).

Comment

It is clear that a key factor underlying the court's decision to order further reviews in this case was the lack of transparency as to the process adopted (both in the failure to consult in advance with the defendants and the failure to provide detail of the process to the court even during the application). To that extent, the decision strongly reinforces the courts' expectation that a CAR process should be transparent and independently verifiable, and that the parties will cooperate or at least consult from the outset not just on whether CAR will be used but, to at least some extent, on the detail of how it will be used.

It is not clear whether the court's expressed concerns with aspects of the process itself (such as the sampling parameters and the number of people involved in the CAR reviews) would, in the absence of the concerns as to transparency, have been sufficient to warrant the order for further review. Certainly, as to the personnel issue, in many document-heavy cases it may not be practicable or cost-effective to comply with the Pyrrho "best practice" of having one senior lawyer conducting the entire CAR "teaching" process, and Coulson J did recognise that this may be a "counsel of perfection".   However, the decision demonstrates that the court is prepared to scrutinise the technical details of the process used (particularly where it was adopted by unilateral decision) and reminds parties using CAR of the need to be able to explain and justify their approach.