All’s not well in the...... District of Columbia

The story

On the 23^rd of April 2015 the Director of the Department of Forensic Sciences (DFS) in Washington DC resigned and senior managers and legal counsel ‘fired’. The departure was the culmination of a series of events that began in October 2014 with a complaint about the way the DFS interprets mixed DNA profiles. Mixed profiles are those with more than one contributor.

On the 8^th of May 2015 the Washington Post, under the headline “A setback for forensic science”, published an opinion piece by William Thomson, professor of criminology and law at the University of California. In it he set out the key events that led to the departure and the related suspension of DNA testing at the DFS.  In Professor Thompson’s opinion at stake is the independence of the DFS as a forensic science laboratory.

The headline “A setback for forensic science” refers to the fact the DFS was established in the light of recommendation #4 in the 2009 National Academy of Science (NAS) report. The NAS report recommended the removal of administrative control of forensic science laboratories from law enforcement agencies and prosecuting authorities. According to Professor Thompson, prosecutors were not happy with the loss of control. A point supported by Jay Siegel, a member of the DFS’s Science Advisory Board who resigned over the issue.

The contention is that these bodies used legitimate differences of opinion regarding the interpretation of mixed DNA profiles as a means of gaining greater control over the operation and management of the DFS, to effect a resumption of ‘normal service’.

An alternative analysis, and that advanced by the prosecuting authorities supported by the Mayor, is that the method used by the DFS for the interpretation of mixed DNA profiles was wrong. The failure by senior management at the DFS to accept and act on that opinion made their position politically untenable.

This analysis is supported by a quality audit conducted by ANFSI-ASQ National Accreditation Board (ANAB) at the request of the Mayor and a review of evidence by experts at the request of prosecuting authorities. 

This episode raises a number of major issues which might have consequences for forensic science provision beyond the District of Columbia.

Matters arising

The issues may be summarised as; political interference in the operation and management of a forensic science laboratory and an accrediting body.

The complicating factor in this episode is that the interpretation of mixed DNA profiles is a controversial area and as the DFS Science Advisory Board reported to the Mayor on April the 26^th 2015:

“Within the scientific community, there is, to date, no single, universally accepted set of best practices....”.

Experts within the field know this full well. There are two fundamental approaches with world-leading and highly regarded experts supporting each approach. It could be argued that, until consensus is reached, DNA evidence derived from mixed profiles with possibly more than two contributors should not be placed before a court.  

The results generated depend on the approach used and the assumptions made. The figures reported in support of the allegation of DFS error are 1 in 3,290 (DFS) and 1 in 9 (other experts). As DNA evidence goes that is hardly any difference at all. The correction of recent errors discovered in FBI DNA evidence is likely to generate far greater differences which the FBI claim are not significant.

If the Director and senior staff at the DFS have been removed by politicians using a legitimate difference of opinion within the relevant scientific community as a pretext for bringing forensic science provision back under the control of prosecutors and law enforcement agents then that indeed is a backward and worrying step; a step which puts at risk the quality of forensic evidence. In addition to the NAS report, there is a body of scholarship pointing to the closeness of the relationship between law enforcement and forensic science providers as a contributor to miscarriages of justice.

Quality matters

The second issue, perhaps of greater concern and certainly wider in scope, is the role of the accrediting body ANAB and the consequences of its involvement.

The key questions regarding ANAB are:

·       Is it subject to inappropriate political interference?

·       Is it a competent accrediting body (compliant with the standard ISO17011)?

·       If it is not competent what effect might that have on the reliability of forensic evidence offered by other labs accredited by ANAB and the non-DNA evidence of the DFS?

ANAB is the DFS’s accrediting body. ANAB assesses compliance with an international standard identified as ISO17025. ISO17025 is the standard to which all forensic science laboratories in the developed world are accredited to or soon will be. Compliance assures the reliability of the forensic evidence offered by the DFS.

Compliance with ISO17025 essentially requires three things;

·       that the laboratory management is competent,

·       that the staff are competent, and

·       that the methods and procedures are valid; i.e. capable of producing evidence that might be relied on.

Of particular relevance is that compliance with both ISO17025 (4.1.5 b) and ISO17011 (4.3) require measures to be in place to protect management from undue influence that might adversely affect the quality of their work. Fairness, impartiality and transparency are basic requirements in quality management.

From the 31^st of October 2013 (the date on which accreditation was granted) until the report that resulted in the suspension of DNA testing was delivered on the 24^th of April 2015, ANAB was satisfied that DFS remained compliant with ISO17025 and quality audits undertaken since accreditation was granted had not revealed any major issues; i.e. those that might warrant suspension of accreditation.

During the period of accreditation an accrediting body will regularly conduct quality audits, usually annually, to ensure continuing compliance. It is often the case that, as a result of these audits, minor non-compliances are identified which need to be cleared in a set period of time to maintain accreditation. In addition to quality auditors from the accrediting body, ISO17025 accredited laboratories will have a team of trained and qualified internal quality auditors conducting an agreed program of audits. The aim is for the entire quality management system (organisation, staff and methods) to be audited over the period of accreditation.

ISO17025 is seen as an essential ‘seal of approval’ by forensic science laboratories. In addition, compliance is often used as a means of preventing scrutiny by independent reviewers. Accredited forensic science laboratories claim that accredited methods should not be subject to scrutiny by an independent third party because an accrediting body (an external and independent agency) assures the validity of the methods used and consequently the reliability of evidence. This episode completely undermines that claim.

The competence of ANAB

Based on the information publicly available, the relationship between ANAB and the parties; the Mayor, prosecution authorities, DFS senior management and the DFS Science Advisory Board, is unclear. However, if the DFS and ANAB are compliant with their respective standards then the quality manual of each organisation will have those relationships clearly detailed and the measures in place to protect against undue influence and conflicts of interest. It would be very interesting to have sight of these quality manuals to see how the parties all relate to one another.

The duties and responsibilities ANAB owe to its customer are set out in the standard to which ANAB is accredited, ISO17011. As ANAB is the accrediting body for the DFS those duties and responsibilities should be towards the DFS. ANAB needs to respond to the suggestion that in its dealings with other parties it was not compliant with its own standard.

ANAB’s  web page headed “ISO/IEC 17025 Accreditation for Forensic Testing Laboratories” is actually about ISO17020, a different standard.

The major evidence that calls into question the competence of ANAB is the eighteen months (and more) during which quality auditors from ANAB and internal auditors of the DFS missed non-compliances of such magnitude as to warrant suspension of accreditation.

DNA profiling evidence is often of the highest probative value and that based on mixed profiles controversial. These factors should enure that the parts of the quality management system relating to this evidence type are kept under close review by quality auditors.

One final twist

On the 11^th of December 2014 it was disclosed that a romantic relationship existed between the US attorney who first complained about the DFS method of interpreting mixed DNA profiles and who is responsible for sending samples for DNA profiling and the supervisor of forensic casework at Bode Technology. Bode Technology does the DNA profiling formerly undertaken by the DFS. On the face of it this suggests a major conflict of interest. Leaving aside quality management issues, there is an old legal aphorism “not only must justice be done it must be seen to be done”; well apparently not in the District of Columbia.

Conclusions

In addition to any non-compliance with ISO17025 in respect of DNA test methods this episode raises the possibility that there may be other non-compliances of the DFS with ISO17025 and ANAB with ISO17011.

An independent review of the quality management systems of both the DFS and ANAB is urgently required.

The facts of this episode undermine the assertion that accredited methods can be accepted as valid and scrutiny by an independent third party is unnecessary.

If the DFS and ANAB have been subject to political interference in their operation and management then the reliability of forensic evidence offered by the DFS and other ANAB accredited labs must be in question. How can an expert witness from those labs claim to be independent?

If the object of the Mayor, prosecutors and/or law enforcement agents was to wrest administrative control from the DFS, contrary to the recommendation of the NAS report, then that is of concern and puts justice at risk.  

Based on 25 years’ experience I am of the opinion that quality management in the USA is not in good shape and the International Standards Organisation needs to look carefully at US based accrediting bodies which seem to take an ‘a la carte’ approach to compliance.

Sean Doyle

July 9^th 2015

©2015 Linked Forensic Consultants Ltd   

DNA Evidence | Current Issues

DNA Evidence | Current issues

In response to an invitation I addressed the latest meeting of the New Zealand Law Society (Wellington Branch) held at the Law Society building in Wellington, New Zealand, on the 18^th of June 2015.  My brief was to try and shed greater light on DNA evidence, an evidence type that members often find confusing. The confusion, I suggest, arises from the terminology and vocabulary used which, in many cases, offers little clue as to meaning. DNA evidence is in fact a combination of chemistry, biology (and a number of sub- branches) and statistics. The same term can have different meanings depending on the science, e.g. to a chemist ‘sensitivity’ is the gradient of a response curve (the size of the signal for a given change in parameter measured) and to a biologist the limit of detection/quantitation (how little can be detected). If scientists can be confused what hope for lawyers?

 

SWGDAM offers a helpful glossary but even then confusions remain, e.g. what is and what is not a random or stochastic effect?

 

Interpretation of complex mixtures (more than two contributors)

After an introduction to the chemistry, biology and statistics behind the evidence type the discussion broadened to current issues.   These issues stem from the ability of the latest profiling products to obtain a DNA profile from just a few cells. The results are very often mixed or complex profiles where there is evidence of more than two contributors. How to interpret these complex profiles has been the subject of much debate over recent years and I think it is fair to say that consensus has yet to emerge.

 

Different labs use different approaches, making different assumptions which obviously yield different results. It could be argued that until consensus is reached the interpretation of complex mixtures should be restricted to use as an investigative tool rather than be adduced as evidence. If science is undecided then what help can it be to the trier-of-fact? 

 

Secondary transfer and Contamination

Other consequences of technological advances are that contamination and secondary transfer become more problematic. One of the fundamentals that assure the forensic utility of DNA evidence is the fact that human beings shed cellular material. Given the large numbers of cells shed, an individual’s cells will transfer by whatever means are available. The finding of an individual’s cellular material at a crime scene may not be evidence of presence. Innocent individuals might find themselves the subject of police enquiries, a situation that should be of concern to citizens and legislators.

 

Anti-contamination procedures become increasingly complex, burdensome and expensive. As complexity increases so does the risk of failure and error.

  

Interpretation and software packages STRmix^TM

The discussion then turned to more parochial matters. The New Zealand forensic science provider ESR is using ,and commercially marketing, a complex mixture interpreter: STRmix^TM. Given that ESR, with its Australian partner, are looking to sell this software package overseas it seems likely that intellectual property issues may be used as an excuse to close the product off from independent scrutiny (so called ‘black boxing’). However, the recent admission by the FBI of errors in its DNA evidence going back to 1999 provides compelling support for the argument that the validation data and all relevant studies for DNA interpretation software must be open to independent scrutiny.

 

Laboratory Error

A step in the interpretation of DNA evidence is the calculation or estimation of a random match probability – the chance of a false positive. Given a full profile and one major contributor this is often an unimaginably large figure. Advances in technology, specifically the addition of extra loci (CODIS 20  and DNA 17), have the potential to significantly increase this number such that the likelihood of laboratory error may have to be taken into account when interpreting DNA evidence.

 

Investigative v Evidential value

Perversely, technological advances have the potential to reduce the probative value of DNA evidence.  Advances certainly improve its utility as an investigative tool but the complexity of the interpretation can put the evidence beyond the understanding of the average juror.

 

---

 

ISO 17025 is not a 'gold' standard

On the 17^th of June 2015 the BBC website reported that, as a result of the fall in the number of forensic fibre experts, the killers of Stephen Lawrence might escape justice if the cold case review that led to their conviction was conducted today rather than in 2007.

In that report the international standard ISO17025 is referred to as a ‘gold’ standard. It is not. It is a minimum standard.  

One of the strengths of quality management systems, of which ISO17025 is a part, is that there are only two conditions; compliance or non-compliance. Either a product or service is fit-for-purpose or it is not. If not then remedial action is required to limit the effects of any non-compliant product or service and ensure a speedy return to compliance.

 In my experience, gained in numerous jurisdictions and over the last 25 years, the accrediting body in the UK (UKAS) operates to a high standard. This results in the forensic science laboratories it accredits working to a higher standard than those accredited by other, less experienced, bodies such as ASCLD/LAB and NATA. Nevertheless, even in England and Wales, accreditation to ISO17025 must still be considered a minimum standard.

Compliance with this minimum standard however is significantly better, in terms of delivering justice, than compliance with no standard.

 

Non-accredited forensic science laboratories are used by the police

The BBC reports that the Met (Metropolitan Police) routinely uses non-accredited laboratories.

It seems extraordinary to me that any police force or law enforcement agency would use a non-accredited laboratory. The first forensic science laboratories were accredited in the late nineties, over 15 years ago. Why are there any non-accredited forensic science laboratories providing services to police forces?

It would also be extraordinary if courts did not question the reliability of and closely scrutinise any scientific evidence provided by a non-accredited laboratory.

 

Response by the Forensic Science Regulator

The reassurance of the Forensic Science Regulator, Dr Gillian Tulley, that “The vast majority of forensic work is done to a very high quality” is worthless. Only an accrediting body, such as UKAS, has the competence to determine whether or not a forensic science laboratory and the evidence it provides is fit-for-purpose.