Should forensic scientists offer non-scientific evidence?

Having just completed a report on a case where DNA transfer was one issue and re-reading Christophe Champod’s review in Forensic Genetics (Champod 2013) brings to mind an additional point in the “experience v scholarship” debate; the ethical responsibility/duty of a forensic scientist acting as an expert witness.

Courts, particularly in England and Wales, have made it clear through a number of judgements (R v Reed, Reed & Garmson 2010) (R v Weller 2010) that opinions supported only by the experience of the expert witness are admissible and will be given weight (twas ever thus).

Numerous authorities (Champod 2013) have also made it clear that expert opinion based solely on experience puts justice at risk.

Even if courts will hear such evidence it is right for scientists to offer non-scientific evidence? In the absence of relevant research and scholarship surely the only answer the scientist can give is that science cannot help.

Furthermore, in the absence of relevant research and scholarship, is the forensic scientist better placed than the fact-finder to evaluate propositions?

Given the plethora of codes of conduct/practice and regulatory frameworks that now exist it surprises me that the questionably ethical practice of offering fact-finders non-scientific evidence has not been identified as malpractice and sanctions put in place.

Champod, Christophe. “DNA transfer: informed judgment or mere guesswork?” Frontiers in Genetics, 2013: 1-3.

R v Reed & Reed & Garmson. Crim 2698 (EWCA, 2010).

R v Weller. Crim 1085 (EWCA, 2010).

All’s not well in the...... District of Columbia

The story

On the 23^rd of April 2015 the Director of the Department of Forensic Sciences (DFS) in Washington DC resigned and senior managers and legal counsel ‘fired’. The departure was the culmination of a series of events that began in October 2014 with a complaint about the way the DFS interprets mixed DNA profiles. Mixed profiles are those with more than one contributor.

On the 8^th of May 2015 the Washington Post, under the headline “A setback for forensic science”, published an opinion piece by William Thomson, professor of criminology and law at the University of California. In it he set out the key events that led to the departure and the related suspension of DNA testing at the DFS.  In Professor Thompson’s opinion at stake is the independence of the DFS as a forensic science laboratory.

The headline “A setback for forensic science” refers to the fact the DFS was established in the light of recommendation #4 in the 2009 National Academy of Science (NAS) report. The NAS report recommended the removal of administrative control of forensic science laboratories from law enforcement agencies and prosecuting authorities. According to Professor Thompson, prosecutors were not happy with the loss of control. A point supported by Jay Siegel, a member of the DFS’s Science Advisory Board who resigned over the issue.

The contention is that these bodies used legitimate differences of opinion regarding the interpretation of mixed DNA profiles as a means of gaining greater control over the operation and management of the DFS, to effect a resumption of ‘normal service’.

An alternative analysis, and that advanced by the prosecuting authorities supported by the Mayor, is that the method used by the DFS for the interpretation of mixed DNA profiles was wrong. The failure by senior management at the DFS to accept and act on that opinion made their position politically untenable.

This analysis is supported by a quality audit conducted by ANFSI-ASQ National Accreditation Board (ANAB) at the request of the Mayor and a review of evidence by experts at the request of prosecuting authorities. 

This episode raises a number of major issues which might have consequences for forensic science provision beyond the District of Columbia.

Matters arising

The issues may be summarised as; political interference in the operation and management of a forensic science laboratory and an accrediting body.

The complicating factor in this episode is that the interpretation of mixed DNA profiles is a controversial area and as the DFS Science Advisory Board reported to the Mayor on April the 26^th 2015:

“Within the scientific community, there is, to date, no single, universally accepted set of best practices....”.

Experts within the field know this full well. There are two fundamental approaches with world-leading and highly regarded experts supporting each approach. It could be argued that, until consensus is reached, DNA evidence derived from mixed profiles with possibly more than two contributors should not be placed before a court.  

The results generated depend on the approach used and the assumptions made. The figures reported in support of the allegation of DFS error are 1 in 3,290 (DFS) and 1 in 9 (other experts). As DNA evidence goes that is hardly any difference at all. The correction of recent errors discovered in FBI DNA evidence is likely to generate far greater differences which the FBI claim are not significant.

If the Director and senior staff at the DFS have been removed by politicians using a legitimate difference of opinion within the relevant scientific community as a pretext for bringing forensic science provision back under the control of prosecutors and law enforcement agents then that indeed is a backward and worrying step; a step which puts at risk the quality of forensic evidence. In addition to the NAS report, there is a body of scholarship pointing to the closeness of the relationship between law enforcement and forensic science providers as a contributor to miscarriages of justice.

Quality matters

The second issue, perhaps of greater concern and certainly wider in scope, is the role of the accrediting body ANAB and the consequences of its involvement.

The key questions regarding ANAB are:

·       Is it subject to inappropriate political interference?

·       Is it a competent accrediting body (compliant with the standard ISO17011)?

·       If it is not competent what effect might that have on the reliability of forensic evidence offered by other labs accredited by ANAB and the non-DNA evidence of the DFS?

ANAB is the DFS’s accrediting body. ANAB assesses compliance with an international standard identified as ISO17025. ISO17025 is the standard to which all forensic science laboratories in the developed world are accredited to or soon will be. Compliance assures the reliability of the forensic evidence offered by the DFS.

Compliance with ISO17025 essentially requires three things;

·       that the laboratory management is competent,

·       that the staff are competent, and

·       that the methods and procedures are valid; i.e. capable of producing evidence that might be relied on.

Of particular relevance is that compliance with both ISO17025 (4.1.5 b) and ISO17011 (4.3) require measures to be in place to protect management from undue influence that might adversely affect the quality of their work. Fairness, impartiality and transparency are basic requirements in quality management.

From the 31^st of October 2013 (the date on which accreditation was granted) until the report that resulted in the suspension of DNA testing was delivered on the 24^th of April 2015, ANAB was satisfied that DFS remained compliant with ISO17025 and quality audits undertaken since accreditation was granted had not revealed any major issues; i.e. those that might warrant suspension of accreditation.

During the period of accreditation an accrediting body will regularly conduct quality audits, usually annually, to ensure continuing compliance. It is often the case that, as a result of these audits, minor non-compliances are identified which need to be cleared in a set period of time to maintain accreditation. In addition to quality auditors from the accrediting body, ISO17025 accredited laboratories will have a team of trained and qualified internal quality auditors conducting an agreed program of audits. The aim is for the entire quality management system (organisation, staff and methods) to be audited over the period of accreditation.

ISO17025 is seen as an essential ‘seal of approval’ by forensic science laboratories. In addition, compliance is often used as a means of preventing scrutiny by independent reviewers. Accredited forensic science laboratories claim that accredited methods should not be subject to scrutiny by an independent third party because an accrediting body (an external and independent agency) assures the validity of the methods used and consequently the reliability of evidence. This episode completely undermines that claim.

The competence of ANAB

Based on the information publicly available, the relationship between ANAB and the parties; the Mayor, prosecution authorities, DFS senior management and the DFS Science Advisory Board, is unclear. However, if the DFS and ANAB are compliant with their respective standards then the quality manual of each organisation will have those relationships clearly detailed and the measures in place to protect against undue influence and conflicts of interest. It would be very interesting to have sight of these quality manuals to see how the parties all relate to one another.

The duties and responsibilities ANAB owe to its customer are set out in the standard to which ANAB is accredited, ISO17011. As ANAB is the accrediting body for the DFS those duties and responsibilities should be towards the DFS. ANAB needs to respond to the suggestion that in its dealings with other parties it was not compliant with its own standard.

ANAB’s  web page headed “ISO/IEC 17025 Accreditation for Forensic Testing Laboratories” is actually about ISO17020, a different standard.

The major evidence that calls into question the competence of ANAB is the eighteen months (and more) during which quality auditors from ANAB and internal auditors of the DFS missed non-compliances of such magnitude as to warrant suspension of accreditation.

DNA profiling evidence is often of the highest probative value and that based on mixed profiles controversial. These factors should enure that the parts of the quality management system relating to this evidence type are kept under close review by quality auditors.

One final twist

On the 11^th of December 2014 it was disclosed that a romantic relationship existed between the US attorney who first complained about the DFS method of interpreting mixed DNA profiles and who is responsible for sending samples for DNA profiling and the supervisor of forensic casework at Bode Technology. Bode Technology does the DNA profiling formerly undertaken by the DFS. On the face of it this suggests a major conflict of interest. Leaving aside quality management issues, there is an old legal aphorism “not only must justice be done it must be seen to be done”; well apparently not in the District of Columbia.

Conclusions

In addition to any non-compliance with ISO17025 in respect of DNA test methods this episode raises the possibility that there may be other non-compliances of the DFS with ISO17025 and ANAB with ISO17011.

An independent review of the quality management systems of both the DFS and ANAB is urgently required.

The facts of this episode undermine the assertion that accredited methods can be accepted as valid and scrutiny by an independent third party is unnecessary.

If the DFS and ANAB have been subject to political interference in their operation and management then the reliability of forensic evidence offered by the DFS and other ANAB accredited labs must be in question. How can an expert witness from those labs claim to be independent?

If the object of the Mayor, prosecutors and/or law enforcement agents was to wrest administrative control from the DFS, contrary to the recommendation of the NAS report, then that is of concern and puts justice at risk.  

Based on 25 years’ experience I am of the opinion that quality management in the USA is not in good shape and the International Standards Organisation needs to look carefully at US based accrediting bodies which seem to take an ‘a la carte’ approach to compliance.

Sean Doyle

July 9^th 2015

©2015 Linked Forensic Consultants Ltd   

DNA Evidence | Current Issues

DNA Evidence | Current issues

In response to an invitation I addressed the latest meeting of the New Zealand Law Society (Wellington Branch) held at the Law Society building in Wellington, New Zealand, on the 18^th of June 2015.  My brief was to try and shed greater light on DNA evidence, an evidence type that members often find confusing. The confusion, I suggest, arises from the terminology and vocabulary used which, in many cases, offers little clue as to meaning. DNA evidence is in fact a combination of chemistry, biology (and a number of sub- branches) and statistics. The same term can have different meanings depending on the science, e.g. to a chemist ‘sensitivity’ is the gradient of a response curve (the size of the signal for a given change in parameter measured) and to a biologist the limit of detection/quantitation (how little can be detected). If scientists can be confused what hope for lawyers?

 

SWGDAM offers a helpful glossary but even then confusions remain, e.g. what is and what is not a random or stochastic effect?

 

Interpretation of complex mixtures (more than two contributors)

After an introduction to the chemistry, biology and statistics behind the evidence type the discussion broadened to current issues.   These issues stem from the ability of the latest profiling products to obtain a DNA profile from just a few cells. The results are very often mixed or complex profiles where there is evidence of more than two contributors. How to interpret these complex profiles has been the subject of much debate over recent years and I think it is fair to say that consensus has yet to emerge.

 

Different labs use different approaches, making different assumptions which obviously yield different results. It could be argued that until consensus is reached the interpretation of complex mixtures should be restricted to use as an investigative tool rather than be adduced as evidence. If science is undecided then what help can it be to the trier-of-fact? 

 

Secondary transfer and Contamination

Other consequences of technological advances are that contamination and secondary transfer become more problematic. One of the fundamentals that assure the forensic utility of DNA evidence is the fact that human beings shed cellular material. Given the large numbers of cells shed, an individual’s cells will transfer by whatever means are available. The finding of an individual’s cellular material at a crime scene may not be evidence of presence. Innocent individuals might find themselves the subject of police enquiries, a situation that should be of concern to citizens and legislators.

 

Anti-contamination procedures become increasingly complex, burdensome and expensive. As complexity increases so does the risk of failure and error.

  

Interpretation and software packages STRmix^TM

The discussion then turned to more parochial matters. The New Zealand forensic science provider ESR is using ,and commercially marketing, a complex mixture interpreter: STRmix^TM. Given that ESR, with its Australian partner, are looking to sell this software package overseas it seems likely that intellectual property issues may be used as an excuse to close the product off from independent scrutiny (so called ‘black boxing’). However, the recent admission by the FBI of errors in its DNA evidence going back to 1999 provides compelling support for the argument that the validation data and all relevant studies for DNA interpretation software must be open to independent scrutiny.

 

Laboratory Error

A step in the interpretation of DNA evidence is the calculation or estimation of a random match probability – the chance of a false positive. Given a full profile and one major contributor this is often an unimaginably large figure. Advances in technology, specifically the addition of extra loci (CODIS 20  and DNA 17), have the potential to significantly increase this number such that the likelihood of laboratory error may have to be taken into account when interpreting DNA evidence.

 

Investigative v Evidential value

Perversely, technological advances have the potential to reduce the probative value of DNA evidence.  Advances certainly improve its utility as an investigative tool but the complexity of the interpretation can put the evidence beyond the understanding of the average juror.

 

---

 

ISO 17025 is not a 'gold' standard

On the 17^th of June 2015 the BBC website reported that, as a result of the fall in the number of forensic fibre experts, the killers of Stephen Lawrence might escape justice if the cold case review that led to their conviction was conducted today rather than in 2007.

In that report the international standard ISO17025 is referred to as a ‘gold’ standard. It is not. It is a minimum standard.  

One of the strengths of quality management systems, of which ISO17025 is a part, is that there are only two conditions; compliance or non-compliance. Either a product or service is fit-for-purpose or it is not. If not then remedial action is required to limit the effects of any non-compliant product or service and ensure a speedy return to compliance.

 In my experience, gained in numerous jurisdictions and over the last 25 years, the accrediting body in the UK (UKAS) operates to a high standard. This results in the forensic science laboratories it accredits working to a higher standard than those accredited by other, less experienced, bodies such as ASCLD/LAB and NATA. Nevertheless, even in England and Wales, accreditation to ISO17025 must still be considered a minimum standard.

Compliance with this minimum standard however is significantly better, in terms of delivering justice, than compliance with no standard.

 

Non-accredited forensic science laboratories are used by the police

The BBC reports that the Met (Metropolitan Police) routinely uses non-accredited laboratories.

It seems extraordinary to me that any police force or law enforcement agency would use a non-accredited laboratory. The first forensic science laboratories were accredited in the late nineties, over 15 years ago. Why are there any non-accredited forensic science laboratories providing services to police forces?

It would also be extraordinary if courts did not question the reliability of and closely scrutinise any scientific evidence provided by a non-accredited laboratory.

 

Response by the Forensic Science Regulator

The reassurance of the Forensic Science Regulator, Dr Gillian Tulley, that “The vast majority of forensic work is done to a very high quality” is worthless. Only an accrediting body, such as UKAS, has the competence to determine whether or not a forensic science laboratory and the evidence it provides is fit-for-purpose.

R v T [2010] EWCA Crim 2439 | A Position Statement

This judgement of the Appeal Court in the jurisdiction of England and Wales was delivered in 2010 and produced what may well have been the greatest response yet from the forensic science community, both academic and practitioner. The assumed support of practitioners is based on the support of ENFSI (European Network of Forensic Science Institutes) directors; i.e. directors of state forensic science laboratories throughout the Europol region.

While the judgement is in fact limited to foot wear marks it is not entirely clear on the point and legal commentators have extended the scope to include other evidence types.

What prompts this statement is that, as feared, R v T is being used to argue that any evaluation using a logical framework, i.e. based on a likelihood ratio approach, should not be admitted. When it suits their purpose, advocates are looking to the judgment as a means of excluding expert opinion evidence based on a likelihood ratio approach.

The purpose of this statement is to clarify the position of Linked Forensic Consultants. We are of course not qualified to comment on the legal aspects of the judgement; our comments are limited solely to the science.

The position of Linked Forensic Consultants regarding the scientific basis of the judgement is as follows.

• The judgment was right in its criticism of the use of unreliable data and a lack of transparency.
• The judgement was helpful in highlighting the need for expert evidence to be closely scrutinised prior to admission.
• The judgement was erroneous in rejecting a logical framework for the evaluation of expert evidence.
• The judgment was unhelpful in suggesting an opinion such as ‘could have’ would be helpful to the trier of fact.

The response of Linked Forensic Consultants to the judgement is as follows.

• There are circumstances in which the use of a logical framework for the evaluation of forensic evidence is inappropriate, but these are few.
• The likelihood ratio stands apart from Bayes’ law and using the terms ‘Bayes’ and ‘Bayesian’ in connection with the likelihood ratio approach can be misleading and is confusing (Bayes’ law may be stated as Posterior is proportional to LR x Prior; posteriors and priors are of no concern to the expert witness).
• We agree with Berger that the likelihood ratio approach is ‘logical and it is coherent. Furthermore, it is the only logical and coherent approach to forensic science interpretation. There are no others’. See Berger et al, Science and Justice 51 (2011) 215.

It is fair to say that from the perspective of forensic science this judgement seems confused. Other judgments which seem to take a variety of positions on a logical framework and evaluative evidence are;

• R v Doheny and Adams [1996] EWCA Crim 726,
• R v George [2007] EWCA Crim 2722, and
• R v Reed, Reed and Garmson [2009] EWCA Crim 2698.

Other relevant papers are
Evett I.W. et al (2011) Science & Justice 52, 1–2.
Redmayne M. et al (2011) Crim.L.R #5

In passing, we are astonished at the implication of not criticising the expert witness. Surely an expert must be responsible for his or her own competence? Excusing the expert on the grounds that he was following the standard practice within the Forensic Science Service (FSS) at the time seems extraordinary and the court’s silence on any remedial action equally extraordinary.

Sean Doyle
29/4/13
©2013 Linked Forensic Consultants Ltd. All rights reserved.

Deja vu

A recent story on a New Zealand news website highlights the poor accuracy of press reporting, if any further evidence were needed. It also draws attention to a report by a lawyer and psychologist which makes a number of familiar recommendations regarding expert evidence.

The headline bears little relation to the contents of Henderson & Seymour's report which is the basis of the story. The headline is drawn from one paragraph in 167 pages and casts an unjust slur on experts called by the defence. The headline suggests an agenda.

The report concerns experts who testify in both the criminal and Family Courts in New Zealand regarding child abuse and neglect, highly controversial and emotive areas of forensic practice far removed from the 'gold' standard of DNA profiling.

Child abuse cases are prone to experts using the witness box to promote their own theories as demonstrated by the case of the paediatrician Professor Sir Roy Meadow. Professor Meadow gave evidence for the Crown in a number of trials of mothers charged with killing their infant children. He relied on flawed statistical analysis and flawed science; a theory he invented [see Meadow R. 1977. "Munchausen syndrome by proxy (MSbP). The hinterland of child abuse" Lancet 2 (8033): 343–5.] is considered by many medical practitioners to be without scientific foundation.  Australia and the UK have established the legal precedent that MSbP does not exist as a medico-legal entity. Indeed, the scientific foundation of expert opinion offered in child abuse cases appears to be a common issue as evidenced by a quote from R v Henderson [2010] EWCA Crim 1269.

“Can the witness point to a recognised, peer-reviewed, source for the opinion? Is the clinical experience of the witness up-to-date and equal to the experience of others whose evidence he seeks to contradict?”

The report of Henderson and Seymour is based on 'qualitative empirical research'; gleaning the opinions of expert witnesses and validating those opinions by reference to lawyers. It is a useful and scholarly review particularly if the areas of forensic practice covered are of interest. However, it contains little that is new, omits some important areas of scholarship on the evaluation of scientific evidence and bias, recycles prosecutorial ‘myths’ (defence disclosure delays) and its recommendations have a familiar ring.

As I have written elsewhere (e.g. Dominion Post, Letters, 26/12/12), justice is best served when the expert evidence is relevant and expert witnesses are objective, impartial, competent, possess practical experience in their specialism and understand their role. Unfortunately that is not always the case. The remedy lies with the experts.

Other issues such as the often gross inequality of arms between the defendant and the State are matters for all the stakeholders.

Sean Doyle
26th of March 2013
Copyright 2013 Linked Forensic Consultants Ltd

Adam Scott - matters arising

The recent report by the England and Wales forensic science regulator into the case of Adam Scott, a  teenager charged with rape on the basis of unreliable DNA evidence, raises  concerns about a number of issues; justice, the reliability of forensic evidence, accreditation and a commercial market in forensic science.  

According to the report, Adam Scott, an innocent man, remained in custody for over four months as a result of a series of errors by an ISO17025 accredited forensic science provider, LGC Forensics. The DNA evidence was only called into question because the investigating officer failed to find 'the hoped for supporting evidence'. Had the 'hoped for' evidence been forthcoming then Mr Scott may have spent a good deal longer in custody. 

While this episode supports the policy of not prosecuting on DNA evidence alone it raises the question of charging on that basis particularly when the accused is then remanded in custody; there does seem to be a hint of a presumption of guilt here.

Another consequence of relying on DNA evidence alone is the potential delay to the investigation. For the victim justice is delayed and therefore denied.

Mr Scott has lost four months of his life, and probably more besides. It seem unjust that the organisation that made the errors which led to this miscarriage of justice suffers little or no sanction as a result. The suspicion being that the commercial provision of forensic science in England & Wales is so delicately balanced between success and failure that the application of any meaningful sanction  might tip it towards failure. Having controversially closed the Forensic Science Service for essentially commercial reasons the success of the forensic market is probably of political importance to the UK Government.

A couple of important questions come to mind.

Can it be right for a forensic science provider to continue to provide a service which has led to a miscarriage of justice before satisfying regulators that remedial action has been taken, lessons have been learned and that its product is now conforming?

Is it not right that the  profit seeking, commercial organisation  which caused this miscarriage of justice be called on to financially compensate Mr Scott for his loss of liberty?

One final point. There is an increasing move towards in-house provision of forensic services within the police forces of England & Wales. However, most police providers have yet to gain accreditation to the standard ISO17025 (or similar). If errors such as those identified in this case can be made by an accredited forensic science provider (and some would say one of the best forensic science laboratories in the world) surely the risk of a miscarriage of justice must be increased by the use of non-accredited providers.

It will be interesting to see the impact of this case on the future development of forensic science provision within England & Wales and perhaps elsewhere.  

 

Sean Doyle

Consultant Forensic Scientist

 © 2012 Linked Forensic Consultants Ltd. All rights reserved.