Useless lie detectors and wasted money: the cost of irrationality
Eriksson, Anders, and Francisco Lacerda. "Charlatanry in forensic speech science: A problem to be taken seriously." International Journal of Speech, Language and the Law 14.2 (2007): 169-193.
I came across this study while doing research on false rape accusations... yes i get around. :P
https://en.wikipedia.org/wiki/Lie_detection#General_accuracy_and_limitations_of_assessments
In the peer-reviewed academic article "Charlatanry in forensic speech science", the authors reviewed 50 years of lie detector research and came to the conclusion that there is no scientific evidence supporting that lie detectors actually work.[24] Lie detector manufacturer Nemesysco threatened to sue the academic publisher for libel resulting in removal of the article from online databases. In a letter to the publisher, Nemesysco's lawyers wrote that the authors of the article could be sued for defamation if they wrote on the subject again.[25][26]
-
Abstract
A lie detector which can reveal lie and deception in some automatic and perfectly
reliable way is an old idea we have often met with in science fiction books and comic
strips. This is all very well. It is when machines claimed to be lie detectors appear
in the context of criminal investigations or security applications that we need to be
concerned. In the present paper we will describe two types of ‘deception’ or ‘stress
detectors’ (euphemisms to refer to what quite clearly are known as ‘lie detectors’). Both
types of detection are claimed to be based on voice analysis but we found no scientific
evidence to support the manufacturers’ claims. Indeed, our review of scientific studies
will show that these machines perform at chance level when tested for reliability. Given
such results and the absence of scientific support for the underlying principles it is
justified to view the use of these machines as charlatanry and we argue that there
are serious ethical and security reasons to demand that responsible authorities and
institutions should not get involved in such practices.
keywords: lie detector, charlatanry, voice stress analysis, psychological
stress evaluator, microtremor, layered voice analysis, airport
security
-
To keep this distinction in mind has methodological implications. It seems
reasonable, from a methodological point of view, to begin by determining
the validity of a suggested method before it makes much sense to study its
reliability. If the method can be shown to lack validity altogether it will as a
consequence also be unreliable and carrying out a reliability test meaningless.
If the validity is not known it will be a ‘black box’ whose reliability, if any,
will remain unexplained. We must keep in mind, however, that validity and
reliability are not all or nothing concepts. A method may be valid to a degree
and reliability may range from very poor to almost perfect. At the far end of
the negative scale we find things like astrology. It would be a complete waste of
time to design experiments to determine how precisely horoscopes may predict
future events when we know that the validity of the method is non-existent. At
the positive end of the scale we find methods like DNA testing whose validity
is solidly supported by scientific evidence and whose reliability is extremely
high, albeit not perfect.
they got the implication wrong. the implication is (∀x)¬reliable(x)→¬valid(x). If a test does not give consistent results for the same stimuli, it cannot possibly be measuring anything. one can infer (contraposition) that (∀x)valid(x)→reliable(x). if a test can be used to predict something, its measurements cannot be random. random input cannot predict anything, not even random output. more technically, (∀x)(∀n)valid(x, n)→(∃m)reliable(x, m)∧m≥n. for any given test and any given number [0-1], if the test has a validity for n for predicting something, it has a reliability of m which is larger than or equal to n. the reliability puts the upper bound on the validity of a test for something.
In fact astrology has both low to zero reliability* and almost zero validity. for instance, if used in a cohort to predict the height of the person, it can work for the simple reason that children with certain star signs are taller than children with certain other star signs given that they are born the same year. this has nothing to do with astrology, and everything to do with some children having more time to grow than others. if one used a non-children cohert, astrology's validity goes to 0 again for height prediction. at least, in modern societies where food intake is not dependent on the seasons. for societies where it is, astrology might have some validity, but not becus of planets having any effect on people, just becus the seasons have some effect on people.
that is, reliability in assigning star signs to people based on a description of their personality, or reliability in giving predictions for the future for any given star sign. for proof of this latter point, find a number of randomly chosen astrology sources, and compare their predictions for the future for any given star sign. they dont agree at all. hence, they as a group cannot possibly predict anything.
-
Is there anything we can do to prevent charlatanry in forensic speech
science?
Charlatanry, fraud, prejudice and superstition have always been with us. If we
look back in history and compare with what we see today there is little that
gives us hope that progress in science will diminish the amount of supersti-
tious nonsense we see around us. Astrology, for example, seems to be more
popular than ever and totally unaffected by how many times astronomers
explain that it is complete nonsense. We are therefore somewhat pessimistic
about the possibility of efficiently removing charlatanry from forensic speech
science. But we hope that responsible authorities like the police and security
services will listen to scientifically trained experts in the field rather than to
smooth talking and wishful thinking from vendors of bogus lie detectors
and similar gadgets. That is probably where we should invest our efforts. We
must also take great care when we present our results so that the issue does
not appear as a scientific controversy, which it is not. No qualified speech
scientist believes in this nonsense so there is absolutely no controversy there,
and it is very important that this becomes clear. We have included sufficient
detail in this paper to provide the reader with useful arguments in the struggle
against charlatanry. We hope that the effort will not turn out to be totally
without effect.