Useless lie detectors and wasted money: the cost of irrationality

Eriksson, Anders, and Francisco Lacerda. "Charlatanry in forensic speech science: A problem to be taken seriously." International Journal of Speech, Language and the Law 14.2 (2007): 169-193.

I came across this study while doing research on false rape accusations... yes i get around. :P

https://en.wikipedia.org/wiki/Lie_detection#General_accuracy_and_limitations_of_assessments

In the peer-reviewed academic article "Charlatanry in forensic speech science", the authors reviewed 50 years of lie detector research and came to the conclusion that there is no scientific evidence supporting that lie detectors actually work.[24] Lie detector manufacturer Nemesysco threatened to sue the academic publisher for libel resulting in removal of the article from online databases. In a letter to the publisher, Nemesysco's lawyers wrote that the authors of the article could be sued for defamation if they wrote on the subject again.[25][26]

-

Abstract

A lie detector which can reveal lie and deception in some automatic and perfectly

reliable way is an old idea we have often met with in science fiction books and comic

strips. This is all very well. It is when machines claimed to be lie detectors appear

in the context of criminal investigations or security applications that we need to be

concerned. In the present paper we will describe two types of ‘deception’ or ‘stress

detectors’ (euphemisms to refer to what quite clearly are known as ‘lie detectors’). Both

types of detection are claimed to be based on voice analysis but we found no scientific

evidence to support the manufacturers’ claims. Indeed, our review of scientific studies

will show that these machines perform at chance level when tested for reliability. Given

such results and the absence of scientific support for the underlying principles it is

justified to view the use of these machines as charlatanry and we argue that there

are serious ethical and security reasons to demand that responsible authorities and

institutions should not get involved in such practices.

keywords: lie detector, charlatanry, voice stress analysis, psychological

stress evaluator, microtremor, layered voice analysis, airport

security

-

To keep this distinction in mind has methodological implications. It seems

reasonable, from a methodological point of view, to begin by determining

the validity of a suggested method before it makes much sense to study its

reliability. If the method can be shown to lack validity altogether it will as a

consequence also be unreliable and carrying out a reliability test meaningless.

If the validity is not known it will be a ‘black box’ whose reliability, if any,

will remain unexplained. We must keep in mind, however, that validity and

reliability are not all or nothing concepts. A method may be valid to a degree

and reliability may range from very poor to almost perfect. At the far end of

the negative scale we find things like astrology. It would be a complete waste of

time to design experiments to determine how precisely horoscopes may predict

future events when we know that the validity of the method is non-existent. At

the positive end of the scale we find methods like DNA testing whose validity

is solidly supported by scientific evidence and whose reliability is extremely

high, albeit not perfect.

they got the implication wrong. the implication is (∀x)¬reliable(x)→¬valid(x). If a test does not give consistent results for the same stimuli, it cannot possibly be measuring anything. one can infer (contraposition) that (∀x)valid(x)→reliable(x). if a test can be used to predict something, its measurements cannot be random. random input cannot predict anything, not even random output. more technically, (∀x)(∀n)valid(x, n)→(∃m)reliable(x, m)∧m≥n. for any given test and any given number [0-1], if the test has a validity for n for predicting something, it has a reliability of m which is larger than or equal to n. the reliability puts the upper bound on the validity of a test for something.

In fact astrology has both low to zero reliability* and almost zero validity. for instance, if used in a cohort to predict the height of the person, it can work for the simple reason that children with certain star signs are taller than children with certain other star signs given that they are born the same year. this has nothing to do with astrology, and everything to do with some children having more time to grow than others. if one used a non-children cohert, astrology's validity goes to 0 again for height prediction. at least, in modern societies where food intake is not dependent on the seasons. for societies where it is, astrology might have some validity, but not becus of planets having any effect on people, just becus the seasons have some effect on people.

that is, reliability in assigning star signs to people based on a description of their personality, or reliability in giving predictions for the future for any given star sign. for proof of this latter point, find a number of randomly chosen astrology sources, and compare their predictions for the future for any given star sign. they dont agree at all. hence, they as a group cannot possibly predict anything.

-

Is there anything we can do to prevent charlatanry in forensic speech

science?

Charlatanry, fraud, prejudice and superstition have always been with us. If we

look back in history and compare with what we see today there is little that

gives us hope that progress in science will diminish the amount of supersti-

tious nonsense we see around us. Astrology, for example, seems to be more

popular than ever and totally unaffected by how many times astronomers

explain that it is complete nonsense. We are therefore somewhat pessimistic

about the possibility of efficiently removing charlatanry from forensic speech

science. But we hope that responsible authorities like the police and security

services will listen to scientifically trained experts in the field rather than to

smooth talking and wishful thinking from vendors of bogus lie detectors

and similar gadgets. That is probably where we should invest our efforts. We

must also take great care when we present our results so that the issue does

not appear as a scientific controversy, which it is not. No qualified speech

scientist believes in this nonsense so there is absolutely no controversy there,

and it is very important that this becomes clear. We have included sufficient

detail in this paper to provide the reader with useful arguments in the struggle

against charlatanry. We hope that the effort will not turn out to be totally

without effect.