Bartlett's Bizarre Bazaar

The Guardian hosts a psychometric test designed by the The Morrisby Organisation. This post is not challenge the basis of psychometric tests, though I am sceptical of the amount of useful information they can provide about an individual. This post is a shocked report that the test is ‘wrong’ – the purported answers to the questions asked are simply incorrect.

The first test is a “verbal critical reasoning test”. They are described by the Guardian as being “used to find out how well you can assess verbal logic. They are usually in the form of a passage, or passages of prose, followed by a number of statements. Your task is to decide if the statements are "True", "False" or if you "Cannot tell" from the information provided. You are to assume that everything that is said in the passages is true.” Then follows two passages on animal testing. After this there is a set of six statements, which the person sitting the test is asked to respond to with one of three categories.

True It follows logically from the information provided
False It is obviously incorrect given the information provided
Cannot tell It is impossible to tell given the information provided

For these statements I made the ‘correct’ judgement in three cases and the ‘incorrect’ judgement in the other three. The three that I got wrong were statements to which I had responded ‘cannot tell’. And I am right. These judgements that I make are not (explicitly) based on knowledge from outside the texts provided, but are responses to what follows logically from the information provided. This is what the test asks for.

4. Vistek is a protein found in the eye. My answer was ‘cannot tell’. The ‘correct’ answer is ‘false’. The rationale given for this judgement is; “It says that Vistek simulates the reaction of the eye.” Now, it does not follow logically from the information found in the text that Vistek is not found in the eye, or indeed that it is found in the eye. The only reasonable response to this statement using the information in the test (“a particular protein Vistek which simulates the reaction of the eye”) is cannot tell. The statement is neither ‘logically true’ nor ‘obviously incorrect’.

5. Animal testing has been used to develop 10 000 ingredients. My answer was ‘cannot tell’. The ‘correct’ answer was ‘false’, on the basis that the text “says that 10 000 ingredients can be used without the need for animal testing.” Now, what the text actually says is that “At present there are over 10 000 ingredients available to the cosmetics industry. These can be used in the development of new products, free from the controversy surrounding animal testing.” The origin of these products has not been discussed. It is perfectly reasonable to suppose that animal testing was used in the process that added these ingredients to the chemist’s existing inventory. It does not logically follow that this is the case, but neither is it obviously incorrect to suggest this.

6. Kidney transplants were first tested in animals. My answer was ‘cannot tell’. The correct answer was ‘true’, on the basis that the text “says that kidney transplants would not exist if it were not for animal testing.” The text actually says “medical techniques such as kidney transplants would not exist were it not for pioneering animal testing.” First, there is the statement of priority. Even if kidney transplants were tested in animals, it does not logically follow that they were first tested in animals. Secondly, the text does not say that kidney transplants were tested on animals, rather that “kidney transplants would not exist if it were not for animal testing.” This statement can also mean that precursors to kidney transplants, necessary procedures involved in kidney transplantation, or the knowledge involved in understanding the kidneys or transplantation, required animal testing in the course of their development. It does not logically follow that “Kidney transplants were first tested on animals.”

The second test is of “numerical critical reasoning”, which “looks at how well you can reason with numbers and understand information presented in numerical form… Your task is to select the right response from five possible answers.”

We are provided with the following table:

Television viewing habits 250 people were asked what sort of programme they liked the most:


Question 2 is: “What proportion of 16-20 year olds watch something other than the News?” From the table provided it is impossible to answer this question. The answer they give is, of course, 82% (100%-18%). But this is utter nonsense. The people responding to the survey were asked what sort of television programme they liked the most. The people who liked News the most are perfectly able of watching programmes other than the News.

The third test is a self-report questionnaire on “typical behaviour”. The answers to these questions are not logically right or wrong (though you can lie), but, given the lack of logic demonstrated in the preceding two tests, how can any person place their trust in the people who will be interpreting the answers to this third part of the test.

I wonder how much money that The Morrisby Organisation rakes in from the design and administration of these tests. More than that, I wonder how many people have been barred from jobs by these quacks. Remember, these are flaws that can be spotted by someone unfamiliar with the theory that underpins psychometric testing. Given that those who hold tight to the usefulness of this theory are capable of such staggering errors in simple reasoning, we ought to assume that their attraction to the ‘power’ of psychometric testing is based on similarly flawed reasoning.

[UPDATE 24th June: I sent this post to Ben Goldacre, the writer of the Guardian's excellent Bad Science column. Looking at the Guardian Online's Work page, I see that the editors have now removed a link to the psychometric test. But not, it seems, the test itself.]