Google

Bartlett's Bizarre Bazaar

Comment, Comics and the Contrary. Contact: aj_bartlett1977*at*yahoo*dot*co*dot*uk
Enter your email address below to subscribe to Bartlett's Bizarre Bazaar!


powered by Bloglet

Monday, June 20, 2005

 

How many people have been cheated by these quacks?

The Guardian hosts a psychometric test designed by the The Morrisby Organisation. This post is not challenge the basis of psychometric tests, though I am sceptical of the amount of useful information they can provide about an individual. This post is a shocked report that the test is ‘wrong’ – the purported answers to the questions asked are simply incorrect.

The first test is a “verbal critical reasoning test”. They are described by the Guardian as being “used to find out how well you can assess verbal logic. They are usually in the form of a passage, or passages of prose, followed by a number of statements. Your task is to decide if the statements are "True", "False" or if you "Cannot tell" from the information provided. You are to assume that everything that is said in the passages is true.” Then follows two passages on animal testing. After this there is a set of six statements, which the person sitting the test is asked to respond to with one of three categories.

True It follows logically from the information provided
False It is obviously incorrect given the information provided
Cannot tell It is impossible to tell given the information provided

For these statements I made the ‘correct’ judgement in three cases and the ‘incorrect’ judgement in the other three. The three that I got wrong were statements to which I had responded ‘cannot tell’. And I am right. These judgements that I make are not (explicitly) based on knowledge from outside the texts provided, but are responses to what follows logically from the information provided. This is what the test asks for.

4. Vistek is a protein found in the eye. My answer was ‘cannot tell’. The ‘correct’ answer is ‘false’. The rationale given for this judgement is; “It says that Vistek simulates the reaction of the eye.” Now, it does not follow logically from the information found in the text that Vistek is not found in the eye, or indeed that it is found in the eye. The only reasonable response to this statement using the information in the test (“a particular protein Vistek which simulates the reaction of the eye”) is cannot tell. The statement is neither ‘logically true’ nor ‘obviously incorrect’.

5. Animal testing has been used to develop 10 000 ingredients. My answer was ‘cannot tell’. The ‘correct’ answer was ‘false’, on the basis that the text “says that 10 000 ingredients can be used without the need for animal testing.” Now, what the text actually says is that “At present there are over 10 000 ingredients available to the cosmetics industry. These can be used in the development of new products, free from the controversy surrounding animal testing.” The origin of these products has not been discussed. It is perfectly reasonable to suppose that animal testing was used in the process that added these ingredients to the chemist’s existing inventory. It does not logically follow that this is the case, but neither is it obviously incorrect to suggest this.

6. Kidney transplants were first tested in animals. My answer was ‘cannot tell’. The correct answer was ‘true’, on the basis that the text “says that kidney transplants would not exist if it were not for animal testing.” The text actually says “medical techniques such as kidney transplants would not exist were it not for pioneering animal testing.” First, there is the statement of priority. Even if kidney transplants were tested in animals, it does not logically follow that they were first tested in animals. Secondly, the text does not say that kidney transplants were tested on animals, rather that “kidney transplants would not exist if it were not for animal testing.” This statement can also mean that precursors to kidney transplants, necessary procedures involved in kidney transplantation, or the knowledge involved in understanding the kidneys or transplantation, required animal testing in the course of their development. It does not logically follow that “Kidney transplants were first tested on animals.”

The second test is of “numerical critical reasoning”, which “looks at how well you can reason with numbers and understand information presented in numerical form… Your task is to select the right response from five possible answers.”

We are provided with the following table:

Television viewing habits 250 people were asked what sort of programme they liked the most:
Image hosted by Photobucket.com

Question 2 is: “What proportion of 16-20 year olds watch something other than the News?” From the table provided it is impossible to answer this question. The answer they give is, of course, 82% (100%-18%). But this is utter nonsense. The people responding to the survey were asked what sort of television programme they liked the most. The people who liked News the most are perfectly able of watching programmes other than the News.

The third test is a self-report questionnaire on “typical behaviour”. The answers to these questions are not logically right or wrong (though you can lie), but, given the lack of logic demonstrated in the preceding two tests, how can any person place their trust in the people who will be interpreting the answers to this third part of the test.

I wonder how much money that The Morrisby Organisation rakes in from the design and administration of these tests. More than that, I wonder how many people have been barred from jobs by these quacks. Remember, these are flaws that can be spotted by someone unfamiliar with the theory that underpins psychometric testing. Given that those who hold tight to the usefulness of this theory are capable of such staggering errors in simple reasoning, we ought to assume that their attraction to the ‘power’ of psychometric testing is based on similarly flawed reasoning.

[UPDATE 24th June: I sent this post to Ben Goldacre, the writer of the Guardian's excellent Bad Science column. Looking at the Guardian Online's Work page, I see that the editors have now removed a link to the psychometric test. But not, it seems, the test itself.]

Comments:
On one occasion I came close to losing a job because according to a psychometric test I was not a "joiner". In fact I had answered that I was unlikely to participate in team sports. True: I'm disabled. I can't.

Fortunately I was able to challenge this result at the time, but it didn't stop the company using the same test in future.
 
Andrew

I think the first is a push poll. The purpose is not to learn but to leave the test taker with a several facts that support animal testing. The others are entertainment.
 
Small world of blogging. You have a namesake, a leftwing australian politician blogger:
http://andrewbartlettonline.blogspot.com/
 
Oh, and I think the use of such tests is disgusting, yet another way the corporate world seeks to homogenise the worker base by promoting irrelevancies over provable talent in the workplace.
 
My namesake has commented on previous posts.

I have noticed that the Guardian Online has removed the link to the test from its 'Work' webpage, where it used to sit alongside the IQ test.

It has not, however, removed the test itself.
 
Bad science depresses me greatly. I see a lot of it. Psychologists are not the only ones to blame, but they are among the worst culprits. The root cause seems to be that they think they are scientists. I have witnessed or participated in a number of psychology experiments that were badly flawed because their designers didn't know what a control group was for, couldn't spot a confounding variable, had only a limited understanding of the nature of cause and effect, and didn't have even the most rudimentary acquaintance with ethics.
 
Post a Comment

<< Home


Archives

August 2004   September 2004   October 2004   November 2004   December 2004   January 2005   February 2005   March 2005   April 2005   May 2005   June 2005   July 2005   August 2005   September 2005   October 2005   November 2005   December 2005   January 2006   February 2006   March 2006   April 2006   May 2006   June 2006   July 2006   August 2006   September 2006   October 2006   November 2006   December 2006   January 2007   March 2007  

This page is powered by Blogger. Isn't yours?

«#?» Listed on Blogwise