Monday, July 30, 2012

The Kitchen Test of Science

My default reaction to scientific papers is, "Sure, pull the other one." It's because things like this keep happening. (Via.)
"[The authors] find that consumers are overconfident in their ability to learn to use skill-based products before trying them out. As soon as they gain experience with the product, however, they flip to the opposite extreme and become under-confident in their ability to use the new product with the consequence that they often quit using it."
For me? The opposite.

Just spent a day and a bit learning NetLogo. Before I started, I thought it would be longer. Now I've spent several hours asking questions of a simple evolutionary model, and haven't yet run into anything I wanted to tell it to do but didn't know how to do. (Syntax not included, as it takes seconds to look up.) My opinion of the effort required has gone down monotonically.

Two possibilities. One, their study is just broken. Two, it doesn't apply to me. Perhaps I'm not a consumer. I can at least be sure that the opposite of their conclusions are true in my kitchen.

Their epistemology is broken. Since they use the same epistemology to measure the effect of e.g. salt, I can be sure they're fooling themselves there, too. Maybe salt is bad for the general populace. Or not. I can't re-run the experiment in general, only on myself, so I can't check how good are studies in general. But their study is simply not evidence one way or the other about whether it's bad for me.

I like them as sources of things to check. How much does the average study cost in grants and wages? Does anyone know? In any case, it is an impressively inefficient hypothesis-generation engine.

As for anyone who doesn't have the necessary time or discipline to check? Worthless at best.


Anonymous said...

“Before the subjects gained direct experience with the new skill, they over-predicted their own performance.”

This does comport with other related phenomenon, particularly the illusion of explanatory depth. [For example, ask the average person if they know (or if they can explain) how a toilet works. Such a ubiquitous device! Of course, it’s obvious! Everyone knows how a toilet works! But ask for the explanation and you’ll get gobbledygook.]

Alrenous said...

Seems a little different.

In one, they use it and therefore think they know how it works.

In the other, they don't use it, and therefore think they know how it works.

I'd be happy if they at least knew how to do their own job. On one of my school field trips, I asked an apple farmer how to farm apples, and got evasion and gobbledygook. To compare, I later asked the teacher how teaching works, and got the same. Subsequent tests have confirmed it.