Does this effect have a name? If so, what is it?

Ask questions, find resources, browse the virtual shelves.
Post Reply
JeanTate
Ad Astra
Posts: 114
Joined: Wed Sep 08, 2010 12:35 am

Does this effect have a name? If so, what is it?

Post by JeanTate » Tue Jan 21, 2014 2:55 am

Consider this thought experiment, conducted in a toy universe.

You select tens of thousands of images of galaxies, and discard those in which two galaxies appear to be interacting or overlapping, and those in which the central galaxy appears to be 'edge-on' or highly inclined. You prepare images of the remaining galaxies so they are framed consistently: centroid in the center of the image, scale such that the galaxy appears to fill ~a quarter of the image, background ('sky') a consistent black, galaxy colors rendered consistently, and so on.

You then ask an awful lot of ordinary people ('classifiers') to look at the galaxies and classify them. You offer them just two choices, 'spiral' and 'elliptical'. There's a tutorial, and some practice images. You make sure the classifiers don't talk to one another, that they are not asked to classify any galaxy more than once, etc, etc, etc.

You have an automated 'classification collection and analysis' system, that no one can interfere with. The system outputs the total number of classifications, the fraction of those which are 'spiral', and the fraction of those which are 'elliptical'.

After your classification project has been going for a while, you check on the results for three galaxies, respectively A, B, and C below.

A -> Image <- A
B -> Image <- B
C -> Image <- C

You find, for each of the three, that the total number of classifications is 10, and that the spiral (elliptical) fractions are 1.0 (0.0), 0.0 (1.0), and 0.5 (0.5), respectively.

No surprises, right?

You keep on collecting classifications, and observe the following, as the number of classifications per galaxy rises:

* for C, the spiral fraction seems to go on a drunken walk; sometimes above 0.5, sometimes below, but tends towards a value that's not far off 0.5
* for A, the spiral fraction remains at 1.0, until - after N classifications - it drops below 1.0; it never returns to 1.0
* for B, the spiral fraction remains at 0.0, until - after M classifications - it rises above 0.0; it never returns to 0.0

After an initial unanimity, spiral (elliptical) fractions will eventually become overwhelming majorities; if a galaxy does not begin with a unanimous classification (assume 'initial' is ten classifications), it will never get one.

Is there a name for this? If so, what is it?

More questions later, but to close with just one: random accidents - classifiers 'clicking the wrong button' - can happen in this toy universe; this is one possible reason why unanimity disappears (and eventually it will affect every galaxy whose initial classification was unanimous). If this were the only cause, and if all classifiers made random mistakes at the same rate (over the very long term), would classifications for 'perfect' galaxies tend towards a fixed value? If so, what?

User avatar
geckzilla
Ocular Digitator
Posts: 9180
Joined: Wed Sep 12, 2007 12:42 pm
Location: Modesto, CA
Contact:

Re: Does this effect have a name? If so, what is it?

Post by geckzilla » Tue Jan 21, 2014 3:09 am

Noise?
Just call me "geck" because "zilla" is like a last name.

User avatar
Nitpicker
Inverse Square
Posts: 2692
Joined: Fri Sep 20, 2013 2:39 am
Location: S27 E153

Re: Does this effect have a name? If so, what is it?

Post by Nitpicker » Tue Jan 21, 2014 3:18 am

What name? I may have misunderstood exactly which "this" you are talking about, but I would be tempted to call them errors. In the case of A and B you might call them one-sided errors. There are always errors when measurement and judgement are involved.

Personally, I think C would end up somewhere between 0.5 and 1.0. If you rounded all the results to just 2 or 3 significant figures, you'd probably hide the errors.

Edit: Noise is also a good word, a la geckzilla's post.

User avatar
Chris Peterson
Abominable Snowman
Posts: 18102
Joined: Wed Jan 31, 2007 11:13 pm
Location: Guffey, Colorado, USA
Contact:

Re: Does this effect have a name? If so, what is it?

Post by Chris Peterson » Tue Jan 21, 2014 3:53 am

geckzilla wrote:Noise?
It's a kind of bounded noise. The real data would be presented with a statistical uncertainty. There are clever ways of drawing error bars in cases like this (where the uncertainty in one direction would place the value outside of physical bounds). There are also analysis techniques for using these kinds of uncertainties in calculations, so they propagate correctly.
Chris

*****************************************
Chris L Peterson
Cloudbait Observatory
https://www.cloudbait.com

JeanTate
Ad Astra
Posts: 114
Joined: Wed Sep 08, 2010 12:35 am

Re: Does this effect have a name? If so, what is it?

Post by JeanTate » Tue Jan 21, 2014 4:03 pm

Thanks everyone. :ssmile:
There are clever ways of drawing error bars in cases like this (where the uncertainty in one direction would place the value outside of physical bounds). There are also analysis techniques for using these kinds of uncertainties in calculations, so they propagate correctly.
Where to find details of such clever ways?

Leaving the toy universe and returning to the real one: if I have thousands and thousands of galaxies, and if the number of classifications per galaxy (call it N) ranges from 15, say, to 150, how do I take account of the dependence of the bounded error on N? Especially for those galaxies whose 'spiral fraction's are close to 1.0 (and those whose 'elliptical fraction's are close to 1.0)?

BDanielMayfield
Don't bring me down
Posts: 2524
Joined: Thu Aug 02, 2012 11:24 am
AKA: Bruce
Location: East Idaho

Re: Does this effect have a name? If so, what is it?

Post by BDanielMayfield » Sun Feb 23, 2014 2:39 am

Having just watched the Winter Olympics, it occurs that you could handle this whole problem of weeding out erroneous judgments the same way they do in sports were subjective judging of performance is involved; toss out small number of high and low scores. As long as you toss out a slightly higher percentage of judgments than the observed error percentage the clear cases will be corrected back to 100 %, while for the somewhat vague cases this would also eliminate the small numbers of erroneous entries. For cases where you get a 50% result tossing the same numbers of spiral and elliptical responses leaves your result unchanged.

Bruce
Just as zero is not equal to infinity, everything coming from nothing is illogical.

User avatar
Chris Peterson
Abominable Snowman
Posts: 18102
Joined: Wed Jan 31, 2007 11:13 pm
Location: Guffey, Colorado, USA
Contact:

Re: Does this effect have a name? If so, what is it?

Post by Chris Peterson » Sun Feb 23, 2014 2:55 am

BDanielMayfield wrote:Having just watched the Winter Olympics, it occurs that you could handle this whole problem of weeding out erroneous judgments the same way they do in sports were subjective judging of performance is involved; toss out small number of high and low scores. As long as you toss out a slightly higher percentage of judgments than the observed error percentage the clear cases will be corrected back to 100 %, while for the somewhat vague cases this would also eliminate the small numbers of erroneous entries. For cases where you get a 50% result tossing the same numbers of spiral and elliptical responses leaves your result unchanged.
I think that would only work if your scoring ends up following a Gaussian distribution. Any skewness, or other effects which made the distribution asymmetrical, would introduce a bias. I'm skeptical, given any real controls, that events that are subjectively judged actually end up with symmetric Gaussian scores.
Chris

*****************************************
Chris L Peterson
Cloudbait Observatory
https://www.cloudbait.com

User avatar
Ann
4725 Å
Posts: 13369
Joined: Sat May 29, 2010 5:33 am

Re: Does this effect have a name? If so, what is it?

Post by Ann » Sun Feb 23, 2014 6:44 am

Image
This reminds me slightly of a method demonstrated by a math teacher who teaches at my school. Several years ago he taught in a school in Eslöv. You can find Eslöv on the map at left, close to the center of the map.

My colleague asked his student to write down the distance between Eslöv and Lund. You can find Lund on the map to the lower left of Eslöv. The true distance between Eslöv and Lund is 23 kilometers, so it isn't far, and most of the students lived in Eslöv. Since Lund is a bigger place than Eslöv, all the students would frequently visit Lund. They had a good intuitive understanding of the distance between Eslöv and Lund.

The students had to write down the distance between Eslöv and Lund, based on their own understanding of it. They were not allowed to consult one another, and since this was before the era of Google, they couldn't look it up unless they went searching for a book on geography, which they were not allowed to do. They could only make a best guess, based on their own experience, and write it down individually.

My colleague collected the paper slips with their individual answers. He looked at all the answers and discarded the answers that gave the largest and the shortest distances. He looked at the other answers, added all of them to get a figure, and then divided this figure by the number of individual answers he had received.

Imagine that there had been six students who tried to estimate the distance between Lund and Eslöv. Imagine that their six individual answers had been 10, 18, 24, 28, 30 and 40 kilometers. My colleague would disregard the "10 kilometers" and "40 kilometers" answers, and instead he would add 18 + 24 + 28 + 30 and get 100. He would then divide 100 by 4, since he added up four answers, and he would see that their "average answer" was 25 kilometers. This would be fairly correct, since the true answer is 23 kilometers.

Ann
Color Commentator

Post Reply