personality psych)self-selected sample bias = smaller variability?

[mungman]

Context: online personality survey (BFI) for age 10~65 (http://www.colby.edu/psychology/labs/personality/publications/Soto_et_al_2011.pdf)
Measure taken: the authors tried to control for possible self-selection bias in the sample by "by examining the variability of Big Five domains and
facets in different age groups."
Specifically, they claim the following: "if our youngest and oldest participants were indeed self-selected on the basis of their personality traits, then the observed variability of scores on those traits would be largest in adolescence and emerging adulthood (where agespecific samples would be most representative) and smallest in late childhood and late middle age (where age-specific samples would be most self-selected)."

Authors' conclusion: Since they didn't find any significant differences between these age groups, self-selection doesn't have enough of an influence on the results!

Although I get the part about self-selecting factions likely being more similar to each other than more representative sectors, I don't think that necessarily eliminates the possibility of self-selection having impacted the results. Also, I don't get why they would just consider the youngest and oldest groups and call it a day... Anyone understand the rationale of this and/or know of a standard "protocol" involved in controlling for self-selected samples?

 

[CB]

Specifically, they claim the following: "if our youngest and oldest participants were indeed self-selected on the basis of their personality traits, then the observed variability of scores on those traits would be largest in adolescence and emerging adulthood (where agespecific samples would be most representative) and smallest in late childhood and late middle age (where age-specific samples would be most self-selected)."

That doesn't make much sense to me. Self-selection might just as easily lead to people with extreme levels (high or low) of some personality trait(s) being over-represented, and greater variance in the sample as opposed to the population. It might also lead to a biased mean, but similar variance to the population of interest; there's no real way to tell. If dealing with self-selection bias was as easy as checking that a sample or subsample variance is within some expected range, survey research would be a helluva lot easier than it actually is.

Anyone understand the rationale of this and/or know of a standard "protocol" involved in controlling for self-selected samples?

Not really. It's not the kind of issue for which there is any quick fix, I think!

 

[spunky]

Authors' conclusion: Since they didn't find any significant differences between these age groups, self-selection doesn't have enough of an influence on the results!

this is so wrong at so many levels it makes me wonder whether JPSP still has any methodology specialists doing reviews or not... (notice how they try to follow the logic of the original, most basic form of the F-test which has been beaten to death because of how sensitive it is to violation of assumptions??)

Anyone understand the rationale of this and/or know of a standard "protocol" involved in controlling for self-selected samples?

i guess once you found on which areas you're having over-representativeness and under-representativeness, you can do some smart sampling weights and weight the hell out of all your data to try and account for the lack of a random sample...that might fix some of the problems but, as Cowboybear said, it ain't easy (and sometimes it even aint meaningful anymore...)