Which statistical test suits my questionnaire?


New Member
My research involves constraining the English language to a lexicon of fewer words (e.g. by ensuring one-word one-meaning and removing redundant words). My hypothesis is that text written in the constrained form of English is:
1. No less correct than the same text written in unconstrained English
2. More understandable than the same text written in unconstrained English

In my questionnaire, I would like to ask the participants comparative-style questions – such as whether the constrained text is (in their opinion) more/less correct/understandable than the unconstrained text. This type of questioning seems more appropriate given the hypothesis I am testing – however I am not clear on which statistical test to use with the comparative-style answers? Would the answers simply be giving me a ranking (i.e. 1 or 2) such that I could use the Mann-Whitney U test?

It seems like a more common approach would be to ask the participants to rate both the constrained and unconstrained text in terms of both correctness and understandability using some ordinal scale (e.g. 1-5). Since the results would (I think) be non-parametric, I understand I could use the Mann-Whitney U test to test each of my two hypotheses?

I would greatly appreciate some advice both on the design of my experiment and on the statistical test to use. I am a relative novice to statistics – so would greatly appreciate simple answers (or pointers to where I can learn more).

Interesting topic.
I myself am prefer experimental manipulations to questionnaires, so if it was me I would test each paragraph type for understandability directly. To do this you could give subjects constrained and unconstrained text and ask them questions about the text to see how much each group understands.
However, if you feel that the questionnaire approach would be best for you then you may want to consider giving people 10 (or more) short examples of each text type and have them rank them for understandability on a likert type scale. Then you could take the mean of the ranking to get an average rating for each type of text for each subject. You could analyze those means using a paired sample t-test.

I'm not sure how much inference you're trying to make... If you want results to apply to a wide range of the population (then say white middle class university students) watch out for what kind of people you're testing (education, age, IQ, gender, ethnic background, Are they ESL or ETL? etc) . I highly recommend if you're in a university environment to make an appointment with some stats people whom you can talk to about your experiment. To have proper statistical validity requires some knowledge of Design of Experiments to make wide inference.