# Calculating the probability that survey responses are arbitrary

#### Remster

##### New Member
I have a set of 200 responses to a Yes/No survey question: 160 Yes; 40 No. Any number of the responses (between 0 and 200) could be honest and truthful, and the rest will be arbitrary. Is there any way to calculate the probability that all the No responses are arbitrary? Does the question even make sense?

I'm not asking for the solution here - just some pointers. I have an inkling that chi-squared will be involved somehow, but (a) my expected value for No is 0, and (b) I have no expected value for Yes. I can't get my head round it.

#### Karabiner

##### TS Contributor
So, simply put, you have conducted a survey with one yes/no-question, and you received 160 "yes" and 40 "no"?

Could you please describe the real situation, i.e. why is this survey conducted, what is the research question, which survey question was aked?

With kind regards

Karabiner

#### katxt

##### Active Member
he probability that all the No responses are arbitrary
I can't see any way to show that all are arbitrary. It implies that all the truthful answers were Yes. Unlikely.
It reminds me of the technique for asking sensitive questions in a survey where responses can be identified. Each subject gets a secret instruction and either told to tell the truth, or else to use a coin toss to answer at random. So in the case above, if 50 subjects were told to randomize and 150 to tell the truth, then you would have about 25 random answers of each sort giving an estimate of 135 yes and 15 no. It is impossible to tell if a yes from any particular respondent is true or not.

#### Remster

##### New Member
@Karabiner. I was actually trying to simplify things in my original post, but since you ask ...

The research question is about the relationship between people's judgements about reasons and the way sentences about reasons are phrased. In the study, 600 participants were presented with the same short scenario and then asked one of three questions, 200 participants per question. Here are the questions, followed by the responses in italics:

Has he any reason to do it? Yes: 64, No: 136
Is there any reason for him to do it? Yes: 127, No: 73
Is there any reason why he should do it? Yes: 156, No: 44

What I'm trying to understand is whether there's any way to assess the amount of noise caused by arbitrary selections of 'Yes' and 'No'.

Last edited:

#### Karabiner

##### TS Contributor
Now I see. Since you only have 1 repsonse per participant,
personally I am not aware of any method which would
achieve what you have in mind.

With kind regards

Karabiner

#### Remster

##### New Member
Yes, I suppose what I need is a way of asking each participant the same question repeatedly while erasing their memory of their previous responses. What I have at the moment is a single instant per participant.

#### Karabiner

##### TS Contributor
I am not sure what you need that for. If there is random noise, then you can still
perform your group comparisons, with less statistical power.

With kind regards

Karabiner

#### Remster

##### New Member
But group comparisons aren't going to help me work out how many of the 64 Yes responses and 136 No responses to the first question are due to chance. Or are they?