Setting decision threshold

#1
Hi,

I am helping my children's school with some data analysis of student exit surveys; the results are provided to the governing council for planning purposes. I am hoping someone can help me determine the correct approach to setting a decision threshold based on the survey results.

There is a set of questions in the survey which are phrased along the following lines:

Please tick those aspects which you consider the school has done well:

... followed by a list of a dozen or so options, e.g. "Provided enough sports facilities".

The next question is

Using the same list from above, please tick those aspects which you consider the school could most improve upon:

... followed by the same list of options.

As you can imagine, responses are varied, where for some options there was a balance of "doing well" and "needs improvement" and for others there is an obvious swing toward one or the other. In same cases the respondents checked both "doing well" and "needs improvement" for the same option on the two questions. The data could be treated either as 2 independent (?) questions with yes/no responses, or as a response with 4 labelled cases - "doing well", "doing well and needs improvement", "needs improvement" or "no response".

From this data, for each option listed, I wish to set decision thresholds so that the result can be classified into one of 3 categories, "Needs action to improve", "Pat on the back", or "Somewhere in the middle".

The approach that I am considering taking is to treat each option in each question as a separate binomial distribution (the response was either checked or not) and then compare the distributions of the two questions ("doing well" or "needs improvement") using a contingency table to test for a significant difference. Where there is a significant difference, I can classify the result as "Needs action to improve" if the larger response was against the "needs improvement" question, and "Pat on the back" if against the "doing well" question. If there is no significant difference, I classify as "Somewhere in the middle".

Does this sound reasonable or is there a better approach?

Thanks in advance
 

JohnM

TS Contributor
#2
I think your approach described in the last paragraph is quite good. Just remember to use a different approach when reporting results - i.e., avoid any mention of "contingency table" and merely say something to the effect for each aspect:

"A higher percentage of respondents felt that improvement was needed in the following areas:"

"A higher percentage of respondents felt that the school was doing quite well in the following areas:"

"Respondents were divided on their opinion in the following areas:"

....and so on.