Item Response theory In variance

[Sourabh]

Hi, I am working on Educational assessment item calibration task.

I am new to IRT and I have read 100s of papers still could not wrap around my head with invariance property of IRT and item calibration.
My question is suppose I have response data of 20,000 students on 10 items, now I want to estimate the item difficulty for those items.
To do so I have two options, either I can use CTT to calculate the p values(which is difficulty) or fit an IRT model to estimate the item difficulty.
I tried both the ways and now I am not sure on,

1. How IRT can say that the item difficulty is not item dependent? I took the sub sample of just high scoring students from 20,000 data and re-estimated the IRT parameters and found that item difficulty has been reduced and vise versa for low scoring student sample.
2. Isn't it making IRT sample dependent?
3. If IRT is sample dependent then how CTT is different from IRT

 

[spunky]

1. How IRT can say that the item difficulty is not item dependent? I took the sub sample of just high scoring students from 20,000 data and re-estimated the IRT parameters and found that item difficulty has been reduced and vise versa for low scoring student sample.

Because invariance is a property of the population of items AND the population of examinees/test-takers. Not of any individual sample.

2. Isn't it making IRT sample dependent?

No (see previous answer).

3. If IRT is sample dependent then how CTT is different from IRT

See previous answer.

 

[Sourabh]

Because invariance is a property of the population of items AND the population of examinees/test-takers. Not of any individual sample.

Thank you so much for your reply.
If that is the case then CTT theory will also give us invariant difficulty estimate for the population right?
P value of any question/item is the probability of correct response. How can we say that IRT difficulty is invariant but CTT difficulty is sample dependent?

 

[spunky]

If that is the case then CTT theory will also give us invariant difficulty estimate for the population right?

True, you are correct. "Invariance" is a property that must be true of any statistical model. It essentially amounts to saying "the model is correct in the population" which is an implicit assumption we almost always make when analyzing data.

How can we say that IRT difficulty is invariant but CTT difficulty is sample dependent?

My guess is that you are reading sources that misinterpret the classic theorems and mathematical results concerning the property of invariance. That happens very often in psychometrics, unfortunately. Consider referring to the classic sources that state the necessary assumptions to clarify these issues. Particularly the classic Lord & Novick:

https://www.amazon.ca/Statistical-Theories-Mental-Test-Scores/dp/1593119348

Also, the reason why we do not make much a deal with invariance in CTT as opposed to IRT is because CTT is unable to give examinee-specific scores on the latent attribute. It can only give you properties of the items, whereas IRT analyses give you both properties of the items AND the examinees. That's why invariance becomes crucial now. IRT assumes sampling from TWO populations: a theoretical population of examinees and a theoretical population of items.

 

[Sourabh]

Hey.... Thankyou so much. It really helped, i hope there should be some good courses in IRT, Educational assessment and measurement like we have in other fields.