Is Chi Square Even The Right Test?!?

I recently collected data for a study that mesaured participants comprehension ratings of collision avoidance warnings. Each warning was presented twice. We have divided up the ratings into two categories and recoded the data. A score of 2 or less = success and a score of 3-7 = failure. My boss said that we sould run a Chi Sqaure. However, when we do, all we are getting is frequency tables that tell us how many people were successful and unsuccessful for each individual warning trial. We would like to know if there was a significant difference between the success and failure rates. I am at a total loss...any help would be appreciated. I have been told that our DV is comprehension rating (successful or unsuccessful) and our IVs are the warnings (16 overall, 8 warnings presented two times each).

I set up the data in SPSS with participants 1-25 as the first column and then the rest of the columns were for each of the 16 warnings. Each of the columns with the warnings have been coded as 0 for unsuccessful and 1 for successful for participants comprehension score. Are we even setting up the data correctly? Is Chi sqaure the right test? HELP!!! :0)