Bootstrapping question - comparing classifier accuracy


I'm developing a classification system for objects.

I have a total of 20 different objects that the system should classify, but I want to test the system with a varying number of classes. (3 objects at a time, 4 objects at a time etc.)

What I thought of doing is the following:
For each group size n:
divide the 20 objects into groups for n, and calculate the average accuracy (across 20/n groups). then randomly shuffle the objects into groups again, and recalculate the average accuracy.

Is this considered bootstrapping?
Is it possible to do this? There is obviously dependence between iterations since the same objects are used on each iteration.

If not, are there any ideas on how to compare the accuracy in respect to the number of objects?


Another explanation, in case this one's not very clear...:)

Think about a system that needs to decide which fruit it's looking at, and there are 20 fruits. I'd like to test the accuracy when the system only needs to decide between 3 fruits, 4 fruits, 5 fruits etc. since I have 20 fruits, I can divide the fruits into groups (e.g., show the system only: {Banana, Apple, Orange},{Pineapple,Watermelon,Grapefruit},...), and calculate the average accuracy across all groups. Then shuffle the fruits again (Now: {Banana,Pineapple,Lemon},{Apple,Orange,Date},...) and do the calculation one more time.
Last edited: