I’m running a surveylogistic procedure with complex sampling variables (stratification, cluster, and weight) to account for sample size in my estimates. Everything is suspiciously significant. Suspicious because when I run the model on a single year some variables prove non-significant. I’m afraid the complex sampling variables aren’t actually adjusting the sample so I decided to randomly split the sample into training and test samples for analysis. When I do this I get some non-significant covariates which feels right.

My question is am I doing something wrong with the full sample and is it inappropriate to take a train and test approach when I’m not really working with a ton of variables. I would like to use the full sample. Any help is appreciated.