Validity of Sample?

#1
Hello, I hope you can help me as I am a bit confused as I don't normally work with one question surveys.

I have the results of a survey sent to a population (all households in a postcode area) with ONE Yes/No question.
Response rate was 1539 out of 3186 and results were 993 Yes and 546 No.

I calculated that this was valid at 99% confidence using this calculator https://www.abs.gov.au/websitedbs/d3310114.nsf/home/sample+size+calculator but strictly speaking, it was not random as was sent to the entire population (also to households and not individuals and there was no instructions for example, for the first household member with a birthday or similar to complete it)

Any help much appreciated (it isn't my survey, I was asked by someone else to give advice)
 

hlsmith

Less is more. Stay pure. Stay poor.
#2
Without random sampling and complete participation, you can have systematic error (i.e., selection bias). Typically, what would be done would be comparing the predictive population characteristics associated with participation via binary modeling (e.g., logistic regression or gboosting). Then you would balance the participating characteristics using stabilized inverse propensity score weighting. However, this process requires that you have characteristic data about the population/sample. If you don't have this information, you are naïve if your sample estimates are biased and not representative of the underlying population. If you have strong enough assumptions, you can try to perform probabilistic quantitative bias analysis to correct for possible selection bias concerns. But this requires assumptions and validation data.
 
#3
Thank you very much; I have broad data about the whole population (Census data) but not of the sample (as the survey was completely anonymous). I have told the person running the survey that I am not prepared to put my name to the advice (from me or my institution - blamed my supervisor who said pretty much what you did). I wrote a couple of paragraphs for them, saying in lay language, what you just said. This wasn't my survey as I would have done it differently (obviously), and that is why I wasn't sure how to answer the person (who is a politician). Much appreciated.