Uncertain about degrees of freedom in value with estimated standard error

I am estimating an evolutionary metric for various genes. Each estimate has an associated standard error. Typically, I'll have a set of genes for which I want to compare its mean to the mean of the rest of the genes. The size of the sets are typically in the 10 to 100 range. What I have been doing is using the estimated standard error of each value within the set to weight each value, and using the weighted Welch's t-test in R to compare the weighted means.

However, if I want to compare the value of a single gene to the rest I'm not quite sure what the best way to do that is. I could just ask if the single value is significantly different than the mean of the other set, but I would like to incorporate the estimated standard error of the single value to make the test more conservative. Can I use this standard error (with N=1) in conjunction with the weighted standard error of the set to calculate the Welch's t statistic and degrees of freedom estimate? N=1 here seems wrong to me because this N should refer to the sampling used to derive the standard error estimate. This standard error is calculated by the curvature method from an optimization algorithm so there isn't really an N associated with it. Would it be fair (conservative) to just use N=1 here to calculate the t statistic and degrees of freedom?