Degrees of freedom

Mark17

New Member
Hi,

Yesterday I did some backtesting of algorithmic trading systems to study 4-rule (n = 67) versus 2-rule (n = 65) strategies. I ran five two-sample t-tests assuming unequal variances (in Excel) to compare profit/loss (PNL), PNL/max drawdown, # trades, average trade, and profit factor.

Across the five tests, degrees of freedom varied from 76 to 125. I thought df was a defined number based on sample size, but obviously I'm wrong. Why would this vary?

Thanks,
Mark

obh

Active Member
Hi Mark,

Are several t-tests related?
when you run several tests the probably to reject the null assumption is higher.

For equal 2 sample t-test with equal variances, DF=n1-1+n2-1=n1+n2-2

The DF in Welch t-test, unequal variances, is more complex, following the approximation
DF=(s1^2/n1+S2^2/n2)^2 / ( s1^4 / n1^2(n1-1)+ s2^4 / n2^2(n2-1) )

https://en.wikipedia.org/wiki/Welch–Satterthwaite_equation

Mark17

New Member
They are related in that I compared the same two groups along the lines of five different variables.

That makes sense if df is related to pooled variance. PNLDD is a number between -1 and 7 or so whereas PNL is between +/- 5000.

Thanks!
Mark

obh

Active Member
Hi Mark,

You may look at multiple comparisons, to think if you need to take a smaller significance level.

As you can see it is the opposite if it is pooled variance the DF calculation is simple.
But for unequal variances, it is more complex (pooled DF)

Mark17

New Member
If I were using Bonferroni correction, could I just divide 0.05 by five for five comparisons?

Mark

obh

Active Member
If I were using Bonferroni correction, could I just divide 0.05 by five for five comparisons?

Mark
Hi Mark,

Correct, but the Bonferroni is not accurate better use the more accurate, The Sidak
But the difference is minor 0.010206 instead of 0.01

The problem of both calculations, that it protects against type 1 error at the expense of increasing type 2 error.
it assumes total independence which is usually not the case.

The Holm correction is a bit more balanced.
http://www.statskingdom.com/doc_anova.html#holm_calculator

Last edited: