Post #2292063

2025-03-26 05:49 UTC

@satrevik@fediscience.org Is this a realistic question ? With two tests A and B I suppose you mean two different statistical approaches ? I could see that one is not significant but the other is, but when they go in different directions, I would suppose that you have a problem with your basic statistical assumption. Either the mean of population X is higher than that of population Y, it cannot be smaller at the same time. But perhaps I don't understand your question, then please give an example.

Replies (1)

@satrevik@fediscience.org 2025-03-26 09:50

@olibrendel@scicomm.xyz I would say it's fairly realistic for social sciences, at least. Here effects are often small and measurements noisy. With large survey samples a test may indicate a small effect to be significant at a 5% level, while there may actually be a such effect (a true positive), no real effect (a false positive), or a false positive where the true effect goes in the opposite direction of what's indicated. Related: https://en.wikipedia.org/wiki/Simpson%27s_paradox I was thinking of a situation where test A and test B both test the same hypothesis with e.g. different operationalizations of the outcome variable. In social science a given study often make several tests/measurements of a given phenomenon, and compare the results.

Open ##2292064