An SNJ Associates Series: Research Methodology
Issue 4: Hypothesis Testing (No Math….Yet)
The focus of this post is hypothesis testing in research that uses a quantitative methodology. This post covers the very basic introduction of hypothesis testing. We are going to look at the language of hypothesis testing and define certain parts of hypothesis testing. Purposely, we do not go into the detail of the mathematical, statistical (probability), or calculations required to carry out hypothesis testing. Rather than a ‘how-to’ complete hypothesis testing, we cover the topic in plain English here so our readers may develop a concept of the idea of hypothesis testing and learn the language of hypothesis testing before proceeding into ‘how’ to complete a hypothesis test.
In our last post, Issue 3 we had a look at operationalizing variables in a quantitative study. Previous to our discussion about how to define and measure variables, Issue 2 defined what a variable was and discussed both independent and dependent variables in quantitative research methodology. [If you would like to start at the beginning of our Research Methodology Series please visit here].
What is Hypothesis Testing?
In quantitative methodology, we always work with a research question and quite often a hypothesis. The purpose of any hypothesis test in a research study is to decide whether or not there is sufficient statistical evidence to support a research hypothesis. For example, we have a group of patients, who are randomly selected to participate in a clinical drug trial investigating the effectiveness of a new drug. One-half of the group will receive the new drug while the other half of the patient group will continue to use the current drug used for their condition. The variables for the clinical trial are measured and the results compared between the two groups. Our hypothesis is that the new drug performs better across a number of markers compared to the current drug. Note, you may see hypothesis testing also referred to as statistical testing.
There are 4 basic components of hypothesis testing:
- The null and alternative hypothesis
- The test statistic
- A P-value (and interpretation of that value)
- The significance level
A study’s research question, which we consistently emphasize here at SNJ Associates is so important to good research is converted into the null and alternative hypothesis. When stating a hypothesis there are two main pieces. The first piece is the null hypothesis and you will see it written like this ‘Ho:’. The second piece of the hypothesis is the alternative hypothesis and it is written like this ‘Ha:’.
The null hypothesis is ALWAYS the piece that means no difference or equal. In our example, the null hypothesis would mean that the new drug is no more effective than the current drug in treating a condition. The null hypothesis is stating that there is no difference between the two drugs or that they are equal in effectiveness.
The alternative hypothesis is the statement that represents the best guess researchers have about what will happen in their study. The alternative hypothesis may also be referred to as the research hypothesis. The alternative hypothesis makes the claim that the null hypothesis must be rejected (in our study we would conclude our new drug is more effective than our currently used drug thus reject the null that the two drugs are equal).
In quantitative research, we are collecting data and using evidence that will support the alternative hypothesis, which is known as deductive logic. As a point of clarification, the alternative hypothesis may be one-sided (our drug was better) or the alternative hypothesis may be two-sided (our new drug could be better but could also be worse that the currently used drug). Once the data is collected a test statistic is derived. There are a number of different test statistics used in completing hypothesis testing; however, we will not cover the details of that here. Suffice to say a test statistic, for example, a Z-statistic (z-score) is used as an anchor to determine if the probability of the observations of a study contain evidence to support the alternative hypothesis. Once there is a test statistic the final step is to convert the test statistic to a P-value.
P-Values
The P-value answers the question ‘What is the probability of the test statistic when the null hypothesis is TRUE’. A key point to remember is that the smaller the P-value such as .05, .10 or .001 all mean that there is evidence against the null hypothesis. Conversely, if the p-value is quite large, say .75 or .99 or higher, then there is more evidence the null hypothesis of no difference cannot be rejected.
Just remember a small p-value=strong evidence and we’ll have to reject the null hypothesis and discard any statements of no difference. In our example, if the test statistics for the new drug was converted to a p-value that value turned out to be quite small, say .01, we would have evidence to reject our null hypothesis and state the new drug and the old drug are showing different outcomes under the same conditions thus, there is a difference.
When you are looking at p-value presented in the literature use the following guide:
P-value of greater than 0.10, the conclusion is there is no significant evidence against the null hypothesis of no difference
P-value of 0.05 up to 0.10, the conclusion is there is just barely significant evidence against the null hypothesis of no difference
P-value of 0.01 up to 0.05, the conclusion is there is significant evidence against the null hypothesis of no difference
P-value of less than 0.01, the conclusion is there is highly significant evidence against the null hypothesis of no difference
*It is a slippery slope to be too rigid in using the p-value cut-offs as presented here because there are many factors that go into making accurate hypothesis testing conclusions; however when you are learning about quantitative research methodology, the list above may act as a guide for you when considering what p-values mean. What is important to know when you are critiquing any study is to look at how the research question was converted into the null and alternative hypothesis, understand what variables are important for the hypothesis, how the variables were defined and measured along with what was the p-value calculated in the study. If you see a p-value between .01 and .05 and the authors are stating that their study is highly significant you know the results are being somewhat exaggerated because significant evidence means the p-value must be a lot smaller and you can delve further into their conclusions.
What is important to know when you are critiquing any study is to look at how the research question was converted into the null and alternative hypothesis, understand what variables are important for the hypothesis, how the variables were defined and measured along with what was the p-value calculated in the study. If you see a p-value between .01 and .05 and the authors are stating that their study is highly significant, you know the results are being somewhat exaggerated because significant evidence means the p-value must be a lot smaller and you can delve further into their conclusions.
Errors Associated with Hypothesis Testing
There are two types of errors you will see regarding hypothesis testing. Type I error is when the null hypothesis of no difference is rejected; however, the null hypothesis is actually true and there really is no difference. This is also referred to as a false positive. Additionally, Type 2 error is failing to reject the null hypothesis that is in fact, false. This can be a bit complicated to get your head around, don’t become discouraged. Knowing that a Type 1 error will lead to conclusions that there is an ‘effect’ of some item when actually, there is not an affect. An example could be a diagnostic test stating a patient has a particular disease when in fact they do not have the disease.
Conversely, a patient who goes for a blood test designed to detect a disease and the test fails to show evidence of a disease when that patient actually has the disease is a Type 2 error. All statistical tests have some probability of making both Type 1 and Type 2 errors. Many steps in hypothesis testing such as determining the ‘power’ of the study and calculating sample size requirements are used to reduce the likelihood of Type 1 and Type 2 errors as much as possible.
The take home information here is to know that there is always an error associated with hypothesis testing and there are two different types of errors.
Our language in hypothesis testing is crucial and may appear awkward at first. Reject the null hypothesis versus failing to reject the null hypothesis may seem very complicated at first. Note, we never, ever never say we accept the null hypothesis.
Let us try and explain why……we design our studies and test our observations in order to examine whether or not the outcome of a study is consistent with the null hypothesis. We assume the null hypothesis is true. This is an assumption that is made. We either reject the null hypothesis as we have significant evidence in support of our research or alternative hypothesis or we fail to reject the null hypothesis and conclude there is no difference – that’s it, two choices. When we fail to reject the null hypothesis, or conclude we have poor evidence in support of our research hypothesis, we are really saying our best evidence suggests within the range of what we are looking at there is a higher probability of a value of no difference.
If you are a student and you are ever asked a question on an exam about ‘accepting’ the null hypothesis run – run – run. We never accept the null don’t do it! Here is a link to a video by Kevin Brown, who does a good job of explaining why we never accept the null.
It is beyond the scope of this post to delve into items such as the test statistic, called the z-statistics and the details of probability as it becomes quite complicated. The purpose here is to explain the overall components of hypothesis testing and introduce some basic hypothesis testing terminology (such as the null and alternative/research hypotheses) so our readers can begin to identify these components of a hypothesis in the literature. Our goal at this point is not to perform the calculations; however, we do encourage you to learn how to complete the process of hypothesis testing as you will acquire a deeper knowledge of quantitative methodology if you do. In our in-depth courses at SNJ Associates, we cover probability, sampling distributions and the process required to calculate p-values.
Sigh…..we hope this helps you and we understand it is difficult to explain and understand what hypothesis testing is all about. Stick with it and use different resources and presentations on the subject of hypothesis testing. If you are becoming comfortable with what the null hypothesis is in research and know what question a p-value answers you are making great progress in your quest to learn more about quantitative methodology in scientific research.
Issue 5 will discuss sampling in quantitative methodology
`