Summary and Analysis of Extension Program Evaluation in R

Salvatore S. Mangiafico

Paired t-test

The paired t-test is commonly used.  It compares the means of two populations of paired observations by testing if the difference between pairs is statistically different from zero.

 

Appropriate data

•  Two-sample data.  That is, one measurement variable in two groups or samples

•  Dependent variable is interval/ratio, and is continuous

•  Independent variable is a factor with two levels.  That is, two groups

•  Data are paired.  That is, the measurement for each observation in one group can be paired logically or by subject to a measurement in the other group

•  The distribution of the difference of paired measurements is normally distributed

•  Moderate skewness is permissible if the data distribution is unimodal without outliers

 

Hypotheses

•  Null hypothesis:  The population mean of the differences between paired observations is equal to zero.

•  Alternative hypothesis (two-sided): The population mean of the differences between paired observations is not equal to zero.

 

Interpretation

Reporting significant results as “Mean of variable Y for group A was different than that for group B.” or “Variable Y increased from before to after” is acceptable.

 

Other notes and alternative tests

•  The nonparametric analogue for this test is the two-sample paired rank-sum test.

•  Power analysis for the paired t-test can be found at Mangiafico (2015) in the “References” section.

 

Packages used in this chapter

 

The packages used in this chapter include:

•  psych

•  rcompanion

•  lsr

 

The following commands will install these packages if they are not already installed:

if(!require(psych)){install.packages("psych")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(lsr)){install.packages("lsr")}

Paired t-test example

 

In the following example, Dumbland Extension had adult students fill out a financial literacy knowledge questionnaire both before and after completing a home financial management workshop.  Each student’s score before and after was paired by student.

 

Note in the following data that the students’ names are repeated, so that there is a before score for student a and an after score for student a.

 

Since the data is in long form, we’ll order by Time, then Student to be sure the first observation for Before is student a and the first observation for After is student a, and so on.

 

Input = ("
Time    Student  Score
Before  a         65
Before  b         75
Before  c         86
Before  d         69
Before  e         60
Before  f         81
Before  g         88
Before  h         53
Before  i         75
Before  j         73
After   a         77
After   b         98
After   c         92
After   d         77
After   e         65
After   f         77
After   g        100
After   h         73
After   i         93
After   j         75
")

Data = read.table(textConnection(Input),header=TRUE)


###  Order data by Time and Student

Data = Data[order(Time, Student),]


###  Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)


### Remove unnecessary objects

rm(Input)

Check the number of paired observations

It is helpful to create a table of the counts of observations to be sure that there is one observation for each student for each time period.

xtabs(~ Student + Time,
      data = Data)

       Time
Student After Before
      a     1      1
      b     1      1
      c     1      1
      d     1      1
      e     1      1
      f     1      1
      g     1      1
      h     1      1
      i     1      1
      j     1      1

Histogram of difference data

A histogram with a normal curve imposed will be used to check if the paired differences between the two populations is approximately normal in distribution.

First, two new variables, Before and After, are created by extracting the values of Score for observations with the Time variable equal to Before or After, respectively.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = After - Before


x = Difference

library(rcompanion)

plotNormalHistogram(x,
                    xlab="Difference (After - Before)")

Plot the paired data

 

Scatter plot with one-to-one line

Paired data can visualized with a scatter plot of the paired cases.  In the plot below, points that fall above and to the left of the blue line indicate cases for which the value for After was greater than for Before.

 

Note that the points in the plot are jittered slightly so that points that would fall directly on top of one another can be seen.

 

First, two new variables, Before and After, are created by extracting the values of Score for observations with the Time variable equal to Before or After, respectively.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

 

A variable Names is also created for point labels.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Names  = Data$Student[Data$Time=="Before"]

                      
plot(Before, jitter(After),    # jitter offsets points so you can see them all
     pch = 16,                 # shape of points
     cex = 1.0,                # size of points
     xlim=c(50, 110),          # limits of x-axis
     ylim=c(50, 110),          # limits of y-axis
     xlab="Before",            # label for x-axis
     ylab="After")             # label for y-axis

text(Before, After, labels=Names,  # Label location and text

     pos=3, cex=1.0)               # Label text position and size

abline(0, 1, col="blue", lwd=2)     # line with intercept of 0 and slope of 1

Bar plot of differences

Paired data can also be visualized with a bar chart of differences.  In the plot below, bars with a value greater than zero indicate cases for which the value for After was greater than for Before.

 

New variables are first created for Before, After, and their Difference.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

 

A variable Names is also created for bar labels.

 

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = After - Before

Names = Data$Student[Data$Time=="Before"]


barplot(Difference,                             # variable to plot
        col="dark gray",                        # color of bars
        xlab="Observation",                     # x-axis label
        ylab="Difference (After – Before)",     # y-axis label
        names.arg=Names)                        # labels for bars

Paired t-test

Note that the output shows the p-value for the test, and the simple difference in the means for the two groups.

 

Note that for this test to be conducted correctly, the first observation for Before is student a and the first observation for After is student a, and so on.

t.test(Score ~ Time,
       data   = Data,
       paired = TRUE)

Paired t-test

t = 3.8084, df = 9, p-value = 0.004163
alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:
  4.141247 16.258753

sample estimates:
mean of the differences
                   10.2

Effect size

 

Cohen’s d can be used as an effect size statistic for a paired t-test.  It is calculated as the difference between the means of each group, all divided by the standard deviation of the data.  The standard deviation used could be calculated from the differences between observations, or, for observations across two times, the observations in the “before” group.

 

It ranges from 0 to infinity, with 0 indicating no effect where the means are equal.  In some versions, Cohen’s d can be positive or negative depending on which mean is greater.

 

A Cohen’s d of 0.5 suggests that the means differ by one-half the standard deviation of the data.  A Cohen’s d of 1.0 suggests that the means differ by one standard deviation of the data.

 

Interpretation of Cohen’s d

Interpretation of effect sizes necessarily varies by discipline and the expectations of the experiment, but for behavioral studies, the guidelines proposed by Cohen (1988) are sometimes followed.  They should not be considered universal.

 

	Small	Medium	Large
Cohen’s d	0.2 – < 0.5	0.5 – < 0.8	≥ 0.8

____________________________

Source: Cohen (1988).

 

 

Cohen’s d for paired t-test

 

Standard deviation calculated from differences in observations

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

library(lsr)

cohensD(Score ~ Time,
       data   = Data,
       method = "paired")

[1] 1.204314

Or we can calculate the value manually.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = After – Before


mean(Before)

[1] 72.5

mean(After)

[1] 82.7

mean(Before) - mean(After)

[1] -10.2

( mean(Before) - mean(After) ) / sd(Difference)

[1] -1.204314

Standard deviation calculated from the “before” observations

Note that for this code to make sense, the “before” observations must correspond to the second level of the Time variable, corresponding to y.sd in the function.

library(lsr)

cohensD(Score ~ Time,
       data   = Data,
       method = "y.sd")

[1] 0.9174268

( mean(Before) - mean(After) ) / sd(Before)

[1] -0.9174268

Optional readings

 

“Paired t–test” in McDonald, J.H. 2014. Handbook of Biological Statistics. www.biostathandbook.com/pairedttest.html.

 

References

 

“Paired t–test” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological Statistics, version 1.09. rcompanion.org/rcompanion/d_09.html.

 

Exercises Q

 

1. Consider the Dumbland Extension data.  Report for each answer How you know, when appropriate, by reporting the values of the statistic you are using or other information you used.

a.  What was the mean of the differences in score before and after the training?

b.  Was this an increase or a decrease?

c.  Is the data distribution for the paired differences reasonably normal?

d.  Was the mean score significantly different before and after the training?

e.  What do you conclude practically?  As appropriate, report the means before and after, the mean difference, the effect size and interpretation, whether the difference is large relative to the scoring system, anything notable on plots, and your practical conclusions.

2. Residential properties in Dougal County rarely need phosphorus for good turfgrass growth.  As part of an extension education program, Early and Rusty Cuyler asked homeowners to report their phosphorus fertilizer use, in pounds of P₂O₅ per acre, before the program and then one year later.

Date              Homeowner  P2O5
'2014-01-01'      a          0.81
'2014-01-01'      b          0.86
'2014-01-01'      c          0.79
'2014-01-01'      d          0.59
'2014-01-01'      e          0.71
'2014-01-01'      f          0.88
'2014-01-01'      g          0.63
'2014-01-01'      h          0.72
'2014-01-01'      i          0.76
'2014-01-01'      j          0.58
'2015-01-01'      a          0.67
'2015-01-01'      b          0.83
'2015-01-01'      c          0.81
'2015-01-01'      d          0.50
'2015-01-01'      e          0.71
'2015-01-01'      f          0.72
'2015-01-01'      g          0.67
'2015-01-01'      h          0.67
'2015-01-01'      i          0.48
'2015-01-01'      j          0.68

For each of the following, answer the question, and show the output from the analyses you used to answer the question.

 

a.  What was mean of the differences in P₂O₅ before and after the training?

b.  Is this an increase or a decrease?

 

c.  Is the data distribution for the paired differences reasonably normal?

d.  Was the mean P₂O₅ use significantly different before and after the training?

e.  What do you conclude practically?  As appropriate, report the means before and after, the mean difference, the effect size and interpretation, whether the difference is large relative to the values, anything notable on plots, and your practical conclusions.

©2016 by Salvatore S. Mangiafico.
Rutgers Cooperative Extension, New Brunswick, NJ.

Non-commercial reproduction of this content, with attribution, is permitted.
For-profit reproduction without permission is prohibited.

If you use the code or information in this site in a published work, please cite it as a source.  Also, if you are an instructor and use this book in your course, please let me know.   My contact information is on the About the Author of this Book page.

Citation

Mangiafico, S.S. 2016. Summary and Analysis of Extension Program Evaluation in R, version 1.20.05, revised 2023. rcompanion.org/handbook/. (Pdf version: rcompanion.org/documents/RHandbookProgramEvaluation.pdf.)