Summary and Analysis of Extension Program Evaluation in R

Salvatore S. Mangiafico

Paired t-test

The paired t-test is commonly used.  It compares the means of two populations of paired observations by testing if the difference between pairs is statistically different from zero.

 

Appropriate data

•  Two-sample data.  That is, one measurement variable in two groups or samples

•  Dependent variable is interval/ratio, and is continuous

•  Independent variable is a factor with two levels.  That is, two groups

•  Data are paired.  That is, the measurement for each observation in one group can be paired logically or by subject to a measurement in the other group

•  The distribution of the difference of paired measurements is normally distributed

•  Moderate skewness is permissible if the data distribution is unimodal without outliers

 

Hypotheses

•  Null hypothesis:  The population mean of the differences between paired observations is equal to zero.

•  Alternative hypothesis (two-sided): The population mean of the differences between paired observations is not equal to zero.

 

Interpretation

Reporting significant results as “Mean of variable Y for group A was different than that for group B.” or “Variable Y increased from before to after” is acceptable.

 

Other notes and alternative tests

•  The nonparametric analogue for this test is the two-sample paired rank-sum test.

•  Power analysis for the paired t-test can be found at Mangiafico (2015) in the “References” section.

 

Packages used in this chapter

 

The packages used in this chapter include:

•  psych

•  rcompanion

•  lsr

•  effsize

•  DescTools

 

 

The following commands will install these packages if they are not already installed:

if(!require(psych)){install.packages("psych")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(lsr)){install.packages("lsr")}
if(!require(effsize)){install.packages("effsize")}
if(!require(DescTools)){install.packages("DescTools")}

Paired t-test example

 

In the following example, Dumbland Extension had adult students fill out a financial literacy knowledge questionnaire both before and after completing a home financial management workshop.  Each student’s score before and after was paired by student.

 

Note in the following data that the students’ names are repeated, so that there is a before score for student a and an after score for student a.

 

Since the data is in long form, we’ll order by Time, then Student to be sure the first observation for Before is student a and the first observation for After is student a, and so on.

 

Data = read.table(header=TRUE, stringsAsFactors=TRUE, text="

Time    Student  Score
Before  a         65
Before  b         75
Before  c         86
Before  d         69
Before  e         60
Before  f         81
Before  g         88
Before  h         53
Before  i         75
Before  j         73
After   a         77
After   b         98
After   c         92
After   d         77
After   e         65
After   f         77
After   g        100
After   h         73
After   i         93
After   j         75
")


###  Order data by Time and Student

Data = Data[order(Data$Time, Data$Student),]

###  Order levels of Time so that “Before” is the first group

 

Data$Time = factor(Data$Time, levels = c("Before", "After"))

###  Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

Check the number of paired observations

It is helpful to create a table of the counts of observations to be sure that there is one observation for each student for each time period.

xtabs(~ Student + Time,
      data = Data)

       Time
Student Before After
      a      1     1
      b      1     1
      c      1     1
      d      1     1
      e      1     1
      f      1     1
      g      1     1
      h      1     1
      i      1     1
      j      1     1

Histogram of difference data

A histogram with a normal curve imposed will be used to check if the paired differences between the two populations is approximately normal in distribution.

First, two new variables, Before and After, are created by extracting the values of Score for observations with the Time variable equal to Before or After, respectively.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = After - Before


x = Difference

library(rcompanion)

plotNormalHistogram(x,
                    xlab="Difference (After - Before)")

Plot the paired data

 

Scatter plot with one-to-one line

Paired data can visualized with a scatter plot of the paired cases.  In the plot below, points that fall above and to the left of the blue line indicate cases for which the value for After was greater than for Before.

 

Note that the points in the plot are jittered slightly so that points that would fall directly on top of one another can be seen.

 

First, two new variables, Before and After, are created by extracting the values of Score for observations with the Time variable equal to Before or After, respectively.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

 

A variable Names is also created for point labels.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Names  = Data$Student[Data$Time=="Before"]

                      
plot(Before, jitter(After),    # jitter offsets points so you can see them all
     pch = 16,                 # shape of points
     cex = 1.0,                # size of points
     xlim=c(50, 110),          # limits of x-axis
     ylim=c(50, 110),          # limits of y-axis
     xlab="Before",            # label for x-axis
     ylab="After")             # label for y-axis

text(Before, After, labels=Names,  # Label location and text
     pos=3, cex=1.0)               # Label text position and size

abline(0, 1, col="blue", lwd=2)     # line with intercept of 0 and slope of 1

Bar plot of differences

Paired data can also be visualized with a bar chart of differences.  In the plot below, bars with a value greater than zero indicate cases for which the value for After was greater than for Before.

 

New variables are first created for Before, After, and their Difference.

 

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

 

A variable Names is also created for bar labels.

 

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = After - Before

Names = Data$Student[Data$Time=="Before"]


barplot(Difference,                             # variable to plot
        col="dark gray",                        # color of bars
        xlab="Observation",                     # x-axis label
        ylab="Difference (After – Before)",     # y-axis label
        names.arg=Names)                        # labels for bars

Paired t-test

Note that the output shows the p-value for the test, and the simple difference in the means for the two groups.

 

Unfortunately, R no longer allows for the use of the old formula interface for the paired t-test.  So, new variables, Before and After, will be created to run this test.

 

Note that the t value is negative, suggesting that the second group has higher values than the first group.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

t.test(Before, After, paired = TRUE)

Paired t-test

t = -3.8084, df = 9, p-value = 0.004163
alternative hypothesis: true mean difference is not equal to 0

95 percent confidence interval:
 -16.258753  -4.141247

sample estimates:
 mean difference
-10.2

Effect size

 

Cohen’s d can be used as an effect size statistic for a paired t-test.  It is calculated as the mean difference between the groups divided by the standard deviation of the differences.  The standard deviation used could be calculated from the differences between observations, or, for observations across two times, the observations in the “before” group.

 

It ranges from 0 to infinity, with 0 indicating no effect where the means are equal.  In some versions, Cohen’s d can be positive or negative depending on whether the mean of the paired differences are greater than or less than zero.

 

A Cohen’s d of 0.5 suggests that the mean of the differences is one-half the standard deviation of the differences.  A Cohen’s d of 1.0 suggests that the mean of the differences is one standard deviation of the differences.

 

Interpretation of Cohen’s d

Interpretation of effect sizes necessarily varies by discipline and the expectations of the experiment, but for behavioral studies, the guidelines proposed by Cohen (1988) are sometimes followed.  They should not be considered universal.

 

	Small	Medium	Large
Cohen’s d	0.2 – <   0.5 −0.2 – > −0.5	0.5 – <   0.8 −0.5 – > −0.8	≥   0.8 ≤ −0.8

____________________________

Source: Cohen (1988).

 

 

Cohen’s d for one-sample t-test

The cohensD function in the lsr package has an option for the one-sample t-test.  Note that this function reports the value as a positive number. 

 

The effsize and DescTools packages could also be used.

 

By convention, a positive Cohen’s d suggests that the values for the first group tend to be larger than the values in the second group.  And a negative Cohen’s d suggests that the values for the first group tend to be smaller than the values in the second group.  By convention, the statistic follows the formula

Cohen’s d = mean(Before - After) / sd(Before - After)

Not all software follows this convention, however.  It’s always best to check the mean of the differences of the paired data or the means of each group or the means of the groups.

 

Standard deviation calculated from differences in observations

Note that for this code to make sense, the first observation for Before is student a and the first observation for After is student a, and so on.

library(lsr)

cohensD(Score ~ Time,
       data   = Data,
       method = "paired")

[1] 1.204314

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = Before – After

library(DescTools)

CohenD(Difference)

[1] -1.204314

attr(,"magnitude")
[1] "large

Or we can calculate the value manually.

Before = Data$Score[Data$Time=="Before"]

After  = Data$Score[Data$Time=="After"]

Difference = Before - After

mean(Before)

[1] 72.5

mean(After)

[1] 82.7

mean(Before) - mean(After)

[1] -10.2

mean(Before - After) / sd(Difference)

[1] -1.204314

Accounting for the correlation among paired values

The effsize package uses a method for the paired case where the correlation between the Before observations and After observations is taken into account.  This results in a different value for the statistic.  This is discussed in Cohen (1988).

library(effsize)

cohen.d(Score ~ Time, data = Data, paired=TRUE)

Cohen's d

d estimate: -0.8795659 (large)

95 percent confidence interval:
 lower      upper
-1.4509767 -0.3081552

An alternative grammar can be used where the value of the pairing variable is specified.

library(effsize)

cohen.d(Score ~ Time | Subject(Student), data = Data, paired=TRUE)

Cohen's d

d estimate: -0.8795659 (large)

95 percent confidence interval:
 lower      upper
-1.4509767 -0.3081552

This can be calculated manually.

Difference = Before – After

R=cor(Before, After)

R

0.7332973

S.Dif=sd(Difference)

StDev = S.Dif / sqrt(2-2*R)

StDev

11.59663

mean(Difference) / StDev

-0.8795659

Standard deviation calculated from the “before” observations

Note that for this code to make sense, the “before” observations must correspond to the first level of the Time variable, corresponding to x.sd in the function.

library(lsr)

cohensD(Score ~ Time,
       data   = Data,
       method = "x.sd")

[1] 0.9174268

mean(Before - After ) / sd(Before)

[1] -0.9174268

Optional readings

 

“Paired t–test” in McDonald, J.H. 2014. Handbook of Biological Statistics. www.biostathandbook.com/pairedttest.html.

 

References

 

 

Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd Edition. Routledge.

 

“Paired t–test” in Mangiafico, S.S. 2015. An R Companion for the Handbook of Biological Statistics. rcompanion.org/rcompanion/d_09.html.

 

Exercises Q

 

1. Consider the Dumbland Extension data.  Report for each answer How you know, when appropriate, by reporting the values of the statistic you are using or other information you used.

a.  What was the mean of the differences in score before and after the training?

b.  Was this an increase or a decrease?

c.  Is the data distribution for the paired differences reasonably normal?

d.  Was the mean score significantly different before and after the training?

e.  What do you conclude practically?  As appropriate, report the means before and after, the mean difference, the effect size and interpretation, whether the difference is large relative to the scoring system, anything notable on plots, and your practical conclusions.

2. Residential properties in Dougal County rarely need phosphorus for good turfgrass growth.  As part of an extension education program, Early and Rusty Cuyler asked homeowners to report their phosphorus fertilizer use, in pounds of P₂O₅ per acre, before the program and then one year later.

Date              Homeowner  P2O5
'2014-01-01'      a          0.81
'2014-01-01'      b          0.86
'2014-01-01'      c          0.79
'2014-01-01'      d          0.59
'2014-01-01'      e          0.71
'2014-01-01'      f          0.88
'2014-01-01'      g          0.63
'2014-01-01'      h          0.72
'2014-01-01'      i          0.76
'2014-01-01'      j          0.58
'2015-01-01'      a          0.67
'2015-01-01'      b          0.83
'2015-01-01'      c          0.81
'2015-01-01'      d          0.50
'2015-01-01'      e          0.71
'2015-01-01'      f          0.72
'2015-01-01'      g          0.67
'2015-01-01'      h          0.67
'2015-01-01'      i          0.48
'2015-01-01'      j          0.68

For each of the following, answer the question, and show the output from the analyses you used to answer the question.

 

a.  What was mean of the differences in P₂O₅ before and after the training?

b.  Is this an increase or a decrease?

 

c.  Is the data distribution for the paired differences reasonably normal?

d.  Was the mean P₂O₅ use significantly different before and after the training?

f.  What do you conclude practically?  As appropriate, report the means before and after, the mean difference, the effect size and interpretation, whether the difference is large relative to the values, anything notable on plots, and your practical conclusions.

©2016 by Salvatore S. Mangiafico.
Rutgers Cooperative Extension, New Brunswick, NJ.
Updated: 2026.

Non-commercial reproduction of this content, with attribution, is permitted.
For-profit reproduction without permission is prohibited.

If you use the code or information in this site in a published work, please cite it as a source.  Also, if you are an instructor and use this book in your course, please let me know.   My contact information is on the About the Author of this Book page.

Citation

Mangiafico, S.S. 2016. Summary and Analysis of Extension Program Evaluation in R, version 1.24.5, revised 2026. rcompanion.org/handbook/. (Pdf version: rcompanion.org/documents/RHandbookProgramEvaluation.pdf.)