Summary and Analysis of Extension Program Evaluation in R

Salvatore S. Mangiafico

One-way Permutation Test for Paired Ordinal Data

When to use this test

 

A permutation test of symmetry can be used for one-way data with an ordinal dependent variable where observations are paired within a blocking variable.  It will determine if there is a difference in the response variable among groups when controlling for the effect of the blocking variable.  There can be two or more groups. 

 

The coin package can accommodate designs used with Friedman, Quade, paired t-test, repeated measures one-way anova, and their ordinal regression equivalents. 

 

The test does not make assumptions about the distribution of values.

 

The test is performed with the symmetry_test function in the coin package.

 

Post-hoc testing can be conducted with pairwise permutation tests across groups.

 

Appropriate data

•  One-way data plus a blocking variable.  That is, one measurement variable in two or more groups, where observations are paired within levels of a blocking variable

•  Here, the dependent variable is ordinal

•  The independent variable is a factor with two or more levels.  That is, two or more groups.  The blocking variable is also a factor variable

•  Here, the data is arranged in an unreplicated complete block design, but there could be one or more observations per cell

 

Hypotheses

•  Null hypothesis:  The response of the dependent variable among groups are equal n the sampled population.

•  Alternative hypothesis (two-sided): The response of the dependent variable among groups are not equal in the sampled population.

 

Interpretation

•  Reporting significant results for the omnibus test as “Significant differences were found in the response among groups.” is acceptable.  Alternatively, “A significant effect for Independent Variable on Dependent Variable was found when controlling for the effect of Blocking Variable.”

•  Reporting significant results for mean separation post-hoc tests as “Response of Dependent Variable for group A was different than that for group B.” is acceptable.

 

Other notes and alternative tests

Ordinal regression is an alternative.

 

The traditional nonparametric tests Friedman, Quade, Paired rank-sum test, or Sign test may be alternatives depending on the design of the experiment.

 

Packages used in this chapter

 

The packages used in this chapter include:

•  psych

•  lattice

•  FSA

•  coin

•  rcompanion

•  multcompView

 

The following commands will install these packages if they are not already installed:

if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(coin)){install.packages("coin")}
if(!require(rcompanion)){install.packages("rcompanion")}
if(!require(multcompView)){install.packages("multcompView")}

One-way ordinal permutation test of symmetry example

 

This example re-visits the Belcher data from the Friedman Test chapter.  Note that each instructor is rated by each of eight raters.  Because of this, we want to stratify the responses by Rater.

 

Here, the ytrafo=rank_tranfo argument is passed to symmetry_test to indicate that the dependent variable should be rank transformed.

Data = read.table(header=TRUE, stringsAsFactors=TRUE, text="

 Instructor        Rater  Likert
 'Bob Belcher'        a      4
 'Bob Belcher'        b      5
 'Bob Belcher'        c      4
 'Bob Belcher'        d      6
 'Bob Belcher'        e      6
 'Bob Belcher'        f      6
 'Bob Belcher'        g     10
 'Bob Belcher'        h      6
 'Linda Belcher'      a      8
 'Linda Belcher'      b      6
 'Linda Belcher'      c      8
 'Linda Belcher'      d      8
 'Linda Belcher'      e      8
 'Linda Belcher'      f      7
 'Linda Belcher'      g     10
 'Linda Belcher'      h      9
 'Tina Belcher'       a      7
 'Tina Belcher'       b      5
 'Tina Belcher'       c      7
 'Tina Belcher'       d      8
 'Tina Belcher'       e      8
 'Tina Belcher'       f      9
 'Tina Belcher'       g     10
 'Tina Belcher'       h      9
 'Gene Belcher'       a      6
 'Gene Belcher'       b      4
 'Gene Belcher'       c      5
 'Gene Belcher'       d      5
 'Gene Belcher'       e      6
 'Gene Belcher'       f      6
 'Gene Belcher'       g      5
 'Gene Belcher'       h      5
 'Louise Belcher'     a      8
 'Louise Belcher'     b      7
 'Louise Belcher'     c      8
 'Louise Belcher'     d      8
 'Louise Belcher'     e      9
 'Louise Belcher'     f      9
 'Louise Belcher'     g      8
 'Louise Belcher'     h     10
")


### Order levels of the factor; otherwise R will alphabetize them

Data$Instructor = factor(Data$Instructor,
                      levels=unique(Data$Instructor))


### Create a new variable which is the likert scores as an ordered factor

Data$Likert.f = factor(Data$Likert,
                          ordered=TRUE)


###  Check the data frame

library(psych)

headTail(Data)

str(Data)

summary(Data)

Summarize data treating Likert scores as factors

Note that the variable we want to count is Likert.f, which is a factor variable.  Counts for Likert.f are cross tabulated over values of Instructor.  The prop.table function translates a table into proportions.  The margin=1 option indicates that the proportions are calculated for each row.

 

xtabs( ~ Instructor + Likert.f,
      data = Data)

                Likert.f
Instructor       4 5 6 7 8 9 10
  Bob Belcher    2 1 4 0 0 0  1
  Linda Belcher  0 0 1 1 4 1  1
  Tina Belcher   0 1 0 2 2 2  1
  Gene Belcher   1 4 3 0 0 0  0
  Louise Belcher 0 0 0 1 4 2  1

XT = xtabs( ~ Instructor + Likert.f,
           data = Data)

prop.table(XT,
           margin = 1)

                Likert.f
Instructor           4     5     6     7     8     9    10
  Bob Belcher    0.250 0.125 0.500 0.000 0.000 0.000 0.125
  Linda Belcher  0.000 0.000 0.125 0.125 0.500 0.125 0.125
  Tina Belcher   0.000 0.125 0.000 0.250 0.250 0.250 0.125
  Gene Belcher   0.125 0.500 0.375 0.000 0.000 0.000 0.000
  Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

Bar plots by group

Note that the variable we want to count is Likert.f, which is a factor variable.  Counts for Likert.f are presented for values of Speaker.  Also note that the bar plots don’t show the effect of the blocking variable.

library(lattice)

histogram(~ Likert.f | Instructor,
          data=Data,
          layout=c(1,5)      #  columns and rows of individual plots
          )

Summarize data treating Likert scores as numeric

It may be useful to look at the minimum, first quartile, median, third quartile, and maximum for Likert for each group.

library(FSA)

Summarize(Likert ~ Instructor,
          data=Data,
          digits=3)

      Instructor n  mean    sd min   Q1 median   Q3 max percZero
1    Bob Belcher 8 5.875 1.885   4 4.75      6 6.00  10        0
2  Linda Belcher 8 8.000 1.195   6 7.75      8 8.25  10        0
3   Tina Belcher 8 7.875 1.553   5 7.00      8 9.00  10        0
4   Gene Belcher 8 5.250 0.707   4 5.00      5 6.00   6        0
5 Louise Belcher 8 8.375 0.916   7 8.00      8 9.00  10        0

Permutation symmetry test

Instructor is the independent variable, and Rater is the blocking variable.  The data= option indicates the data frame that contains the variables.  For the meaning of other options, see library(coin); ?symmetry_test.

library(coin)

symmetry_test(Likert ~ Instructor | Rater,
              data     = Data,
              ytrafo   = rank_trafo,
              teststat = "quadratic")

Asymptotic General Symmetry Test

chi-squared = 22.283, df = 4, p-value = 0.0001761

Note that there are built-in functions in the coin package to conduct an analysis analogous to the Friedman test and Quade test.  These give different results than the symmetry_test used above.

friedman_test(Likert ~ Instructor | Rater,
              data = Data)

Asymptotic Friedman Test

chi-squared = 23.139, df = 4, p-value = 0.0001188

quade_test(Likert ~ Instructor | Rater,
           data = Data)

Asymptotic Quade Test

chi-squared = 17.092, df = 4, p-value = 0.001855

See Hothorn et al. for options in the symmetry_test function that correspond to common tests.

 

Post-hoc test: pairwise permutation tests

If the symmetry test is significant, a post-hoc analysis can be performed to determine which groups differ from which other groups.

 

The pairwisePermutationSymmetry and pairwisePermutationSymmetryMatrix functions in the rcompanion package conduct permutation tests across groups in a pairwise manner.   See library(rcompanion); ?pairwisePermutationSymmetry for further details.

 

Because the post-hoc test will produce multiple p-values, adjustments to the p-values can be made to avoid inflating the possibility of making a type-I error.  Here, the method of adjustment is indicated with the method option.  There are a variety of methods for controlling the familywise error rate or for controlling the false discovery rate.  See ?p.adjust for details on these methods.

 

Before conducting the pairwise tests, we will re-order the levels of the grouping variable by the median of each group.  This makes interpretation of the pairwise comparisons and compact letter display easier.

 

Table output and compact letter display

### Order groups by median

Data$Instructor = factor(Data$Instructor,
                   levels = c("Linda Belcher", "Louise Belcher",
                              "Tina Belcher", "Bob Belcher",
                              "Gene Belcher"))


### Pairwise permutation tests

library(rcompanion)

PT = pairwisePermutationSymmetry(Likert ~ Instructor | Rater,
                                 data     = Data,
                                 ytrafo   = rank_trafo,
                                 teststat = "quadratic",
                                 method   = "fdr")

PT

                           Comparison     Stat  p.value p.adjust
1  Linda Belcher - Louise Belcher = 0   0.6761   0.4109  0.51360
2    Linda Belcher - Tina Belcher = 0 0.008163    0.928  0.92800
3     Linda Belcher - Bob Belcher = 0    5.912  0.01503  0.03210
4    Linda Belcher - Gene Belcher = 0    6.778  0.00923  0.03210
5   Louise Belcher - Tina Belcher = 0   0.8491   0.3568  0.50970
6    Louise Belcher - Bob Belcher = 0    5.163  0.02307  0.03845
7   Louise Belcher - Gene Belcher = 0    7.516 0.006117  0.03210
8      Tina Belcher - Bob Belcher = 0    5.797  0.01605  0.03210
9     Tina Belcher - Gene Belcher = 0    6.531   0.0106  0.03210
10     Bob Belcher - Gene Belcher = 0   0.4737   0.4913  0.54590

### Compact letter display

library(rcompanion)

cldList(p.adjust ~ Comparison,
        data = PT,
        threshold  = 0.05)

          Group Letter MonoLetter
1  LindaBelcher      a         a
2 LouiseBelcher      a         a
3   TinaBelcher      a         a
4    BobBelcher      b          b
5   GeneBelcher      b          b

   Groups sharing a letter are not significantly different (alpha = 0.05).

Matrix output and compact letter display

A compact letter display condenses a table of p-values into a simpler format.  In the output, groups sharing a same letter are not significantly different.  Compact letter displays are a clear and succinct way to present results of multiple comparisons.

 

Here the fdr p-value adjustment method is used.  See ?p.adjust for details on available methods.

 

The code creates a matrix of p-values called PM which is then passed to the multcompLetters function to be converted to a compact letter display.

### Order groups by median

Data$Instructor = factor(Data$Instructor,
                   levels = c("Linda Belcher", "Louise Belcher",
                              "Tina Belcher", "Bob Belcher",
                              "Gene Belcher"))


### Pairwise permutation tests

library(rcompanion)

PM = pairwisePermutationSymmetryMatrix(Likert ~ Instructor | Rater,
                                       data     = Data,
                                       ytrafo   = rank_trafo,

                                       teststat = "quadratic",

                                       method   = "fdr")

 

PM$Adjusted

               Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
Linda Belcher         1.0000        0.51360       0.9280     0.03210       0.0321
Louise Belcher        0.5136        1.00000       0.5097     0.03845       0.0321
Tina Belcher          0.9280        0.50970       1.0000     0.03210       0.0321
Bob Belcher           0.0321        0.03845       0.0321     1.00000       0.5459
Gene Belcher          0.0321        0.03210       0.0321     0.54590       1.0000

# Produce compact letter display

library(multcompView)

multcompLetters(PM$Adjusted,
                compare="<",
                threshold=0.05,
                Letters=letters,
                reversed = FALSE)

Linda Belcher Louise Belcher   Tina Belcher    Bob Belcher   Gene Belcher
          "a"            "a"            "a"            "b"            "b"

### Groups sharing a letter are not significantly different (alpha = 0.05).

©2016 by Salvatore S. Mangiafico.
Rutgers Cooperative Extension, New Brunswick, NJ.
Updated: 2025.

Non-commercial reproduction of this content, with attribution, is permitted.
For-profit reproduction without permission is prohibited.

If you use the code or information in this site in a published work, please cite it as a source.  Also, if you are an instructor and use this book in your course, please let me know.   My contact information is on the About the Author of this Book page.

Citation

Mangiafico, S.S. 2016. Summary and Analysis of Extension Program Evaluation in R, version 1.23.0, revised 2025. rcompanion.org/handbook/. (Pdf version: rcompanion.org/documents/RHandbookProgramEvaluation.pdf.)