## Summary and Analysis of Extension Program Evaluation in R

Salvatore S. Mangiafico

The Quade test is used for similar data and hypotheses as the Friedman test, namely for unreplicated complete block designs.

While the Friedman test is a generalization of the paired sign test, the Quade test is a generalization of the two-sample signed-rank test.

Some authors indicate that the Quade test is appropriate for ordinal data, while others suggest that it is appropriate only for interval or ratio data.  It is included in this section as an alternative to the Friedman test.

##### Post-hoc tests

The outcome of the Quade test tells you if there are differences among the groups, but doesn’t tell you which groups are different from other groups.  In order to determine which groups are different from others, post-hoc testing can be conducted with the pairwise.wilcox.test function.

##### Appropriate data

•  Two-way data arranged in an unreplicated complete block design

•  Dependent variable is ordinal, interval, or ratio, although some authors say data must be interval or ratio only.

•  Treatment or group independent variable is a factor with two or more levels.  That is, two or more groups

•  Blocking variable is a factor with two or more levels

•  Blocks are independent of each other and have no interaction with treatments

•  In order to be a test of medians, the distributions of values for each group should have similar shape and spread.  Otherwise the test is a test of distributions.

##### Hypotheses

If the distribution of the differences in scores between each pair of groups are all symmetrical, or the distributions of values for each group have similar shape and spread:

•  Null hypothesis:  The medians of values for each group are equal.

•  Alternative hypothesis (two-sided): The medians of values for each group are not equal.

If the above conditions are not met:

•  Null hypothesis:  The distributions of values for each group are equal.

•  Alternative hypothesis (two-sided): There is systematic difference in the distribution of values for the groups.

##### Interpretation

If the distribution of the differences in scores between each pair of groups are all symmetrical, or the distributions of values for each group have similar shape and spread:

Significant results can be reported as “There was a significant difference in median values across groups.”

Post-hoc analysis allows you to say “The median for group A was higher than the median for group B”, and so on.

If the above conditions are not met:

Significant results can be reported as “There was a significant difference in values among groups.”

##### Other notes and alternative tests

The Friedman test is used for the same kinds of data and hypotheses, The Friedman test is described in the previous chapter.

Another alternative is to use cumulative link models for ordinal data, which are described later in this book.

### Packages used in this chapter

The packages used in this chapter include:

•  psych

•  FSA

•  lattice

•  multcompView

•  PMCMR

•  rcompanion

The following commands will install these packages if they are not already installed:

if(!require(psych)){install.packages("psych")}
if(!require(FSA)){install.packages("FSA")}
if(!require(lattice)){install.packages("lattice")}
if(!require(multcompView)){install.packages("multcompView")}
if(!require(PMCMR)){install.packages("PMCMR")}
if(!require(rcompanion)){install.packages("rcompanion")}

### Quade test example

Input =("
Instructor        Rater  Likert
'Bob Belcher'        a      4
'Bob Belcher'        b      5
'Bob Belcher'        c      4
'Bob Belcher'        d      6
'Bob Belcher'        e      6
'Bob Belcher'        f      6
'Bob Belcher'        g     10
'Bob Belcher'        h      6
'Linda Belcher'      a      8
'Linda Belcher'      b      6
'Linda Belcher'      c      8
'Linda Belcher'      d      8
'Linda Belcher'      e      8
'Linda Belcher'      f      7
'Linda Belcher'      g     10
'Linda Belcher'      h      9
'Tina Belcher'       a      7
'Tina Belcher'       b      5
'Tina Belcher'       c      7
'Tina Belcher'       d      8
'Tina Belcher'       e      8
'Tina Belcher'       f      9
'Tina Belcher'       g     10
'Tina Belcher'       h      9
'Gene Belcher'       a      6
'Gene Belcher'       b      4
'Gene Belcher'       c      5
'Gene Belcher'       d      5
'Gene Belcher'       e      6
'Gene Belcher'       f      6
'Gene Belcher'       g      5
'Gene Belcher'       h      5
'Louise Belcher'     a      8
'Louise Belcher'     b      7
'Louise Belcher'     c      8
'Louise Belcher'     d      8
'Louise Belcher'     e      9
'Louise Belcher'     f      9
'Louise Belcher'     g      8
'Louise Belcher'     h     10
")

### Order levels of the factor; otherwise R will alphabetize them

Data\$Instructor = factor(Data\$Instructor,
levels=unique(Data\$Instructor))

### Create a new variable which is the likert scores as an ordered factor

Data\$Likert.f = factor(Data\$Likert,
ordered=TRUE)

###  Check the data frame

library(psych)

str(Data)

summary(Data)

### Remove unnecessary objects

rm(Input)

#### Summarize data treating Likert scores as factors

xtabs( ~ Instructor + Likert.f,
data = Data)

Likert.f
Instructor       4 5 6 7 8 9 10
Bob Belcher    2 1 4 0 0 0  1
Linda Belcher  0 0 1 1 4 1  1
Tina Belcher   0 1 0 2 2 2  1
Gene Belcher   1 4 3 0 0 0  0
Louise Belcher 0 0 0 1 4 2  1

XT = xtabs( ~ Instructor + Likert.f,
data = Data)

prop.table(XT,
margin = 1)

Likert.f
Instructor           4     5     6     7     8     9    10
Bob Belcher    0.250 0.125 0.500 0.000 0.000 0.000 0.125
Linda Belcher  0.000 0.000 0.125 0.125 0.500 0.125 0.125
Tina Belcher   0.000 0.125 0.000 0.250 0.250 0.250 0.125
Gene Belcher   0.125 0.500 0.375 0.000 0.000 0.000 0.000
Louise Belcher 0.000 0.000 0.000 0.125 0.500 0.250 0.125

#### Bar plots by group

Note that the bar plots don’t show the effect of the blocking variable.

library(lattice)

histogram(~ Likert.f | Instructor,
data=Data,
layout=c(1,5)      #  columns and rows of individual plots
)

#### Summarize data treating Likert scores as numeric

library(FSA)

Summarize(Likert ~ Instructor,
data=Data,
digits=3)

Instructor n  mean    sd min   Q1 median   Q3 max percZero
1    Bob Belcher 8 5.875 1.885   4 4.75      6 6.00  10        0
2  Linda Belcher 8 8.000 1.195   6 7.75      8 8.25  10        0
3   Tina Belcher 8 7.875 1.553   5 7.00      8 9.00  10        0
4   Gene Belcher 8 5.250 0.707   4 5.00      5 6.00   6        0
5 Louise Belcher 8 8.375 0.916   7 8.00      8 9.00  10        0

#### Quade test example

This example uses the formula notation indicating that Likert is the dependent variable, Instructor is the independent variable, and Rater is the blocking variable.  The data= option indicates the data frame that contains the variables.  For the meaning of other options, see ?quade.test.

quade.test(Likert ~ Instructor | Rater,
data = Data)

Quade F = 8.0253, num df = 4, denom df = 28, p-value = 0.0001924

#### Post-hoc test: pairwise two-sample paired rank-sum test for multiple comparisons

To prevent the inflation of type I error rates, adjustments to the p-values can be made using the p.adjust.method option.  See ?p.adjust for details on available p-value adjustment methods.

Note that the data must be ordered by the blocking variable so that the first observation for Bob will be paired with the first observation for Linda, and so on.

### Order groups by median

Data\$Instructor = factor(Data\$Instructor,
levels=c("Linda Belcher", "Louise Belcher",

"Tina Belcher", "Bob Belcher",

"Gene Belcher"))

Data

### Pairwise Mann–Whitney

PT = pairwise.wilcox.test(Data\$Likert,
Data\$Instructor,
paired=TRUE)\$p.value
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

### Convert PT to a full table and call it PT1

library(rcompanion)

PT1 = fullPTable(PT)

PT1

Linda Belcher Louise Belcher Tina Belcher Bob Belcher Gene Belcher
Linda Belcher     1.00000000     0.61269120   0.85010674  0.05090808   0.04637053
Louise Belcher    0.61269120     1.00000000   0.47198578  0.05090808   0.04637053
Tina Belcher      0.85010674     0.47198578   1.00000000  0.05090808   0.04637053
Bob Belcher       0.05090808     0.05090808   0.05090808  1.00000000   0.65530421
Gene Belcher      0.04637053     0.04637053   0.04637053  0.65530421   1.00000000

# Produce compact letter display

library(multcompView)

multcompLetters(PT1,
compare="<",
threshold=0.05,  # p-value to use as significance threshold
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher   Tina Belcher    Bob Belcher   Gene Belcher
"a"            "a"            "a"           "ab"            "b"

### Values sharing a letter are not significantly different

#### Post-hoc test: Quade post-hoc test

A different post-hoc test for the Quade test can be conducted with the posthoc.quade.test function in the PMCMR package.  The output can be converted to a compact letter display using the multcompLetters function in the multcompView package.

### Order groups by median

Data\$Instructor = factor(Data\$Instructor,
levels=c("Linda Belcher", "Louise Belcher",

"Tina Belcher", "Bob Belcher",

"Gene Belcher"))

Data

### Post-hoc test

library(PMCMR)

PT = posthoc.quade.test(y      = Data\$Likert,
groups = Data\$Instructor,
blocks = Data\$Rater,
# Adjusts p-values for multiple comparisons;
# See ?p.adjust for options

PT

Pairwise comparisons using posthoc-Quade test with TDist approximation

Linda Belcher Louise Belcher Tina Belcher Bob Belcher
Louise Belcher 0.00902       -              -            -
Tina Belcher   0.04533       0.50547        -            -
Bob Belcher    0.36572       0.00108        0.00539      -
Gene Belcher   0.00544       0.77620        0.38132      0.00099

P value adjustment method: fdr

### Compact letter display

PT0 = as.matrix(PT\$p.value)

library(rcompanion)

PT1 = fullPTable(PT0)

library(multcompView)
multcompLetters(PT1,
compare="<",
threshold=0.05,
Letters=letters,
reversed = FALSE)

Linda Belcher Louise Belcher   Tina Belcher    Bob Belcher   Gene Belcher
"a"            "b"            "b"            "a"            "b"