Summary and Analysis of Extension Program Evaluation in R

Salvatore S. Mangiafico

Introduction to Likert Data

Likert data—properly pronounced like “LICK-ert”—are ordered responses to questions or ratings.  Responses could be descriptive words, such as “agree”, “neutral”, or “disagree,” or numerical, such as “On a scale of 1 to 5, where 1 is ‘not interested’ and 5 is ‘very interested’…”  Likert data is commonly collected from surveys evaluating education programs, as well in a variety of opinion surveys and social science surveys.


Likert data


Numbers of responses

Most commonly, a 5- or 7- point scale is used for Likert items.  It is believed that most people can think about or visualize 5 or 7 ordered options easily.  Younger children, however, may do better with a 3-point scale or a simple dichotomous question.  On the other hand, if the audience is educated about a subject and trained in the evaluation, a 10-point scale could be used. 



Responses to Likert items are usually symmetrical.  That is, if there are options for “agree” and “strongly agree”, there should be options for “disagree” and “strongly disagree”.


Neutral responses

Responses to Likert items also tend to have a neutral option, such as “neutral”,  “neither agree nor disagree”.  Neutral responses may also be terms like “sometimes” or “occasionally” if “never” and “rarely” on one side are balanced with “often” and “always”.


Form of responses

Numbered responses are typically described with descriptive terms, either for every number, for just the end points, or for the end points and the middle points, for example:

Strongly     Agree      Neutral     Disagree    Strongly
agree                                           disagree

   1           2           3           4           5


Strongly                                       Strongly
agree                                          disagree

   1           2           3           4           5


Strongly                Neutral                Strongly
agree                                          disagree

   1           2           3           4           5

Other options for Likert responses include faces (smiley face, neutral face, frowny face), and a line on which respondents mark their response.


Opt-out answers

Questions may also include opt-out responses, like “Don’t know” or “Not applicable”.  These are included outside the Likert responses.

Strongly                Neutral                Strongly
agree                                          disagree

   1           2           3           4           5           Not applicable

I am in favor of including opt-out responses as it tends to encourage more honest responses.  It seems to me it is better to allow a respondent to opt out of answering a question rather than force an inauthentic response.  It is possible that a respondent has no opinion or doesn’t understand a question, or that a question is not applicable for them.


That being said, the opt-out answer “Don’t know” may not be a great choice, simply because respondents and researchers may interpret “Don’t know” as a “Neutral” answer.  It may be better to choose less ambiguous opt-out answers like “Not applicable”.


Examples of Likert item responses


Vagias, W.M. (2006). Likert-type scale response anchors. Clemson International Institute for Tourism & Research Development, Department of Parks, Recreation and Tourism Management.  Clemson University. www.clemson.edu/centers-institutes/tourism/documents/sample-scales.pdf.


Brown, S. Likert Scale Examples for Surveys. Iowa State University Extension. www.extension.iastate.edu/ag/staff/info/likertscaleexamples.pdf.


Likert items and Likert scales


A Likert item is a single question with Likert responses, whereas a Likert scale is a group of items viewed together as a single measure.  For example, one could have several Likert items with various questions about religious attitudes or behaviors, and then combine those items to a single Likert scale on religiosity.


When presenting methods and results, it is important to be clear if data were handled as Likert item data or Likert scale data.

This book will treat Likert data as individual Likert items, and will not create Likert scales.


Analysis of Likert data


Likert data should be treated as ordinal data

There is some agreement that Likert item data should generally be treated as ordinal and not treated as interval/ratio data. 


One consideration is that values in interval/ratio data need to be equally spaced.  That is, 2 is equally between 1 and 3, and you could average 1 and 3 and the response would be 2.  But it is not clear that “agree” is equally spaced between “strongly agree” and “neutral”.  Nor is it clear that “strongly agree” and “neutral” could be averaged for a result of “agree”.  Simply numbering the response levels does not make the responses interval/ratio data.


This book will treat Likert data as ordinal data.  It will avoid using parametric tests, such as t-test and ANOVA, with Likert data.  Likert data typically do not meet the assumptions of those parametric tests

Instead, we will use nonparametric tests, permutation tests, and ordinal regression.


A couple other properties suggest that Likert data should not be treated as interval/ratio data.  Likert data are not continuous; that is, there typically aren’t any decimal points in Likert responses.  Also, the responses in Likert data are constrained at their ends; that is, on a five-point scale, the responses cannot be below 1 or above 5.


Where it is useful, this book will treat Likert data as nominal data for certain types of summaries.  In general it is better to not treat ordinal data as nominal data in statistical analyses.  One reason is that when treating the data as nominal data, the information about the ordered nature of the response categories is lost.  However, sometimes it is useful to collapse Likert responses into categories;  for example, grouping “strongly agree” and “agree” together as one category and reporting its frequency as a percentage of responses.


Some people treat Likert data as interval/ratio data

Not everyone agrees that Likert item data should not be treated as interval/ratio data.  A quick search of the internet will produce plenty of examples of people defending treating Likert data as interval/ratio data.


Cases in which treating Likert responses as interval/ratio data may be reasonable include:

•  When there are a high number of response options per question (say 10)

•  When only the endpoints of the responses are indicated with text descriptors

•  When response options are assumed to be equally spaced

•  When respondents mark their answer on a line so that the precise location of the mark can be measured


Analysis of Likert scale data

When several Likert items are combined into a scale, so that there are many possible numeric outcomes, the results are often treated as interval/ratio data.

This is not entirely permissible from a theoretical point of view since Likert scales are made up of Likert items, and so have the same properties. But it is often a reasonable approach if the data meet the assumptions of the analysis. This is particularly the case if the scale data take on many values.

Analysis of Likert data


Ordinal regression

Probably the best tool for the analysis of experiments with Likert item data as the dependent variable is ordinal regression.  The ordinal package in R provides a powerful and flexible framework for ordinal regression.  It can handle a wide variety of experimental designs, including those with paired or repeated observations.  Ordinal regression is relatively easy to perform in R, but might be somewhat challenging for the novice in statistical analyses.  Occasionally there are problems with fitting models or checking model assumptions.  These cases may be frustrating for the novice user.


Permutation tests

Another tool appropriate for the analysis of Likert item data are permutation tests.  The coin package in R provides a relatively powerful and flexible framework for permutation tests with ordinal dependent variables.  It can handle models analogous to a one-way analysis of variance with blocks, including paired or repeated observations in a complete block design.  This covers more than all the designs that can be handled with the common traditional nonparametric tests.


Traditional nonparametric tests

Traditional nonparametric tests are generally considered appropriate for analyses with ordinal dependent variables.  They have the advantages of being widely used and likely to be familiar for readers.  One disadvantage of these tests is that the variety of designs they can handle is limited.  The Kruskal–Wallis test can analyze a model analogous to a one-way analysis of variance.  The Friedman and Quade tests can analyze data in an unreplicated complete block design with paired or repeated observations.


As a technical note, some authors have questioned using these tests with Likert item data.  One consideration is that the underlying statistics for some tests are based on the dependent variable being continuous in nature.  Another consideration is that, while these tests have provisions to handle tied values, that they may not behave well when there are many ties, as is likely for Likert data. 


Optional reading


“Oh Ordinal data, what do we do with you?” from Dr. Nic. 2013. Learn and Teach Statistics & Operations Research. learnandteachstatistics.wordpress.com/2013/07/08/ordinal/.


“Can Likert Scale Data ever be Continuous?” from Grace-Martin, K. 2008. The Analysis Factor. www.theanalysisfactor.com/can-likert-scale-data-ever-be-continuous/.