A workflow for automating scale scoring in R

Scoring scales is easily one of the most boring and repetitive tasks in the social sciences. In this post, we are going to learn how we can largely automate the process using R. The major advantages are both convenience, time-saving, and quick retrieval of the array of useful statistics for the items and scales in a dataset.

The R package psych features powerful functions for scoring survey scales. In this tutorial, we will use a wrapper for the scoreItems function to automate scale scoring.

Data management: Naming variables in a consistent manner

A crucial inital step is to name the variables of a dataset consistently. This will later come in handy when we automate the scale construction, because we will construct an algorithm that looks for a certain pattern in the name of variables to identify them as items. Our goal is to create variable names that make clear that a specific item belongs to a specific scale.

  • bad example: “happy”, “serene”, “calm”
  • good example: “posaffect_1”“,”posaffect_2“,”posaffect_3"

I recommend using the following pattern: “scalename_itemnumber”. For example

  • agr_1 for the first item of an agreeableness scale, or
  • neu_1 for the first item of a neuroticism scale.

Non-scale variables should not follow the same pattern in order to make it easy for the algorithm to distinguish items we want to score from variables we dont want to score. So, its best not to use the “characters_number” pattern for any non-scale variable. For example, - gender_1 should be renamed to gender

Example dataset

This is the structure of our dataset:

str(df)
## Classes 'tbl_df', 'tbl' and 'data.frame':    2800 obs. of  28 variables:
##  $ agr_1    : int  2 2 5 4 2 6 2 4 4 2 ...
##  $ agr_2    : int  4 4 4 4 3 6 5 3 3 5 ...
##  $ agr_3    : int  3 5 5 6 3 5 5 1 6 6 ...
##  $ agr_4    : int  4 2 4 5 4 6 3 5 3 6 ...
##  $ agr_5    : int  4 5 4 5 5 5 5 1 3 5 ...
##  $ con_1    : int  2 5 4 4 4 6 5 3 6 6 ...
##  $ con_2    : int  3 4 5 4 4 6 4 2 6 5 ...
##  $ con_3    : int  3 4 4 3 5 6 4 4 3 6 ...
##  $ con_4    : int  4 3 2 5 3 1 2 2 4 2 ...
##  $ con_5    : int  4 4 5 5 2 3 3 4 5 1 ...
##  $ ext_1    : int  3 1 2 5 2 2 4 3 5 2 ...
##  $ ext_2    : int  3 1 4 3 2 1 3 6 3 2 ...
##  $ ext_3    : int  3 6 4 4 5 6 4 4 NA 4 ...
##  $ ext_4    : int  4 4 4 4 4 5 5 2 4 5 ...
##  $ ext_5    : int  4 3 5 4 5 6 5 1 3 5 ...
##  $ neu_1    : int  3 3 4 2 2 3 1 6 5 5 ...
##  $ neu_2    : int  4 3 5 5 3 5 2 3 5 5 ...
##  $ neu_3    : int  2 3 4 2 4 2 2 2 2 5 ...
##  $ neu_4    : int  2 5 2 4 4 2 1 6 3 2 ...
##  $ neu_5    : int  3 5 3 1 3 3 1 4 3 4 ...
##  $ ope_1    : int  3 4 4 3 3 4 5 3 6 5 ...
##  $ ope_2    : int  6 2 2 3 3 3 2 2 6 1 ...
##  $ ope_3    : int  3 4 5 4 4 5 5 4 6 5 ...
##  $ ope_4    : int  4 3 5 3 3 6 6 5 6 5 ...
##  $ ope_5    : int  3 3 2 5 3 1 1 3 1 2 ...
##  $ gender   : int  1 2 2 2 1 2 1 1 1 2 ...
##  $ education: int  NA NA NA NA NA 3 NA 2 1 NA ...
##  $ age      : int  16 18 17 17 17 21 18 19 19 17 ...

We use a redacted bfi dataset from psych, which consists of 25 personality self-report items taken from the International Personality Item Pool (ipip.ori.org). We have measured each of the Big Five traits with five items.

Now, instead of calculating the scores of each of these five scales manually, we are going to automate the process1.

Extracting all items

As you may have seen, the dataset also contains three demographic variables in addition to the scale items we want to use. Thus, we first have to select and extract all variables that are scale items from our dataset. This is where we need the unique naming structure of item-type variables, that is, variables which follow the structure scalename_itemnumber.

bigfiveitems <- dplyr::select(df, matches("_\\d$|_\\d.$"))
names(bigfiveitems)
##  [1] "agr_1" "agr_2" "agr_3" "agr_4" "agr_5" "con_1" "con_2" "con_3"
##  [9] "con_4" "con_5" "ext_1" "ext_2" "ext_3" "ext_4" "ext_5" "neu_1"
## [17] "neu_2" "neu_3" "neu_4" "neu_5" "ope_1" "ope_2" "ope_3" "ope_4"
## [25] "ope_5"

What we make use of here is the select function from the dplyr package. It allows to select variables based upon regular expressions 2. Regular expressions allow us to describe patterns in strings, such as variable names here. The regular expression we use here tells the function to look for variable names that have an underdash ("_") followed by a single digit number (e.g., 1). Since all items follow this naming pattern, and, importantly, no non-item does, the functions quickly selects all the item variables.

Next, we automatically look for the scale names:

scalelist <- str_extract(names(bigfiveitems), "\\w+(?=_)")
scalelist <- unique(scalelist)
scalelist
## [1] "agr" "con" "ext" "neu" "ope"

As we can see, this next bit of code extracts the names of the scales, again using a regular expression. We could also supply the scalelist manually, e.g. scalelist <- c("agr", "con", "ext", "neu", "ope")

We now use the scalelist to tell scoreItems from the psych package to create scale scores for all of the scales listed in scalelist. This is handled via a function called scoreItemsMulti you can source from my github3.

source("https://github.com/franciscowilhelm/r-collection/raw/master/scoreItemsMulti.R")
bigfivescores <- scoreItemsMulti(scalelist, bigfiveitems)

The beauty of this function is that it takes the names of the scales and creates not only the scale scores, but also several useful statistics such as internal reliabilities, correlation matrices and others.

The scores themselves are available in the scores sublist of the object.

head(bigfivescores$scores)
##      agr con ext neu ope
## [1,] 4.0 2.8 3.8 2.8 3.0
## [2,] 4.2 4.0 5.0 3.8 4.0
## [3,] 3.8 4.0 4.2 3.6 4.8
## [4,] 4.6 3.0 3.6 2.8 3.2
## [5,] 4.0 4.4 4.8 3.2 3.6
## [6,] 4.6 5.6 5.6 3.0 5.0

Check out the documentation of the psych::scoreItems function for more information on what the object contains. For example, summary gives you a correlation table:

summary(bigfivescores)
## Call: psych::scoreItems(keys = keys_negative, items = dataframe_exclude, 
##     impute = "none")
## 
## Scale intercorrelations corrected for attenuation 
##  raw correlations below the diagonal, (standardized) alpha on the diagonal 
##  corrected correlations above the diagonal:
##       agr   con   ext    neu   ope
## agr  0.70  0.36  0.63 -0.244  0.23
## con  0.26  0.73  0.35 -0.306  0.30
## ext  0.46  0.26  0.76 -0.285  0.32
## neu -0.18 -0.24 -0.22  0.814 -0.12
## ope  0.15  0.20  0.22 -0.086  0.60

A word of caution: If your scales contain reverse-worded items, it will try to automatically detect these4. However, in such cases you should proceed with caution and consider checking manually whether it got it right. You can do this by inspect the $keys.list element of your object.

Summary: The workflow

This workflow vastly speeds up the process of scale scoring and documentation of psychometrical attributes such as Cronbach’s alpha.

Here is the recommended workflow in short

  1. Name variables such that all items follow the pattern ‘scalename_itemnumber’.
  2. Optional: Make sure that no other variables follow the pattern of ‘characters_number’.
  3. Create a character vector holding all scale names either automatically or manually.
  4. Use scoreItemsMulti to create the scale scores and other attributes.
  5. If reverse-worded items are part of your scales, check the $keys.list element of your object whether the function identified these correctly.

  1. with just five scales, this may seem like of over-engineering. Usually however, you will have a much larger dataset, where such an approach saves a lot of time.↩︎

  2. More info on regular expressions in R, especially the tidyverse packages: https://stringr.tidyverse.org/articles/regular-expressions.html↩︎

  3. You can also find more information for the “under-the-hood” workings on my github if you are interested.↩︎

  4. The automatic detection of reverse-worded items is accomplished by running a PCA and reverse scoring those items that load negatively on the factor. This may not always work correctly.↩︎

Francisco Wilhelm
Francisco Wilhelm
Doctoral Researcher in I/O Psychology

I study the psychological processes underlying career development