Use dplyr's group_by to perform split-apply-combine

Question

I am trying to use dplyr to do the following:

 tapply(iris$Petal.Length, iris$Species, shapiro.test)

I want to split the Petal.Lengths by Speicies, and apply a function, in this case shapiro.test. I read this SO question and quite a number of other pages. I am sort of able to split the variable into groups, using do:

iris %>%
  group_by(Species) %>%
  select(Petal.Length) %>%
  do(print(.$Petal.Length)) 

 [1] 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 1.5 1.6 1.4 1.1 1.2
[16] 1.5 1.3 1.4 1.7 1.5 1.7 1.5 1.0 1.7 1.9 1.6 1.6 1.5 1.4 1.6
[31] 1.6 1.5 1.5 1.4 1.5 1.2 1.3 1.4 1.3 1.5 1.3 1.3 1.3 1.6 1.9
[46] 1.4 1.6 1.4 1.5 1.4
 [1] 4.7 4.5 4.9 4.0 4.6 4.5 4.7 3.3 4.6 3.9 3.5 4.2 4.0 4.7 3.6
[16] 4.4 4.5 4.1 4.5 3.9 4.8 4.0 4.9 4.7 4.3 4.4 4.8 5.0 4.5 3.5
[31] 3.8 3.7 3.9 5.1 4.5 4.5 4.7 4.4 4.1 4.0 4.4 4.6 4.0 3.3 4.2
[46] 4.2 4.2 4.3 3.0 4.1

The 'splitting' of the column into groups seems to be working. But the way to pass the pieces to shapiro.test is still eluding me. I see that group_by is different from split into.

I tried lots of variations, including:

iris %>%
  group_by(Species) %>%
  select(Petal.Length) %>%
  summarise(shapiro.test)

and also

iris %>%
  group_by(Species) %>%
  select(Petal.Length) %>%
  summarise_each(funs(shapiro.test))

 # Error: expecting a single value

How can I make dplyr run shapiro.test() thrice, once for the Petal.Lengths of each Species?

Answer 1

I could see two ways to do it, depending on how you want to use the output. You could pull out just the p-values from shapiro.test in summarise. Alternatively you could use do and save the results of each test in a list.

library(dplyr)

With summarise, pulling out just the p-values:

iris %>%
    group_by(Species) %>%
    summarise(stest = shapiro.test(Petal.Length)$p.value)

     Species      stest
1     setosa 0.05481147
2 versicolor 0.15847784
3  virginica 0.10977537

Using do:

tests = iris %>%
    group_by(Species) %>%
    do(test = shapiro.test(.$Petal.Length))

# Resulting list
tests$test

[[1]]

    Shapiro-Wilk normality test

data:  .$Petal.Length
W = 0.955, p-value = 0.05481


[[2]]

    Shapiro-Wilk normality test

data:  .$Petal.Length
W = 0.966, p-value = 0.1585


[[3]]

    Shapiro-Wilk normality test

data:  .$Petal.Length
W = 0.9622, p-value = 0.1098

Answer 2

If you use tidy() function from the broom package, to turn the output of shapiro.test() into a data.frame then you can use do().

iris %>%
  group_by(Species) %>%
  do(tidy(shapiro.test(.$Petal.Length)))

This gives you:

Source: local data frame [3 x 4]
Groups: Species [3]

Species statistic    p.value                      method
      <fctr>     <dbl>      <dbl>                      <fctr>
1     setosa 0.9549768 0.05481147 Shapiro-Wilk normality test
2 versicolor 0.9660044 0.15847784 Shapiro-Wilk normality test
3  virginica 0.9621864 0.10977537 Shapiro-Wilk normality test

This is adapted from my answere here.