Worked examples | Learning from Data: R programming

Exploratory Data Analysis for Modelling

Tue, 28 Jul 2020 00:00:00 +0000

You may remember the Happy Feet movie and here is an Adélie penguin singing My Way. We shall analyse data that were collected and made available by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER. The dataset contains data for 344 penguins on 3 different species of penguins (Adélie, Chinstrap, and Gentoo), collected from 3 islands in the Palmer Archipelago, Antarctica.

This great artwork was made by [@allison_horst](https://twitter.com/allison_horst)

and if you are a Happy Feet fan, these are the penguins we have data on

Penguin Bill dimensions

The culmen is the upper ridge of a bird’s bill. In the simplified penguins data, culmen length and depth are renamed as variables bill_length_mm and bill_depth_mm to be more intuitive.

For this penguin data, the culmen (bill) length and depth are measured as shown below:

Exploratory Data Analysis (EDA).

The variable of interest is penguin body mass. We want to see whether body mass is related to any of the other variables included in the dataframe. So let us start by looking at the data

library(palmerpenguins)
glimpse(penguins)

## Rows: 344
## Columns: 8
## $ species           <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, A...
## $ island            <fct> Torgersen, Torgersen, Torgersen, Torgersen, Torge...
## $ bill_length_mm    <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34....
## $ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18....
## $ flipper_length_mm <int> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, ...
## $ body_mass_g       <int> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 347...
## $ sex               <fct> male, female, female, NA, female, male, female, m...
## $ year              <int> 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2...

skimr::skim(penguins)

Table 1: Data summary
Name	penguins
Number of rows	344
Number of columns	8
_______________________
Column type frequency:
factor	3
numeric	5
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
species	0	1.00	FALSE	3	Ade: 152, Gen: 124, Chi: 68
island	0	1.00	FALSE	3	Bis: 168, Dre: 124, Tor: 52
sex	11	0.97	FALSE	2	mal: 168, fem: 165

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bill_length_mm	2	0.99	43.92	5.46	32.1	39.23	44.45	48.5	59.6	▃▇▇▆▁
bill_depth_mm	2	0.99	17.15	1.97	13.1	15.60	17.30	18.7	21.5	▅▅▇▇▂
flipper_length_mm	2	0.99	200.92	14.06	172.0	190.00	197.00	213.0	231.0	▂▇▃▅▂
body_mass_g	2	0.99	4201.75	801.95	2700.0	3550.00	4050.00	4750.0	6300.0	▃▇▆▃▂
year	0	1.00	2008.03	0.82	2007.0	2007.00	2008.00	2009.0	2009.0	▇▁▇▁▇

`GGally::ggpairs()` to get scatter plot + correlation matrix

GGally::ggpairs() is a useful tool for exploring distributions and correlations.

ggpairs1 <- penguins %>% 
  na.omit() %>% 
  select( body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm, species, sex, island) %>% 
  rename(`Flipper length(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs() +
  theme_minimal() 

ggpairs1

ggpairs() provides a lot of information, so let us spend some time deciphering this chart.

Along the diagonal we get a density plot of each variable of interest. For instance, body mass seems to be right skewed, and the rest of the variables seem bimodal.
The upper part of the matrix, shows correlation coefficients– to determine between which variables, read off the corresponding row and column header
The bottom part provides a scatterplot between any two variables which you can again determine by looking at the row/column they correspond.
For categorical variables (species, sex, island), we do not get any numerical values, but rather histograms and boxplots that show the distribution of outcomes. If we wanted to get a numerical correlation value, we would use the polycor package and polycor::hetcor() that calculates polyserial correlations between numeric and ordinal variables, and polychoric correlations between ordinal variables, but this is outside the scope of this class.

Which correlation is the highest? That between body mass and flipper length, or 0.873. We can see the scatterplot of these two variables in the upper left, just underneath the density plot of body mass, whereas to the right of the body mass density plot we can see the correlation value of 0.873. We could somehow “see” a line through these points. In building a model for body mass, surely flipper length will be the first variable to consider and the one that would explain a fair bit of the variation in body mass.

What about the lowest correlation? That seems to be the one between bill length and bill depth, with a correlation value of -0.229. On the face of it, there seems to be very weak relationship between these two variables.

What about the second lowest correlation? Numerically, it is the one between body mass and bill depth (-0.472). However, if you look at the corresponding scatterplot in the lower left corner you see two clusters of points, so we need to dig a bit deeper.

Dummy variables

Dummy, or categorical, variables allows us to incorporate non-numeric data into our analysis. In this example we have two categorical variables (species, sex) and we can use ggpairs() to dig a bit deeper and then use these categorical variables in our modelling. The first scatterplot matrix does not take into consideration the species or the sex of the penguins, variables that may help explain differences in body mass.

ggpairs2 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4) +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs2

When we include the categorical (factor) variables of species and sex, and since these are no longer numerical values, we get boxplots and histograms. For instance to explore the relationship between body mass and species, look at the upper left of the table; yo can see that the green penguins (Gentoo) seem to be heavier than the other two species which may have no difference between them.

Since this plot may be confusing, let us just concentrate on the scatterplot matrix of numerical values coloured by species.

ggpairs3 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4,
                  columns = c(3:6)) + 
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs3

Modelling considerations of numerical variables

Let us examine the relationship between body mass and flipper length, and body mass and bill depth.

Penguin body mass vs flipper length

mass_flipper <- ggplot(data = penguins,
                       aes(x = flipper_length_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.2, 0.7),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_flipper

Penguin body mass vs bill depth

mass_bill_depth <- ggplot(data = penguins,
                       aes(x = bill_depth_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Bill depth and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Bill depth (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.8, 0.8),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_bill_depth

This shows that there is very little difference between Adelie and Chinstrap, but Gentoo is markedly different. All three species have a positive relationship (as bill depth increases, so does body mass), which is not what the numerical correlation coefficient of -0.472 ould have us believe.

Penguin body mass and flipper size, faceted by sex

ggplot(penguins, aes(x = flipper_length_mm,
                            y = body_mass_g)) +
  geom_point(aes(colour = sex)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","cyan4"), na.translate = FALSE) +
  labs(title = "Penguin flipper and body mass",
       subtitle = "Dimensions for male and female Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin sex") +
  theme(legend.position = "bottom",
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  facet_wrap(~species)

On average, male penguins seem to be heavier than female ones, and this is consistent along all three species.

Simpson’s Paradox: Penguin bill length vs bill depth

In the original scatterplot matrix without species, the lowest correlation coefficient of -0.235 was between bill length and bill depth. The following is just the scatterplot with the line of best fit added.

bill_no_species <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm)) +
  geom_point() +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions (omit species)",
       subtitle = "Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)") +
  theme(plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  geom_smooth(method = "lm", se = FALSE, color = "blue")

bill_no_species

However, if we plot the same scatterplot colouring points by species, we get a completely different story.

bill_len_dep <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm,
                             group = species)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.8) +
  geom_smooth(method = "lm", se = FALSE, aes(colour = species)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions",
       subtitle = "Bill length and depth for Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.85, 0.10),
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

bill_len_dep

Simpson’s paradox

Is Y heavily skewed? Does it need to be transformed?

THe variable we are interested to explain, body mass, does not appear to be heavily skewed so there is no need for any transformation.

Acknowledgements

This page is adapted from the Palmer Palmer Penguins package Vignette

Import data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Load external data from a .csv file into a data frame.
2. Describe what a data frame is.
3. Use indexing to subset specific portions of data frames.
4. Describe what a factor is.
5. Reorder and rename factors.
6. Format dates.

Overview

One of the things that I found strange when I started working with R was that, unlike other software like Excel, Stata, SPSS, etc., you couldn’t just double click on an .xls, .dta, or .sav file, load the data and look at its contents. In R, we must use a command to explicitly import the data into memory.

While there are many possible data formats, we will concentrate on CSV files, namely Comma Separated Values files that are a common way to save the raw data from spreadsheets, without any of the formatting, etc. The readr R package contains functions for importing data saved as flat file documents; readr is a core member of the tidyverse and is loaded everytime you call library(tidyverse).

CSV file names end with a .csv and if you opened one inside Excel, it would look like a regular Excel file. NASA provides an estimate of global surface temperature change which allows us to calculate weather anomalies. The data is available at https://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts+dSST.csv as a CSV file which you open inside Excel looks something like this:

However, this is what a CSV file looks like on the inside: a bunch of values separated with commas.

By the way, if you look at the data closely, you will notice that the values in the D-N (December-November) and DJF (December-January-February) columns for the year 1880 are ***. These *** denote a missing value, in the same way that R uses the NA (or not available) value.

If you’d like R to treat these *** values as missing, you will need to convert them to NAs. One way to do this is to ask read_csv() to parse *** values as NA values when it reads in the data.

Importing CSV files: `read_csv()`

Importing CSV is part of base R using the read.csv() command. However, we will use the readr package and its read_csv() command that allows us to read flat data. read_csv() is significantly (8-10 times) faster and more user friendly than the base R command, with no need to define rownames, no stringsAsFactors = TRUE.

Even though we only concentrate on CSV files, readr has several functions that allow you to import a specific flat file format.

Function	Reads
`read_csv()`	Comma separated values
`read_csv2()`	Semi-colon separate values
`read_delim()`	General delimited files
`read_fwf()`	Fixed width files
`read_log()`	Apache log files
`read_table()`	Space separated files
`read_tsv()`	Tab delimited values

Just as you can import data, readr allows you to export data and save it locally. These functions are similar to the read_ functions and each save a tibble (or data frame) in the specific file format.

Function	Writes
`write_csv()`	Comma separated values
`write_excel_csv()`	CSV that you plan to open in Excel
`write_delim()`	General delimited files
`write_file()`	A single string, written as is
`write_lines()`	A vector of strings, one string per line
`write_tsv()`	Tab delimited values

To use a write_ function, first give it the name of the data frame to save, then give it a filename to save in your working directory.

Importing CSV files directly off the internet

If the CSV exists on the internet and you have the URL address, you don’t have to download it to your local machine and then import it; you can import it directly off the web using the URL link.

url <- "https://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts+dSST.csv"
weather <- read_csv(url, skip = 1, na = "***")

When using the read_csv() function, we added two options:

skip = 1 option is there as the real data table only starts in Row 2, so we need to skip one row.
na = "***"option informs R how missing observations in the spreadsheet are coded. As discussed earlier, it is best to specify NA values here, as otherwise some of the data may not be recognized as numeric data.

Notice that the code above saves the output to an object named weather. You must save the output of read_csv() to an object if you wish to use it later; otherwise, read_csv() will just print the contents of the data set at the command line.

Also, the assignment statement doesn’t produce any output for weather because assignments don’t display anything. If we want to check that our data has been loaded, we can see glimpse the structure of the dataframe using glimpse() or see its contents by just typing its name: weather.

glimpse(weather) shows us the number of observations and variables, and then, for each variable, shows the variable type; in our case, all of the variables are dbl or double, namely numeric variables.
weather, just invoking the name of the dataframe, shows the contents of the dataframe in the tabular form it is saved in

glimpse(weather)

## Rows: 140
## Columns: 19
## $ Year  <dbl> 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1...
## $ Jan   <dbl> -0.54, -0.19, 0.22, -0.59, -0.23, -1.00, -0.68, -1.07, -0.53,...
## $ Feb   <dbl> -0.38, -0.25, 0.22, -0.67, -0.11, -0.37, -0.68, -0.58, -0.59,...
## $ Mar   <dbl> -0.26, 0.02, 0.00, -0.16, -0.65, -0.21, -0.57, -0.36, -0.58, ...
## $ Apr   <dbl> -0.37, -0.02, -0.36, -0.27, -0.62, -0.53, -0.34, -0.42, -0.24...
## $ May   <dbl> -0.11, -0.06, -0.32, -0.32, -0.42, -0.55, -0.34, -0.27, -0.16...
## $ Jun   <dbl> -0.22, -0.36, -0.38, -0.26, -0.52, -0.47, -0.43, -0.20, -0.04...
## $ Jul   <dbl> -0.23, -0.06, -0.37, -0.09, -0.48, -0.39, -0.20, -0.23, 0.04,...
## $ Aug   <dbl> -0.24, -0.03, -0.14, -0.26, -0.50, -0.44, -0.47, -0.52, -0.19...
## $ Sep   <dbl> -0.26, -0.23, -0.17, -0.33, -0.45, -0.32, -0.34, -0.17, -0.12...
## $ Oct   <dbl> -0.32, -0.40, -0.53, -0.21, -0.41, -0.30, -0.31, -0.40, 0.04,...
## $ Nov   <dbl> -0.37, -0.42, -0.32, -0.40, -0.48, -0.28, -0.45, -0.19, -0.03...
## $ Dec   <dbl> -0.48, -0.28, -0.42, -0.25, -0.40, 0.00, -0.17, -0.43, -0.26,...
## $ `J-D` <dbl> -0.32, -0.19, -0.21, -0.32, -0.44, -0.40, -0.42, -0.40, -0.22...
## $ `D-N` <dbl> NA, -0.21, -0.20, -0.33, -0.43, -0.44, -0.40, -0.38, -0.24, -...
## $ DJF   <dbl> NA, -0.31, 0.06, -0.56, -0.20, -0.59, -0.45, -0.61, -0.52, -0...
## $ MAM   <dbl> -0.24, -0.02, -0.22, -0.25, -0.56, -0.43, -0.42, -0.35, -0.33...
## $ JJA   <dbl> -0.23, -0.15, -0.30, -0.20, -0.50, -0.44, -0.37, -0.32, -0.06...
## $ SON   <dbl> -0.32, -0.35, -0.34, -0.32, -0.44, -0.30, -0.37, -0.25, -0.04...

weather

## # A tibble: 140 x 19
##     Year   Jan    Feb    Mar   Apr   May   Jun   Jul   Aug   Sep   Oct   Nov
##    <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  1880 -0.54 -0.38  -0.26  -0.37 -0.11 -0.22 -0.23 -0.24 -0.26 -0.32 -0.37
##  2  1881 -0.19 -0.25   0.02  -0.02 -0.06 -0.36 -0.06 -0.03 -0.23 -0.4  -0.42
##  3  1882  0.22  0.22   0     -0.36 -0.32 -0.38 -0.37 -0.14 -0.17 -0.53 -0.32
##  4  1883 -0.59 -0.67  -0.16  -0.27 -0.32 -0.26 -0.09 -0.26 -0.33 -0.21 -0.4 
##  5  1884 -0.23 -0.11  -0.65  -0.62 -0.42 -0.52 -0.48 -0.5  -0.45 -0.41 -0.48
##  6  1885 -1    -0.37  -0.21  -0.53 -0.55 -0.47 -0.39 -0.44 -0.32 -0.3  -0.28
##  7  1886 -0.68 -0.68  -0.570 -0.34 -0.34 -0.43 -0.2  -0.47 -0.34 -0.31 -0.45
##  8  1887 -1.07 -0.580 -0.36  -0.42 -0.27 -0.2  -0.23 -0.52 -0.17 -0.4  -0.19
##  9  1888 -0.53 -0.59  -0.580 -0.24 -0.16 -0.04  0.04 -0.19 -0.12  0.04 -0.03
## 10  1889 -0.31  0.35   0.07   0.15 -0.05 -0.12 -0.1  -0.16 -0.26 -0.34 -0.61
## # ... with 130 more rows, and 7 more variables: Dec <dbl>, `J-D` <dbl>,
## #   `D-N` <dbl>, DJF <dbl>, MAM <dbl>, JJA <dbl>, SON <dbl>

When you use read_csv(), read_csv() tries to match each column of input to one of the basic data types in R. In our case, read_csv() misidentified the contents of the Year column to a real number, rather than an integer. You can correct this with R’s as.integer() function, or you can read the data in again, this time instructing read_csv() to parse the column as integers.

To do this:

add the argument col_types to read_csv()
set equal the col_types arguent to a list.
add a named element to the list for each column you would like to manually parse; in our case, we want to make column ‘Year’ an integer.

weather <- read_csv(url, skip = 1, na = "***",
                   col_types = list(Year = col_integer()))

To complete the code, set Year equal to one of the functions below, each function instructs read_csv() to parse Year as a specific type of data.

Type function	Data Type
`col_character()`	character
`col_date()`	Date
`col_datetime()`	POSIXct (date-time)
`col_double()`	double (numeric)
`col_factor()`	factor
`col_guess()`	let readr geuss (default)
`col_integer()`	integer
`col_logical()`	logical
`col_number()`	numbers mixed with non-number characters
`col_numeric()`	double or integer
`col_skip()`	do not read this column
`col_time()`	time

Importing CSV files saved locally

If you want to read a CSV file you have saved locally in your computer, you must let RStudio know which folder the file lives in; in more technical terms, you have to set the Working Directory. You can determine the location of your working directory by running getwd(). You can change the location of your working directory by going to Session > Set Working Directory in the RStudio IDE menu bar or use the Ctrl+Shift+H shortcut in Windows, Cmd+Shift+H in Mac to browse for the folder where the file resides.

Need for speed: Enter `data.table::fread()`

If you have a file which is fairly large, the fread() function from the data.table package, data.table::fread(), can make your life easier. You use it in a similar way to read_csv(), but it is faster. The following table compares how long it takes to read 1.1 million rows from CDC’s COVID-19 Case Surveillance Public Use Data. We downloaded the CSV locally and then read it with base R read.csv(), readr::read_csv(), and data.table::fread()

library(microbenchmark)
mbm = microbenchmark(
  baseR =   read.csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  readr =   read_csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  data.table =   fread("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  times=10
)
mbm

Unit: seconds
expr	min	lq	mean	median	uq	max	neval	cld
baseR	6.314878	7.598240	9.930234	9.556545	11.353675	15.014476	10	c
readr	4.450299	5.270310	6.300699	6.317381	7.489372	8.341003	10	b
data.table	1.080079	1.950533	2.240901	2.251069	2.729169	3.030911	10	a

The diffenrence in speed is remarkarble; base R is the slowest with an avearge loading time of 9.93 seconds. read_csv() yields an average of 6.3 seconds, and data.table:fread() reduces loading time to 2.24 seconds.

Other data formats

The readr package provides efficient functions for reading and saving common flat file data formats. For other data types, consider using :

Package	Reads
readxl	Excel files (.xls, .xlsx)
haven	SPSS, Stata, and SAS files
jsonlite	json
xml2	xml
httr	web API’s
rvest	web pages (web scraping)
DBI	databases
sparklyr	data loaded into spark

`rio`: a swiss-army knife for data input-output

A really neat package to handle importing- exporing data is rio whose authors call it A Swiss-Army knife for data input-output. It works by determining the data structure from the file extension, uses reasonable defaults for data import and export (e.g., ‘stringsAsFactors=FALSE’), supports web-based import (including from SSL/HTTPS). It also has a useful function, ‘convert()’, that provides a simple method for converting between file types. You can read more about rio here

Never work directly on the raw data

In 2012 Cecilia Giménez, an 83-year-old widow and amateur painter, attempted to restore a century-old fresco of Jesus crowned with thorns in her local church in Borja, Spain. The restoration didn’t go very well, but, surprisingly, the botched restoration of Jesus fresco miraculously saved the Spanish Town.

As a most important rule, please do not work on the raw data; it’s unlikely you will have Cecilia Giménez’s good fortune to become (in)famous for your not-so-brilliant work. Make sure you import the data in R, leave the raw data aside, and if you make any changes tidying and wrangling your data, save it using write_csv() with a different file name.

Sampling and confidence intervals (CI)

Wed, 29 Jul 2020 00:00:00 +0000

We are back to dealing with penguins, and we want to explore body mass and flipper length across the three different species.

Summary statistics

penguins %>%
  group_by(species) %>%
  summarize(across(c( body_mass_g, flipper_length_mm),
                   mean, na.rm = TRUE)) %>% 
  kable()

species	body_mass_g	flipper_length_mm
Adelie	3701	190
Chinstrap	3733	196
Gentoo	5076	217

Boxplots

body_mass_plot <- ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

body_mass_plot

flipper_plot <- ggplot(data = penguins, aes(y = species, x = flipper_length_mm)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")



flipper_plot

Confidence Intervals (CI)

CIs for body mass

formula_ci_body_mass <- penguins %>%
  group_by(species) %>%
  summarise( mean_body_mass = mean(body_mass_g, na.rm = TRUE), 
             sd_mass = sd(body_mass_g, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_mass/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_body_mass - margin_of_error,
             ci_high = mean_body_mass + margin_of_error
  )


formula_ci_body_mass %>% 
  kable()

species	mean_body_mass	sd_mass	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	3701	459	152	1.98	37.2	73.5	3627	3774
Chinstrap	3733	384	68	2.00	46.6	93.0	3640	3826
Gentoo	5076	504	124	1.98	45.3	89.6	4986	5166

#visualise  CIs for all species 
ggplot(formula_ci_body_mass, 
       aes(x=reorder(species, mean_body_mass), 
           y=mean_body_mass, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean body mass (grams)", 
       title="Which species has the highest mean weight?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_body_mass,
             aes(x=mean_body_mass, y = species), colour = "orange", size = 8)+

  
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

CIs for flipper length

formula_ci_flipper_length <- penguins %>%
  group_by(species) %>%
  summarise( mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE), 
             sd_flipper_length = sd(flipper_length_mm, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_flipper_length/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_flipper_length - margin_of_error,
             ci_high = mean_flipper_length + margin_of_error
  )


formula_ci_flipper_length %>% 
  kable()

species	mean_flipper_length	sd_flipper_length	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	190	6.54	152	1.98	0.530	1.05	189	191
Chinstrap	196	7.13	68	2.00	0.865	1.73	194	198
Gentoo	217	6.49	124	1.98	0.582	1.15	216	218

#visualise  CIs for all species 
ggplot(formula_ci_flipper_length, 
       aes(x=reorder(species, mean_flipper_length), 
           y=mean_flipper_length, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean flipper length (mm)", 
       title="Which species has the longest mean flipper?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= flipper_length_mm)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_flipper_length,
             aes(x=mean_flipper_length, y = species), colour = "orange", size = 8)+

  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length (in mm) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")

Remember in the penguins data, we saw that Gentoo penguins are very unlike the other two; however, what if we wanted to compare Adelie and Chinstrap both in terms of body mass and flipper length? By looking at the confidence intervals, we already have an indication as to whether there is a difference or not. We will use a t-Test to check if the group means are different.

Briefly, a t-Test should be used when we want to assess whether the mean between two groups are similar or not. The null hypothesis for a t-test is that the two means are equal, and the alternative is that they are not.

t-Test assuming unequal variance

R’s built-in function for running a t-test is t.test() and by default R assumes that the variance in the two groups’ populations is not equal.

t-Test for body mass

Remember that in our plots, body mass seemed to be fairly similar. While there was variability between the two species, the two average values were fairly similar and the two Confidence Intervals ovelapped quite a bit.

When we run our hypothesis test, we must first set up the hypotheses.

Null Hypothesis, \(H_0\): There is no difference in mean body mass measurements between the two species (Adelie and Chinstrap). In other words \(\mu_1 = \mu_2\), or their difference is equal to 0.
Altenative Hypothesis, \(H_1\): There is a difference in mean body mass measurements between the two species.

Does the data provide enough evidence to reject the null hypothesis, or could the variation be due to luck? Typically, we wanr the p-value to be less than 5%, or equivalently the t-stat to be roughly more than 2, as fairly strong evidence to reject the null hypothesis.

#select only Adelie and Chinstrap penguins
adelie_chinstrap_test_data <- penguins %>%
  filter(species %in% c("Adelie", "Chinstrap"))


test1 <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data) 

test1

## 
##  Welch Two Sample t-test
## 
## data:  body_mass_g by species
## t = -0.5, df = 152, p-value = 0.6
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -150.4   85.5
## sample estimates:
##    mean in group Adelie mean in group Chinstrap 
##                    3701                    3733

In our case, the t-test confirms what we already knew. First, the t-value is -0.5 and the p-value=0.6. Another way to look at it, is that the CI for the difference between the two means is [-150.4, 85.5] which contains zero indicating that we do not have strong evidence to reject the null hypothesis.

We can use broom:tidy() to convert these t-test results to a nice data frame.

test1_tidy <- tidy(test1) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.543   0.588      152.    -150.      85.5
## # ... with 2 more variables: method <chr>, alternative <chr>

A much cleaner output! The estimated average difference in body mass is -32.4g (we subtracted Adelie - Chinstrap, 3701-3733), the t-statistic = -0.543 and the p-value = 0.588, way greater than 0.05.

t-Test for flipper length

Null Hypothesis, \(H_0\): There is no difference in mean flipper length measurements between the two species (Adelie and Chinstrap).
Altenative Hypothesis, \(H_1\): There is a difference in mean flipper length measurements between the two species.

test2 <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data) 

test2_tidy <- tidy(test2) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.78 6.05e-8      120.    -7.88     -3.86
## # ... with 2 more variables: method <chr>, alternative <chr>

In our case, the t-test confirms what we already knew. the t-value is well above 2 and the p-value well below 0.05, indicating that we have strong evidence to reject the null hypothesis and therefore determine that there is a difference in mean flipper length.

The estimated average difference in flipper length is -5.9mm, the t-statistic is t-stat = -5.78 and the p-value = 6.05e-08 = \(6.05*10^{-8} = 0.00000605\), a tiny number which is way less than 0.05.

So where does this leave us?

In terms of body mass even though we measured an average difference of 32 grams, this is not statistically significant, as its t-statistic was less than 2 and, equivalently, its p-value is >> 0.05
In terms of lfipper length, he measured average difference of 5.87mm is statistically significant as the t-statistic is 5.78 and the p-vaue <<< 0.0.5

t-Test assuming equal variance

We can run t.test() assuming the two groups have equal variance.

test1_equal_variance <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test1_tidy_equal_variance <- tidy(test1_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.508   0.612       217    -158.      93.4
## # ... with 2 more variables: method <chr>, alternative <chr>

test2_equal_variance <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test2_tidy_equal_variance <- tidy(test2_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.97 9.38e-9       217    -7.81     -3.93
## # ... with 2 more variables: method <chr>, alternative <chr>

How do we test whether the two groups have equal variance?

There are several ways to check if the two groups have equal variance. For all these tests, the null hypothesis is that the two groups have equal variances.

As in all hypothesis tests, if the p-value is less than 0.05, we can assume that they have unequal variances.

Bartlett test: Check equality of variances based on the mean

body_mass_variance <- bartlett.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data)
body_mass_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  body_mass_g by species
## Bartlett's K-squared = 3, df = 1, p-value = 0.1

flipper_variance <- bartlett.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data)
flipper_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Bartlett's K-squared = 0.7, df = 1, p-value = 0.4

In both cases, since the p-value is greater than 0.05, we cannot reject the null hypothesis so we assume that the two groups have equal variances.

Levene test Check equality of variances based on the median

Levene’s test also checks for homogeneity of variance and can based either on the mean or on the median. The median is a robust statistic, as it’s not influenced by outliers as much as the mean can be.

car::leveneTest(body_mass_g ~ species, 
                center = mean,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)  
## group   1    4.63  0.032 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = mean,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

car::leveneTest(body_mass_g ~ species, 
                center = median,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)  
## group   1    4.82  0.029 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = median,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

Checking for homogeneity of variance based on the median, we can reject the null hypothesis for body mass (p-value = 0.029 < 0.05), but not for flipper length.

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

fligner.test(body_mass_g ~ species, 
             data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  body_mass_g by species
## Fligner-Killeen:med chi-squared = 4, df = 1, p-value = 0.04

fligner.test(flipper_length_mm ~ species, 
              data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Fligner-Killeen:med chi-squared = 0.5, df = 1, p-value = 0.5

Let us summarise all the p-values from these tests

Test	Body Mass	Flipper Length
Bartlett	0.1	0.4
Levene (mean)	0.032	0.43
Levene (median)	0.029	0.43
Fligner-Killeen	0.04	0.4

In all of the Body mass tests, with the exception of the Bartlett tests, the p-value is less than 0.05. In other words, we sem to have enough evidence to conclude that the variances are different.

However, in all of the flipper length tests,, all of the p-values are > 0.0.5, which means we cannot reject the null hypothesis so we’re probably safe assuming the variances are equal and leaving var.equal = TRUE on.

Acknowledgements

This page is adapted from the Palmer Penguins package Vignette

Testing for differences in mean values

Wed, 29 Jul 2020 00:00:00 +0000

We are back to dealing with penguins, and we want to explore body mass and flipper length across the three different species.

Summary statistics

penguins %>%
  group_by(species) %>%
  summarize(across(c( body_mass_g, flipper_length_mm),
                   mean, na.rm = TRUE)) %>% 
  kable()

species	body_mass_g	flipper_length_mm
Adelie	3701	190
Chinstrap	3733	196
Gentoo	5076	217

Boxplots

body_mass_plot <- ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

body_mass_plot

flipper_plot <- ggplot(data = penguins, aes(y = species, x = flipper_length_mm)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")



flipper_plot

Confidence Intervals (CI)

CIs for body mass

formula_ci_body_mass <- penguins %>%
  group_by(species) %>%
  summarise( mean_body_mass = mean(body_mass_g, na.rm = TRUE), 
             sd_mass = sd(body_mass_g, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_mass/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_body_mass - margin_of_error,
             ci_high = mean_body_mass + margin_of_error
  )


formula_ci_body_mass %>% 
  kable()

species	mean_body_mass	sd_mass	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	3701	459	152	1.98	37.2	73.5	3627	3774
Chinstrap	3733	384	68	2.00	46.6	93.0	3640	3826
Gentoo	5076	504	124	1.98	45.3	89.6	4986	5166

#visualise  CIs for all species 
ggplot(formula_ci_body_mass, 
       aes(x=reorder(species, mean_body_mass), 
           y=mean_body_mass, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean body mass (grams)", 
       title="Which species has the highest mean weight?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_body_mass,
             aes(x=mean_body_mass, y = species), colour = "orange", size = 8)+

  
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

CIs for flipper length

formula_ci_flipper_length <- penguins %>%
  group_by(species) %>%
  summarise( mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE), 
             sd_flipper_length = sd(flipper_length_mm, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_flipper_length/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_flipper_length - margin_of_error,
             ci_high = mean_flipper_length + margin_of_error
  )


formula_ci_flipper_length %>% 
  kable()

species	mean_flipper_length	sd_flipper_length	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	190	6.54	152	1.98	0.530	1.05	189	191
Chinstrap	196	7.13	68	2.00	0.865	1.73	194	198
Gentoo	217	6.49	124	1.98	0.582	1.15	216	218

#visualise  CIs for all species 
ggplot(formula_ci_flipper_length, 
       aes(x=reorder(species, mean_flipper_length), 
           y=mean_flipper_length, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean flipper length (mm)", 
       title="Which species has the longest mean flipper?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= flipper_length_mm)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_flipper_length,
             aes(x=mean_flipper_length, y = species), colour = "orange", size = 8)+

  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length (in mm) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")

t-Test assuming unequal variance

R’s built-in function for running a t-test is t.test() and by default R assumes that the variance in the two groups’ populations is not equal.

t-Test for body mass

When we run our hypothesis test, we must first set up the hypotheses.

Null Hypothesis, \(H_0\): There is no difference in mean body mass measurements between the two species (Adelie and Chinstrap). In other words \(\mu_1 = \mu_2\), or their difference is equal to 0.
Altenative Hypothesis, \(H_1\): There is a difference in mean body mass measurements between the two species.

#select only Adelie and Chinstrap penguins
adelie_chinstrap_test_data <- penguins %>%
  filter(species %in% c("Adelie", "Chinstrap"))


test1 <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data) 

test1

## 
##  Welch Two Sample t-test
## 
## data:  body_mass_g by species
## t = -0.5, df = 152, p-value = 0.6
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -150.4   85.5
## sample estimates:
##    mean in group Adelie mean in group Chinstrap 
##                    3701                    3733

We can use broom:tidy() to convert these t-test results to a nice data frame.

test1_tidy <- tidy(test1) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.543   0.588      152.    -150.      85.5
## # ... with 2 more variables: method <chr>, alternative <chr>

A much cleaner output! The estimated average difference in body mass is -32.4g (we subtracted Adelie - Chinstrap, 3701-3733), the t-statistic = -0.543 and the p-value = 0.588, way greater than 0.05.

t-Test for flipper length

Null Hypothesis, \(H_0\): There is no difference in mean flipper length measurements between the two species (Adelie and Chinstrap).
Altenative Hypothesis, \(H_1\): There is a difference in mean flipper length measurements between the two species.

test2 <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data) 

test2_tidy <- tidy(test2) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.78 6.05e-8      120.    -7.88     -3.86
## # ... with 2 more variables: method <chr>, alternative <chr>

The estimated average difference in flipper length is -5.9mm, the t-statistic is t-stat = -5.78 and the p-value = 6.05e-08 = \(6.05*10^{-8} = 0.00000605\), a tiny number which is way less than 0.05.

So where does this leave us?

In terms of body mass even though we measured an average difference of 32 grams, this is not statistically significant, as its t-statistic was less than 2 and, equivalently, its p-value is >> 0.05
In terms of lfipper length, he measured average difference of 5.87mm is statistically significant as the t-statistic is 5.78 and the p-vaue <<< 0.0.5

t-Test assuming equal variance

We can run t.test() assuming the two groups have equal variance.

test1_equal_variance <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test1_tidy_equal_variance <- tidy(test1_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.508   0.612       217    -158.      93.4
## # ... with 2 more variables: method <chr>, alternative <chr>

test2_equal_variance <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test2_tidy_equal_variance <- tidy(test2_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.97 9.38e-9       217    -7.81     -3.93
## # ... with 2 more variables: method <chr>, alternative <chr>

How do we test whether the two groups have equal variance?

There are several ways to check if the two groups have equal variance. For all these tests, the null hypothesis is that the two groups have equal variances.

As in all hypothesis tests, if the p-value is less than 0.05, we can assume that they have unequal variances.

Bartlett test: Check equality of variances based on the mean

body_mass_variance <- bartlett.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data)
body_mass_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  body_mass_g by species
## Bartlett's K-squared = 3, df = 1, p-value = 0.1

flipper_variance <- bartlett.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data)
flipper_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Bartlett's K-squared = 0.7, df = 1, p-value = 0.4

In both cases, since the p-value is greater than 0.05, we cannot reject the null hypothesis so we assume that the two groups have equal variances.

Levene test Check equality of variances based on the median

car::leveneTest(body_mass_g ~ species, 
                center = mean,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)  
## group   1    4.63  0.032 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = mean,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

car::leveneTest(body_mass_g ~ species, 
                center = median,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)  
## group   1    4.82  0.029 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = median,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

Checking for homogeneity of variance based on the median, we can reject the null hypothesis for body mass (p-value = 0.029 < 0.05), but not for flipper length.

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

fligner.test(body_mass_g ~ species, 
             data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  body_mass_g by species
## Fligner-Killeen:med chi-squared = 4, df = 1, p-value = 0.04

fligner.test(flipper_length_mm ~ species, 
              data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Fligner-Killeen:med chi-squared = 0.5, df = 1, p-value = 0.5

Let us summarise all the p-values from these tests

Test	Body Mass	Flipper Length
Bartlett	0.1	0.4
Levene (mean)	0.032	0.43
Levene (median)	0.029	0.43
Fligner-Killeen	0.04	0.4

In all of the Body mass tests, with the exception of the Bartlett tests, the p-value is less than 0.05. In other words, we sem to have enough evidence to conclude that the variances are different.

Acknowledgements

This page is adapted from the Palmer Penguins package Vignette

Inpect data

Fri, 24 Jul 2020 00:00:00 +0000

Learning Objectives
1. Glimpse the structure of the dataframe
3. Summarize the structure of a dataframe

Overview

Once you have loaded your data set into R, you have to inspect and get a feel for the data. We are typically interested in the following:

The dimensions of the data set; how many rows (cases) and how many columns does the data frame have.
The types of variables we have; are they integer, character, logical, factor (categorical) etc.
The number of missing, or NA, values in the dataframe.
A quick look at some summary statistics

First, if we wanted to look at the dataset in a spreadsheet-style data viewer, we can just invoke View(gapminder) (View with a capital V).

While this is nice, it is not very useful, as we cannot dig deeper and see what kind of variables we have, whether there are any missing values, etc.

There are two functions that we will talk about, dplyr::glimpse() and skimr::skim().

`dplyr::glimpse()`

glimpse() is like a transposed version of print(): It first gives you the dimensions (rows and columns) and then gives us the dataframe’s columns (or variables), the variable type (, , ), and then gives us the first few values of each variable. Let us look at the outcome of

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

We see that we have 1704 rows, or cases. We also have 6 columns, or variables and right underneath we see each column individually:

country is a categorical, or factor variable of type <fct>. and the first few cases are all Afghanistan, just because it is the first one alphabetically
continent is also a factor variable, and the first values of this categorical variable are “Asia”, "Europe’, etc.
year is an integer variable of type <int>. This is the year for which we have data for each country, between 1952 and 2007 in 5-year intervals.
lifeExp is a double precision, or real number, of type <dbl> that refers to life expectancy
pop is an integer variable of type <int> that refers to the population
gdpPercap is a double precision, or real number, of type <dbl> that refers to GDP per capita

`skimr::skim()`

While glimpse() allows us to look at the contents of the dataframe, skimr::skim() is more useful and I always use it in my workflow.

skimr::skim(gapminder)

Table 1: Data summary
Name	gapminder
Number of rows	1704
Number of columns	6
_______________________
Column type frequency:
factor	2
numeric	4
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
country	0	1	FALSE	142	Afg: 12, Alb: 12, Alg: 12, Ang: 12
continent	0	1	FALSE	5	Afr: 624, Asi: 396, Eur: 360, Ame: 300

Variable type: numeric

skim_variable	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
year	1	1979.50	17.27	1952.00	1965.75	1979.50	1993.25	2007.0	▇▅▅▅▇
lifeExp	1	59.47	12.92	23.60	48.20	60.71	70.85	82.6	▁▆▇▇▇
pop	1	29601212.32	106157896.74	60011.00	2793664.00	7023595.50	19585221.75	1318683096.0	▇▁▁▁▁
gdpPercap	1	7215.33	9857.45	241.17	1202.06	3531.85	9325.46	113523.1	▇▁▁▁▁

skimr::skim() give us a data summary with the dimensions (rows and columns) of the dataframe, and the type of columns; in this case, we have 2 factor and 2 numeric columns (variables).

For all variable types, it gives us the number of missing values (n_missing) and the complete_rate; in the gapminder data, there are no missing values, so n_mising = 0 and complete_rate = 1.

For factor variables, skim() provides information on
- whether it is an ordered factor; if false, the default ordering is alphabetical, otherwise one has to explicitly specify the order of the categories.
- the n_unique, or distinct instances of each country; in gapminder we have data on 142 distinct countries and 5 continents
- the top_counts shows the top number of instances for each factor; each country has 12 observations, but in continent Africa has 624 observations, Asia 396, etc.
For numeric variables, skim() provides summary statistics; mean, standard deviation and the 0th (min), 25th, 50, 75th and 100th (max) percentile. It also gives us a rough histogram to get an idea on the shape of the distribution (normal, skewed, uniform)

On you own

The following dataframe has data on London’d cycle hire scheme, Santander Cycles. Besides the number of bikes rented out, the dataframe also contains weather information.

bikes <- read_csv(here("data", "londonBikes.csv"))

skimr::skim(bikes)

Table 2: Data summary
Name	bikes
Number of rows	3439
Number of columns	14
_______________________
Column type frequency:
character	1
logical	4
numeric	9
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	empty	n_unique	whitespace
date	0	1	8	8	0	3439	0

Variable type: logical

skim_variable	n_missing	complete_rate	mean	count
rain	851	0.75	0.62	TRU: 1595, FAL: 993
fog	851	0.75	0.07	FAL: 2403, TRU: 185
thunderstorm	851	0.75	0.03	FAL: 2512, TRU: 76
snow	851	0.75	0.02	FAL: 2533, TRU: 55

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bikes_hired	0	1.00	26158.95	9135.13	3531.0	19626.00	26022.0	32759.0	73094.0	▃▇▅▁▁
season	0	1.00	2.46	1.12	1.0	1.00	2.0	3.0	4.0	▇▇▁▇▇
max_temp	1877	0.45	16.48	6.19	-1.2	11.93	16.7	20.9	36.7	▁▆▇▃▁
min_temp	1929	0.44	7.62	5.14	-8.2	3.90	7.9	11.8	20.0	▁▅▇▇▂
avg_temp	27	0.99	11.70	5.41	-4.1	7.60	11.6	15.9	28.6	▁▆▇▅▁
avg_humidity	745	0.78	74.91	10.84	37.0	67.00	76.0	83.0	100.0	▁▂▆▇▂
avg_pressure	773	0.78	1015.10	10.24	979.0	1009.00	1016.0	1022.0	1044.0	▁▂▇▆▁
avg_windspeed	745	0.78	14.01	6.10	3.0	10.00	13.0	18.0	47.0	▇▇▂▁▁
rainfall_mm	51	0.99	1.67	3.68	0.0	0.00	0.0	1.5	48.0	▇▁▁▁▁

A couple of graded learnr interactive exercices?

What kind of variable is date? What kind of variable is season?
How often does it rain in London?
What is the average annual temperature (in degrees C)?
What is the maximum rainfall?

Simulation-based tests

Wed, 29 Jul 2020 00:00:00 +0000

There is only one test

We can use t.test() to check whether two means are equal or not. Instead of dealing with the assumptions of the data and finding the appropriate statistical test to test for equality of variance, we can use the power of bootstrapping, permutation, and simulation to construct a null distribution, calculate confidence intervals and run ny kind of test for inference.

The beauty and power of simulation is that we can run hypothesis test not just on differences of means that we have fomuals for, but also on other parameters, like medians

According to Allen Downey, there is only one statistical test and that all statistical tests follow the same pattern:

Step 1: Calculate a sample statistic from the observed data, or \(\delta^{*}\). This is the measure you care about: the difference in means, the mean, the median, the proportion, the difference in proportions, anything you want really!
Step 2: Use simulation to invent a world where \(\delta\) is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in medians or proportions, or where the average value is a specific number.
Step 3: Look at \(\delta^{*}\) in the null world. Put the observed sample statistic in the null world and see if it fits well.
Step 4: Calculate the probability that \(\delta^{*}\) could exist in null world. This is the p-value, or the probability that you’d see a \(\delta^{*}\) at least that high in a world where there’s no difference.
Step 5: Decide if \(\delta^{*}\) is statistically significant. Choose some threshold, cutoff value (e.g., 0.05) for deciding if there’s sufficient proof for rejecting the null world.

Bootstrap simulation, using `infer` package

CI for median

Clean data

Fri, 24 Jul 2020 00:00:00 +0000

`janitor` package for cleaning variable names

When we create data files, we frequently use variable names and formats that are easily readable for humans, but no so for computers.

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets. – For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights The New York Times, 2014

janitor has many functions, but its core function is clean_names() which will make your life easier if you call it whenever you load data into R. The following example is taken from janitor’s documentation page

Let us read an Excel file with a roster of teachers at a fictional American high school, stored in the Microsoft Excel file dirty_data.xlsx.

Some of the variable names, e.g., First Name, Last Name, are not only capitalised, but also contain a space in the variable name. Let us read in the file and have a glimpse inside it.

roster <- readxl::read_excel(here("data", "dirty_data.xlsx"))

glimpse(roster)

## Rows: 13
## Columns: 11
## $ `First Name`        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Ch...
## $ `Last Name`         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice",...
## $ `Employee Status`   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Ad...
## $ Subject             <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics...
## $ `Hire Date`         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037...
## $ `% Allocated`       <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0...
## $ `Full time?`        <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes"...
## $ `do not edit! --->` <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ Certification...9   <chr> "Physical ed", "Physical ed", "Instr. music", "...
## $ Certification...10  <chr> "Theater", "Theater", "Vocal music", "Computers...
## $ Certification...11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

We notice that if we wanted to refer to the variable for a first name (1st in the list) or percent allocated (6th in the list), we would need to refer to them as the string “First Name” and “% Allocated” respectively. To avoid this, we can use janitor::clean_names()

roster_clean <- roster %>% 
  clean_names()

glimpse(roster_clean)

## Rows: 13
## Columns: 11
## $ first_name        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Chie...
## $ last_name         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice", "...
## $ employee_status   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Admi...
## $ subject           <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics",...
## $ hire_date         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037, ...
## $ percent_allocated <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0.5...
## $ full_time         <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", ...
## $ do_not_edit       <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ certification_9   <chr> "Physical ed", "Physical ed", "Instr. music", "PE...
## $ certification_10  <chr> "Theater", "Theater", "Vocal music", "Computers",...
## $ certification_11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

Now, the variable names contain no spaces, are all lower case, and we can explicitly refer to them rather than using a string of characters– it all makes life a bit easier!

Code that works is not necessarily good code

According to Phil Karlton, there are only two hard things in Computer Science: cache invalidation and naming things. It is good practice to use meaningful names for variables and data frames, use spacing, comments, etc. Both Google and Hadley Wickham have great style guides for programming in R and the janitor package helps in creating variable names with a consistent style.

Linear Model Fitting

Tue, 28 Jul 2020 00:00:00 +0000

We will be using the Palmer penguins data to understand body mass.

Fit a model using `lm(Y ~ X1 + X2 +..., data = dataframe)`

The function to fit a linear regression model in R is lm(Y ~ X1 + X2 +..., data = mydataframe). lm, as many other functions in R, uses the formula interface The tilde (~) can be translated as is a function of. We are saying that \(Y\) is a function of \(X1\), \(X2\), etc., and the data for our analysis comes from mydataframe.

Back to our penguins, we want to see whether body mass is a function of flipper length. We create an object called model1 that holds the results of this linear regression model.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

Look at model output

We will be using the broom package to make modelling easier to work with. There are 3 main functions in broom:

tidy() - This is where you get most of the output you want, including coefficients and p-values
glance() - additional measures on your model, including R-squared, log likelihood, and AIC/BIC
augment() - make predictions with your model using new data

For now, we will use broom::tidy() and broom::glance() to get model results.

model1 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-5.78e+03	306	-18.9	5.59e-55
flipper_length_mm	49.7	1.52	32.7	4.37e-107

model1 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.759	0.758	394	1.07e+03	4.37e-107	1	-2.53e+03	5.06e+03	5.07e+03	5.29e+07	340	342

Write down the equation for model1

\[ \text{body_mass_g} = \alpha + \beta_{1}(\text{flipper_length_mm}) + \epsilon \] \[ \text{body_mass_g} = -5780.83 + 49.69(\text{flipper_length_mm}) + \epsilon \]

Plot scatterplot and residuals

Add more explanatory variables

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

# Fit a model with all explanatory variables (~ .)
model6 <- lm(body_mass_g ~ . , data = penguins)

Check collinearity

With so many explanatory variables, we need to worry about colinearity, i.e., whether the explanatory variables (all of the \(X\)’s) are highly correlated among themselves.

#model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)
car::vif(model2)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 4.51  1            2.12
## species           4.51  2            1.46

# model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)
car::vif(model3)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 6.05  1            2.46
## species           5.65  2            1.54
## sex               1.36  1            1.17

# model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)
car::vif(model4)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.44  1            2.54
## species           18.16  2            2.06
## sex                1.81  1            1.35
## bill_length_mm     5.95  1            2.44

# model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)
car::vif(model5)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.69  1            2.59
## species           41.07  2            2.53
## sex                2.31  1            1.52
## bill_length_mm     6.07  1            2.46
## bill_depth_mm      6.08  1            2.47

# model6 <- lm(body_mass_g ~ . , data = penguins)
car::vif(model6)

##                    GVIF Df GVIF^(1/(2*Df))
## species           71.20  2            2.90
## island             3.76  2            1.39
## bill_length_mm     6.12  1            2.47
## bill_depth_mm      6.27  1            2.50
## flipper_length_mm  7.78  1            2.79
## sex                2.34  1            1.53
## year               1.17  1            1.08

model7 <- lm(body_mass_g ~ flipper_length_mm +  sex + bill_depth_mm , data = penguins)
car::vif(model7)

## flipper_length_mm               sex     bill_depth_mm 
##              2.44              1.89              2.65

Summary model comparison table using `huxtable::huxreg()`

Which of the six models we have fit seems to be the best one? Let us compare them on one table.

huxreg(model1, model2, model3, model4, model5, model6, model7,
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
(Intercept)	-5780.831 ***	-4031.477 ***	-365.817	-759.064	-1460.995 *	84087.945 *	-2246.829 ***
	(305.815)	(584.151)	(532.050)	(541.377)	(571.308)	(41912.019)	(625.286)
flipper_length_mm	49.686 ***	40.705 ***	20.025 ***	17.847 ***	15.950 ***	18.504 ***	38.190 ***
	(1.518)	(3.071)	(2.846)	(2.902)	(2.910)	(3.128)	(2.084)
speciesChinstrap		-206.510 ***	-87.634	-291.711 ***	-251.477 **	-282.539 **
		(57.731)	(46.347)	(81.502)	(81.079)	(88.790)
speciesGentoo		266.810 **	836.260 ***	707.028 ***	1014.627 ***	890.958 ***
		(95.264)	(85.185)	(94.359)	(129.561)	(144.563)
sexmale			530.381 ***	465.395 ***	389.892 ***	378.977 ***	538.080 ***
			(37.810)	(43.081)	(47.848)	(48.074)	(51.310)
bill_length_mm				21.633 **	18.204 *	18.964 **
				(7.148)	(7.106)	(7.112)
bill_depth_mm					67.218 ***	60.798 **	-86.947 ***
					(19.742)	(20.002)	(15.456)
islandDream						-21.180
						(58.390)
islandTorgersen						-58.777
						(60.852)
year						-42.785 *
						(20.949)
#observations	342	342	333	333	333	333	333
R squared	0.759	0.783	0.867	0.871	0.875	0.877	0.823
Adj. R Squared	0.758	0.781	0.865	0.869	0.873	0.873	0.821
Residual SE	394.278	375.535	295.565	291.955	287.338	286.524	340.427
* p < 0.001; p < 0.01; * p < 0.05.

The best model seems to be model 7, so we will use broom::tidy() and broom::glance() to get model results.

model7 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-2.25e+03	625	-3.59	0.000376
flipper_length_mm	38.2	2.08	18.3	3.47e-52
sexmale	538	51.3	10.5	2.17e-22
bill_depth_mm	-86.9	15.5	-5.63	3.96e-08

model7 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.823	0.821	340	509	2.9e-123	3	-2.41e+03	4.83e+03	4.85e+03	3.81e+07	329	333

Let us write down its equation

\[ \begin{aligned} \text{body_mass_g} &= \alpha + \beta_{1}(\text{flipper_length_mm}) + \beta_{2}(\text{sex}_{\text{male}})\ + \beta_{3}(\text{bill_depth_mm}) + \epsilon \end{aligned} \]\[ \begin{aligned} \text{body_mass_g} &= -2246.83 + 38.19(\text{flipper_length_mm}) + 538.08(\text{sex}_{\text{male}})\ - 86.95(\text{bill_depth_mm}) + \epsilon \end{aligned} \]

Fitting multiple regression models in one go

Let us recall the relationship between body mass and bill depth and have a look at the scatteplot.

We could run three separate regression, but we can estimate three regression models with a few lines of code.

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm))
  )

## # A tibble: 6 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     -297.      469.    -0.634  5.27e- 1
## 2 Adelie    bill_depth_mm    218.       25.5    8.55   1.67e-14
## 3 Chinstrap (Intercept)      -36.2     613.    -0.0591 9.53e- 1
## 4 Chinstrap bill_depth_mm    205.       33.2    6.16   4.79e- 8
## 5 Gentoo    (Intercept)     -422.      488.    -0.864  3.89e- 1
## 6 Gentoo    bill_depth_mm    368.       32.5   11.3    1.64e-20

What we see is BLAH…

What if we add sex? First, let us facet_wrap() our scatter plot to see what it looks like

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm + sex))
  )

## # A tibble: 9 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     1931.      452.      4.28  3.47e- 5
## 2 Adelie    bill_depth_mm     81.6      25.6     3.19  1.74e- 3
## 3 Adelie    sexmale          556.       62.1     8.96  1.63e-15
## 4 Chinstrap (Intercept)      830.      861.      0.964 3.39e- 1
## 5 Chinstrap bill_depth_mm    153.       48.9     3.14  2.55e- 3
## 6 Chinstrap sexmale          156.      110.      1.42  1.60e- 1
## 7 Gentoo    (Intercept)     2741.      579.      4.73  6.37e- 6
## 8 Gentoo    bill_depth_mm    136.       40.6     3.35  1.08e- 3
## 9 Gentoo    sexmale          604.       79.8     7.57  9.94e-12

Simpson’s paradox

Recall from our EDA, we saw no relationship between bill length and depth.

bill_no_species

If we fit a simple regression model, we get the following

simpsons_model <- lm( bill_depth_mm ~ bill_length_mm, 
                      data = penguins %>% na.omit())

simpsons_model %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	20.8	0.854	24.3	1.03e-75
bill_length_mm	-0.0823	0.0193	-4.27	2.53e-05

simpsons_model %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.0523	0.0494	1.92	18.3	2.53e-05	1	-689	1.38e+03	1.39e+03	1.22e+03	331	333

The slope is significant, but the model r.squared (R2) explains only 5% of the overall variability.

However, when we plotted the same scatterplot colouring points by species, we got a completely different story.

bill_len_dep

We can again fit three individual models in one go

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( bill_depth_mm ~ bill_length_mm )),
    broom::glance(lm( bill_depth_mm ~ bill_length_mm ))
  )

## # A tibble: 6 x 16
## # Groups:   species [3]
##   species term  estimate std.error statistic  p.value r.squared adj.r.squared
##   <fct>   <chr>    <dbl>     <dbl>     <dbl>    <dbl>     <dbl>         <dbl>
## 1 Adelie  (Int~   11.5      1.37        25.2 1.51e- 6     0.149         0.143
## 2 Adelie  bill~    0.177    0.0352      25.2 1.51e- 6     0.149         0.143
## 3 Chinst~ (Int~    7.57     1.55        49.2 1.53e- 9     0.427         0.418
## 4 Chinst~ bill~    0.222    0.0317      49.2 1.53e- 9     0.427         0.418
## 5 Gentoo  (Int~    5.12     1.06        87.5 7.34e-16     0.428         0.423
## 6 Gentoo  bill~    0.208    0.0222      87.5 7.34e-16     0.428         0.423
## # ... with 8 more variables: sigma <dbl>, df <dbl>, logLik <dbl>, AIC <dbl>,
## #   BIC <dbl>, deviance <dbl>, df.residual <int>, nobs <int>

Alternatively, we can run a multiple regression model and the adjusted R2 = 76.5%

simpsons_model2 <- lm( bill_depth_mm ~ bill_length_mm + species, 
                      data = penguins %>% na.omit())

simpsons_model2 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	10.6	0.691	15.3	2.98e-40
bill_length_mm	0.2	0.0177	11.3	2.26e-25
speciesChinstrap	-1.93	0.226	-8.56	4.26e-16
speciesGentoo	-5.1	0.194	-26.3	1.04e-82

simpsons_model2 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.767	0.765	0.954	362	8.88e-104	3	-455	920	939	300	329	333

Model diagnostics

Wed, 29 Jul 2020 00:00:00 +0000

Linear Regression Assumptions: L-I-N-E

L: Linear relationship between (Y) and the explanatory variable (X)
I: Independence of errors—there’s no connection between how far any two points lie from the regression line _ N: Normal distribution of Y at each level of X
E: equality of variance of the errors – variability remains the same for all levels of X.

In other words, the residuals (errors) should satisfy the following:

L: The mean value for Y at each level of X lies on regression line.
I: There is no clear pattern in the errors
N: At each level of X, the values for Y are normally distributed.
E: The variability in the Y’s for each level of X is the same

Figure 1: Assumptions for linear ordinary least squares (OLS) regression

Regression diagnostic plots with `ggfortify::autoplot()`

Let us see what is happening in our models. We will use the ggfortify package and its autoplot() command to get the following regression diagnostic plots:

Residuals vs. Fitted: checks Linearity assumption. Residuals should be random, with no pattern, and around Y = 0; if not, there is a pattern in the data that is currently unaccounted for.
Normal Q-Q: checks residual Normality assumption. Deviations from a straight line indicate that residuals do not follow a Normal distribution.
Scale-Location: checks whether residuals have equal/constant variance or not. Positive or negative trends across the fitted values indicate variability that is not constant.
Residuals vs. Leverage: check for influential points. Points with high leverage (having unusual values of the predictors) and/or high absolute residuals can have an undue influence on estimates of model parameters.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

model6 <- lm(body_mass_g ~ . , data = penguins)


library(ggfortify)

autoplot(model1) +
  theme_minimal() + 
  labs (title = "Model 1 Diagnostic Plots")

autoplot(model2) +
  theme_minimal() + 
  labs (title = "Model 2 Diagnostic Plots")

autoplot(model3) +
  theme_minimal() + 
  labs (title = "Model 3 Diagnostic Plots")

autoplot(model4) +
  theme_minimal() + 
  labs (title = "Model 4 Diagnostic Plots")

autoplot(model5) +
  theme_minimal() + 
  labs (title = "Model 5 Diagnostic Plots")

autoplot(model6) +
  theme_minimal() + 
  labs (title = "Model 6 Diagnostic Plots")

Binary classification model

Tue, 28 Jul 2020 00:00:00 +0000

Visualise Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Produce scatter plots, boxplots, and time series plots using ggplot.
2. Set universal plot settings
3. Describe what faceting is and apply faceting in ggplot.
4. Modify the aesthetics of an existing ggplot plot (including axis labels and colour).
5. Build complex and customized plots from data in a data frame.

Overview

Above all else show the data.
–Edward Tufte, The Visual Display of Quantitative Information, 2001

ggplot2 has become the de facto standard for visualising data in R. The ggplot system moves away from a defined set of graphs (e.g., scatterplot, bar chart, etc) and instead breaks graphics down to their basic components and allows you to build plots layer by layer.

“In brief… a statistical graphic is a mapping from data to aesthetic attributes (colour, shape, size) of geometric objects (points, lines, bars). The plot may also contain statistical transformations of the data and is drawn on a specific coordinates system”
– Hadley Wickham (ggplot2 creator)

It may seem verbose and unwieldy, but the idea of building a plot on a layer-by-layer basis is very powerful.

You begin a plot by defining the dataset we will use.
Then, we specify aesthetics, namely (x,y) coordinates, colour, size, etc.
Finally, we choose what geom (or geometric shape) we want to use to represent our data.

We can then add more layers, like legends, labels, facets. etc.

In the following examples, we will use the gapminder dataset with data on life expectancy lifeExp, population pop, and GDP per capita gdpPerCap for a number of countries between 1952 and 2007. We want to build a graph that shows the relationship between GDP per capita and life expectancy.

As we said, first we define the dataset we are using

library(gapminder) #load the package gapminder that contains the data

ggplot(data=gapminder)

We just get an empty canvas, as we haven’t done much with our dataset.

The next thing is to map aesthetics. In our case, we will map gdpPercap to the x-axis, and lifeExp to the y-axis.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp))

This is an improvement over the blank canvas we got earlier, as we have mapped the x- and y- axes and we see the likely ranges of both variables. However, to see the scatter plot we want, we must add a geometry; as scatter plots are a bunch of points, the relevant geometry is geom_point().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp)) +
  geom_point()

What if we wanted to colour the points by the continent each country is in? This is a change of the aesthetic properties, so we just add colour = continent.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_point()

What if instead of a scatter plot we wanted to create a line plot? It would be the same code as before, but now the relevant geometry we should is geom_line insrtead of geom_point.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_line()

However, this is not a particularly useful plot, so let us go back to our scatter plot.

What if we wanted to have the size of each point correspond to the population of the country? This is not a geometry, but an aesthetic property. If we add size = pop, the points produced will be proportional to the country’s population, and we still have the aesthetic property colour = continent that will colour its point with the continent the country is in.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()

This is a more interesting graph, but given the non-linear pattern we see, we can perhaps improve it by taking the logarithm of the x-axis, GDP per capita. At the end of the commands, or layers, that make up our graph we add scale_x_log10(). This will take the logarithm of the values in the x-axis and should produce a scatterplot with a linear pattern.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10()

If we wanted to change the labels on the x-axis to dollars, we add labels = scales::dollar to the function scale_x_log10().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10(labels = scales::dollar)

Any graph should be properly labelled, and we can add labels by adding another layer: labs will add the relevant labels (title, subtitle, x- and y-axes, and a caption) as shown below.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  )+
  NULL

Style advice: While you can have the entire code for a ggplot in one single line, please don’t!
First, it makes it very hard to read and understand.
Secondly, you build a ggplot in layers; by having each layer in a separate line, you can easily comment out a line (just add a hashtag # at the beginning of the line) and see what is the effect of removing that layer.
What about the final NULL? Well, it’s there to ensure that no matter how many lines you comment out, you have no orphan +s and your code will run fine.

Finally, we can change the default theme which is a plot on a grey background; for this graph, we have chosen theme_minimal().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()+
  NULL

Your turn:
Try exprimenting with different themes.
1. Change theme_minimal() to theme_bw(). What’s the difference?
2. Now use theme_void() which is an even more minimal theme!

Let us revisit our simple scatter plot. Because we have too may data points, we can add alpha = 0.4 in geom_point() to make some of the points more transparent; alpha = 1 means solid colour and opaque data points, whereas lower values of alpha make some points more transparent.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop, 
    )) +
  geom_point(alpha = 0.4) +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Layers

ggplot create graphics in layers. Once you define your data and the aesthetics [(x,y) coordinates, colour, size, fill, etc.], you can then add add more layers in that you keep on ‘doing’ things to the data.

In essence, each geom layer specifies

A geom: the graphical object to be drawn (histogram, boxplot, density plot, etc.)
A stat: what “statistic” it is applied to
A position: how it is placed; identity, jitter, dodge, stack, fill

Unfortunately, due to an early design mistake I called these either stat_() or geom_(). A better decision would have been to call them layer_() functions: that’s a more accurate description because every layer involves a stat and a geom. – Hadley Wickham

Position adjustments are used, as the name says, to adjust the position of each geom. The following position adjustments and their defaults are shown below:

position_identity - default of most geoms– Doesn’t adjust position
position_jitter - default of geom_jitter. Adding random noise to a plot can sometimes make it easier to read. Jittering is particularly useful for small datasets with at least one discrete position.
position_dodge - default of geom_boxplot. Dodging preserves the vertical position of an geom while adjusting the horizontal position.
position_stack - default of geom_bar==geom_histogram and geom_area– it stacks bars on top of each other
position_fill - useful for geom_bar==geom_histogram and geom_area– stacks bars and standardises each stack to have constant height

Let us create a base plot of life expectancy, coloured by continent

life_exp_plot <- 
  ggplot(
    data = gapminder,
    mapping = aes(
      x = lifeExp,
      fill = continent)
  )

Nothing much happens, as we have just defined the base plot. Let us now plot a geom_histogram(), which uses position_fill as its default.

life_exp_plot + 
  geom_histogram()

Is this a useful graph? position_stack, the deafult for geom_histogram(), stacks bars on top of each other. Look at the bar that appears right after the 70 year life expectancy. Right at the bottomw, we have a a few observations from, followed by the blue European one, the green Asia, etc. all the way to the top where you see the few red observations that correspond to Africa.

We can improve on this by using position = "identity" that doesn’t adjust position. We also use alpha = 0.3 to make the bars more transparent. We also plot a density plot, a smoothed version of a histogram using geom_density; its default position is identity and both plots are equivalent.

life_exp_plot + 
  geom_histogram(
    position = "identity",
    alpha = 0.3
  )

life_exp_plot + 
  geom_density( alpha = 0.3)

If we again think what each layer specifies

A geom: density plot
A stat: density
A position: identity

What if we change the position and we use stack for the position layer?

life_exp_plot + 
  geom_histogram(
    position = "stack",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(alpha = 0.3)

Both plots are identical; in the second one, we didn’t specify what position should be, so ggplot used the default position for a histogram, which is position = stack.

Finallt, we can also use position = "fill" which stacks bars and standardises each stack to have constant height, or position = "dodge" (to separate each continent) for the position layer

life_exp_plot + 
  geom_histogram(
    position = "fill",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(
    position = "dodge",
    alpha = 0.5
  )

Facetting

One of the nice features of ggplot2 is a special technique called faceting that allows us to split one plot into multiple plots based on a factor included in the dataset. In the gapminder scatterplot example, we can use facetting and produce one scatter plot for each continent separately by using facet_wrap and facet_grid as shown below.

Before proceeding, we will define an object gapminder_scatterplot with the sequence of layers that gives us the ‘core’ life expectancy vs GDP scatterplot. Having stored the ‘core’ plot into an object, we can then add layers to it as needed, something which is useful for programming, as it saves you from retyping things.

facet_wrap() allows us to get the same graph, but looking at by changing another variable; in our case, we will look at the core scatterplot first by continent, and then by `year.

#define the core gapminder scatterplot of life expectancy vs GDP
# store it in an obect called `gapminder_scatterplot`
gapminder_scatterplot <-  
  ggplot(
    data = gapminder,
    mapping = aes(
      x = gdpPercap,
      y = lifeExp, 
      colour = continent), 
      alpha = 0.2) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita, 1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  ) +
  theme_minimal()


# We now add a new layer to our base plot: facet_wrap(~x), 
# where x is the variable you want to facet by

# first, facet the scatterplot by continent
gapminder_scatterplot +
  facet_wrap(~continent)

# then, facet the scatterplot by year
gapminder_scatterplot +
  facet_wrap(~year)

We can use the nrow argument to manually control the number of rows in the faceting. We will consider the faceting by continent plot and want to have the output in 3 rows, so nrow = 3. Also, we do not want any legends for the colours used, as ggplot will explicitly name the continents. To remove the legends, we add theme(legend.position="none") to our ggplot.

gapminder_scatterplot +
  facet_wrap(
    facets = vars(continent),
         nrow = 3) +
  theme(legend.position="none")

If you want to facet a plot and have its results appear in grid, we can use facet_grid(). You can define what the row and the columns in your grid should correspond to.

# use facet_grid(), where rows refer to continents
gapminder_scatterplot +
  facet_grid(vars(rows=continent)) +
  theme(legend.position="none")

# we can create a facet_grid where you can define *both* rows and columns
# in our scatterplot, we add a facet_grid() layer where columns = continents and rows =  year
gapminder_scatterplot+
  theme_minimal(8) + # just make the font size smaller
  facet_grid(
    cols = vars(continent), 
    rows = vars(year)
    ) +
  theme(legend.position="none")

Finally, if instead of a scatter plot we wanted to create a boxplot of life expectancy by continent, we use similar aesthetics, but the relevant geometry is geom_boxplot().

ggplot(
  data = gapminder,
  mapping = aes(
    x = continent,
    y = lifeExp, 
    fill = continent)) +
  geom_boxplot() +
  labs(title = "Life Expectancy among the continents, 1952-2007", 
       x = " ", # Empty, as the levels of the x-variable are the continets
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Tweaking graphics for publication quality

ggplot comes with many other options for tweaking plots to get them just the way you want for publication. These can be a bit hard to remember, but I usually look them up in R graphics cookbook and the BBC Visual and Data Journalism cookbook for R graphics, both of which have example code to cover most use cases!

In the example below, we select only those observations between 1997 and 2007, calculate the average life expectancy, average GDP per capita, and average population. We then create a new object, gapminder9707_plot which is the series of commands that make up our plot. To actually see the plot, we either use print(gapminder9707_plot) or just gapminder9707_plot.

gapminder9707 <- gapminder %>% 
  group_by(continent, country) %>%
  filter(year %in% c(1997, 2002, 2007)) %>%
  summarise(avg_life = mean(lifeExp, na.rm = TRUE),
            avg_gdp = mean(gdpPercap, na.rm = TRUE),
            avg_population_millions = mean(pop/1000000, na.rm = TRUE)) %>% 
  ungroup()          
            
gapminder9707_plot <- ggplot(data = gapminder9707,
       mapping = aes(x = avg_gdp,
                     y = avg_life,
                     colour = continent,
                     size = avg_population_millions,
                     label = country)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  labs(title = "Life Expectancy vs GDP per capita, 1997-2007", 
       x = "Average GDP per capita", 
       y = "Average Life Expectancy",
       caption = "Source: Gapminder") +
  geom_text(nudge_y = -.8, size = 2.2, check_overlap = TRUE)+
  theme(legend.position="none") 


gapminder9707_plot

Making plots interactive using `plotly`

We can make our plots interactive using the plotly package, which allows us to look at each point, zoon in/out, etc. Once you load the plotly library, it is simply a matter or using the ggplotly command. Move your cursor on the graph and see what happens!

plotly::ggplotly(gapminder9707_plot)

Animated Graphs

Animated graphs have recently become popular. The internet is full of tutorials and code-throughs where people explain how to do something interesting with R, so here is one if you wanted to know more about animations in R. You have to install the gganimate package and the animated graphs usually take some time to produce, as R needs to generates a number of GIF files and then create the animation, so please be patient!

Gapminder Animations - `transition_time()`

First we look at an animated boxplot of life expectancy by continent over time. The code to produce the plot is fairly straight-forward ggplot, but the last couple of lines ( transition_time(year) + ease_aes("linear")) are the ones that produce the animation.

library(gganimate)

boxplot_animation <- ggplot(data = gapminder,
       mapping = aes(x = continent,
                     y = lifeExp,
                     fill = continent)) +
  geom_boxplot() +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "Continent", 
       y = "Life Expectancy") +  
  transition_time(year) +
  ease_aes("linear")


animate(boxplot_animation, height=600, width = 600)

If we want to animate the evolution of the relationship between life expectancy and GDP, similar to Hans Rosling’s 200 Countries, 200 Years, 4 Minutes, we can use the code below

animation1 <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(animation1, height=600, width = 600)

Finally, instead of one scatter plot, if we wanted to facet our animation by continent, we just add the facet_wrap(~continent) line of code as shown below

faceted_animation <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  facet_wrap(~continent) +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(faceted_animation, height=800, width = 800)

Why you should always plot your data

We have touched on the basics of ggplot visualisations, but in this section we wanted to discuss why one should always plot the data and not just rely on tables of summary statistics.

Let us consider thirteen datasets all of which have 142 observations of (x,y) values. The table below shows the average value of X and Y, the standard deviation of X and Y, as well as the correlation coefficient between X and Y.

id	n	mean_x	mean_y	sd_x	sd_y	correlation
1	142	54.3	47.8	16.8	26.9	-0.064
2	142	54.3	47.8	16.8	26.9	-0.069
3	142	54.3	47.8	16.8	26.9	-0.068
4	142	54.3	47.8	16.8	26.9	-0.064
5	142	54.3	47.8	16.8	26.9	-0.060
6	142	54.3	47.8	16.8	26.9	-0.062
7	142	54.3	47.8	16.8	26.9	-0.069
8	142	54.3	47.8	16.8	26.9	-0.069
9	142	54.3	47.8	16.8	26.9	-0.069
10	142	54.3	47.8	16.8	26.9	-0.063
11	142	54.3	47.8	16.8	26.9	-0.069
12	142	54.3	47.8	16.8	26.9	-0.067
13	142	54.3	47.8	16.8	26.9	-0.066

Since our datasets contain values for X and Y, we can estimate 13 regression models and plot the values for each of the 13 intercepts and slope for X.

If we just looked at either the summary statistics table, or the plots of intercepts and slopes, we may be tempted to conclude that the 13 datasets are either identical or very much alike. However, this is far from the truth, as this is what the 13 individual datasets look like.

What data patterns can lie behind a correlation coefficient?

Jan Vanhove has written about the data patterns that can lie behind a correlation coefficient and why you should always plot and visualise a scatter plot; he has created a package, cannoball, where you specify a correlation coefficient r and a sample size n, and you get multiple scatterplots of the same correlation value, but fairly different in their scatter.

We will visualise 16 different datasets, all of which have a correlation of 0.50, and a sample of size n = 100.

RStudio’s primers for ggplot2

You can work through RStudio’s introductory primers for ggplot2; these are fairly short once you get used to the syntax of ggplot().

RStudios’s primers on visualising data

Further resources

Manipulate Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Select certain variables (or columns) in a dataframe with the dplyr function select dplyr::select()
2. Select certain cases (or rows) in a dataframe according to filtering conditions with the dplyr function filter dplyr::filter()
3. Pass the output of one dplyr function to the input of another function with the ‘pipe’ operator %>%
4. Create new variables (columns) in a dataframe that are functions of existing columns with dplyr::mutate()
5. Use dplyr::group_by(), dplyr::summarise(), and dplyr::count() to split a dataframe into groups of observations, calculate summary statistics for each group, and also count the number of total observations in each group

When working on a real project, data will seldom (if ever!) arrive in exactly the format you would like to have it in in order to analyse it. We need to manipulate and transform data and just as we have a grammar for generating graphics (the layered grammar of graphics in ggplot), we also have a syntax for data transformation.

dplyr is a package that contains useful functions for transforming and manipulating data frames. You can think of these functions as verbs, that do something to the data. All of the dplyr verbs (or functions), and in fact pretty much everything in the tidyverse, works in the following fashion:

The first argument is a data frame
Subsequent arguments describe what to do with the data frame
The result is a new data frame

The `pipe` operator, or `%>%`

The pipe operator, this strange %>% thing, takes the value to the left of it and passes it through to the thing to the right of it. Let us create a couple of lists and a simple function to see an example

# A list (or vector) of multiple values too:
my_first_list <- c(1, 2, 3, 5, 8, 13, 21, 34, 55, 89)
my_second_list <- c(1, 1, 2, 3, 5, 8, 13, 21, 34, 55)

# Define a function that takes X and adds 100
my_function <- function(x) {
  new_x <- x + 100
  return(new_x)
}

Functions work on single values and on lists (or vectors):

# call my_function with x=14 as an argument
my_function(14)

## [1] 114

# call my_function with x=my_first_list as an argument; this is a 
# vectorised operation, as it will add 100 to each value in my_first_list
my_function(my_first_list)

##  [1] 101 102 103 105 108 113 121 134 155 189

# call my_function with my_first_list+my_second_list as argument; this is a 
# vectorised operation, as it will first add my_first_list+my_second_list 
# and then add 100 to each value 
my_function(my_first_list+my_second_list)

##  [1] 102 103 105 108 113 121 134 155 189 244

We can nest functions inside each other and use mean(my_function(my_first_list))

mean(my_function(my_first_list))

## [1] 123

But this can get really hard to read, since you have to read from the inside out. In English, this nested mess reads “Calculate the mean of the results of my_function applied to my_first_list.” We can simplify this by reversing the nested chain and using the pipe operator

my_first_list %>% 
  my_function() %>% 
  mean()

## [1] 123

Here we start with the data and then describe the actions/verbs to do something to the data. We can read this chain as "Take my_first_list, pass it through my_function, and calculate the mean of that.

The %>% is called a pipe and you can also read or think of the pipe operator as the words “and then.” There’s also a keyboard shortcut for this too, since typing %>% all the time can be tedious: In Windows you would use Ctrl + Shift + M and in Mac wou would use ⌘ /Cmd+ shift + M

Similarly, we frequently need to perform a series of intermediate steps to transform data for analysis. If we write each step as a discrete command and store their contents as new objects, our code becomes difficult to read and understand.

When speaking or writing, we never start with a sentence with a verb, but rather with a noun (subject). It is good practice to start with a dataframe/object and then use verbs (or functions) to describe what you want to do.

Suppose we wanted to look at the first few rows of life expectancy values, using the head() function, of the gapminder dataframe.

# Nested command, rather hard to read, since we read from the inside out
head(select(gapminder,lifeExp))

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

# using the pipe operator: Start with gapminder, and then
gapminder %>% 
  
  # select the column (or variable) lifeExp, and then 
  select(lifeExp) %>% 
  
  # use the head() function to return the first few rows of the dataset 
  head()

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

Key functions in `dplyr`

There are 6 important verbs that you’ll typically use when working with data:

Extract columns/variables with select()
Extract rows/cases with filter()
Arrange/sort rows with arrange()
Make new columns/variables with mutate()
Make group summaries with group_by %>% summarise()

`function()`	Action performed
`select()`	Selects a subset of columns (or variables) from the data frame
`filter()`	Subsets observations based on their values
`arrange()`	Changes the order of observations based on their values
`mutate()`	Creates new columns (or variables)
`group_by()`	Changes the unit of analysis from the complete dataset to individual groups of columns
`summarise()`	Collapses the data frame to a smaller number of rows which summarise the larger data

Every dplyr verb follows the same pattern. The first argument is always a data frame, and the function always returns a data frame:

VERB(DATA_TO_TRANSFORM, STUFF_IT_DOES)

Pick columns with `select()`

If we want to select ], or drop, specific columns from a tibble, we use the select() verb. For instance, if we wanted to keep only the lifeExp and year columns:

gapminder %>% select(lifeExp, year)


## # A tibble: 1,704 x 2
##    lifeExp  year
##      <dbl> <int>
##  1    28.8  1952
##  2    30.3  1957
##  3    32.0  1962
##  4    34.0  1967
##  5    36.1  1972
##  6    38.4  1977
##  7    39.9  1982
##  8    40.8  1987
##  9    41.7  1992
## 10    41.8  1997
## # ... with 1,694 more rows

You can remove specific columns by prefacing the column names with a minus sign -. SO to drop -lifeExp from our tibble, we would use:

gapminder %>% select(-lifeExp)


## # A tibble: 1,704 x 5
##    country     continent  year      pop gdpPercap
##    <fct>       <fct>     <int>    <int>     <dbl>
##  1 Afghanistan Asia       1952  8425333      779.
##  2 Afghanistan Asia       1957  9240934      821.
##  3 Afghanistan Asia       1962 10267083      853.
##  4 Afghanistan Asia       1967 11537966      836.
##  5 Afghanistan Asia       1972 13079460      740.
##  6 Afghanistan Asia       1977 14880372      786.
##  7 Afghanistan Asia       1982 12881816      978.
##  8 Afghanistan Asia       1987 13867957      852.
##  9 Afghanistan Asia       1992 16317921      649.
## 10 Afghanistan Asia       1997 22227415      635.
## # ... with 1,694 more rows

You can also rename columns using select(), using the syntax select(new_name = old_name).

gapminder %>% select(year, country, life_expectancy = lifeExp)


## # A tibble: 1,704 x 3
##     year country     life_expectancy
##    <int> <fct>                 <dbl>
##  1  1952 Afghanistan            28.8
##  2  1957 Afghanistan            30.3
##  3  1962 Afghanistan            32.0
##  4  1967 Afghanistan            34.0
##  5  1972 Afghanistan            36.1
##  6  1977 Afghanistan            38.4
##  7  1982 Afghanistan            39.9
##  8  1987 Afghanistan            40.8
##  9  1992 Afghanistan            41.7
## 10  1997 Afghanistan            41.8
## # ... with 1,694 more rows

Alternatively, there’s a special rename() verb with the same syntax, i.e., rename(new_name = old_name) that will rename a column to a new name, while keeping all the other columns:

gapminder %>% rename(life_expectancy = lifeExp)


## # A tibble: 1,704 x 6
##    country     continent  year life_expectancy      pop gdpPercap
##    <fct>       <fct>     <int>           <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952            28.8  8425333      779.
##  2 Afghanistan Asia       1957            30.3  9240934      821.
##  3 Afghanistan Asia       1962            32.0 10267083      853.
##  4 Afghanistan Asia       1967            34.0 11537966      836.
##  5 Afghanistan Asia       1972            36.1 13079460      740.
##  6 Afghanistan Asia       1977            38.4 14880372      786.
##  7 Afghanistan Asia       1982            39.9 12881816      978.
##  8 Afghanistan Asia       1987            40.8 13867957      852.
##  9 Afghanistan Asia       1992            41.7 16317921      649.
## 10 Afghanistan Asia       1997            41.8 22227415      635.
## # ... with 1,694 more rows

Pick rows with `filter()`

The filter() function takes two arguments: a tibble to transform, and a set of tests. It will return each row for which the test is TRUE.

This code, for instance, will look at the gapminder dataset and return all rows where country is equal to “Jordan”.

filter(gapminder, country == "Jordan")


## # A tibble: 12 x 6
##    country continent  year lifeExp     pop gdpPercap
##    <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Jordan  Asia       1952    43.2  607914     1547.
##  2 Jordan  Asia       1957    45.7  746559     1886.
##  3 Jordan  Asia       1962    48.1  933559     2348.
##  4 Jordan  Asia       1967    51.6 1255058     2742.
##  5 Jordan  Asia       1972    56.5 1613551     2111.
##  6 Jordan  Asia       1977    61.1 1937652     2852.
##  7 Jordan  Asia       1982    63.7 2347031     4161.
##  8 Jordan  Asia       1987    65.9 2820042     4449.
##  9 Jordan  Asia       1992    68.0 3867409     3432.
## 10 Jordan  Asia       1997    69.8 4526235     3645.
## 11 Jordan  Asia       2002    71.3 5307470     3845.
## 12 Jordan  Asia       2007    72.5 6053193     4519.

Notice that there are two equal signs (==). Please note that when testing for equality, we use a double equal sign, (==). If you had used a single equal sign, that would be the assignment operator, i.e., you set an argument (like data = gapminder); when you use two equal signs, you are running a logical a test.

Test	Meaning
`x < y`	Less than
`x > y`	Greater than
`x == y`	Equal to
`x <= y`	Less than or equal to
`x >= y`	Greater than or equal to
`x != y`	Not equal to
`x %in% y`	In (group membership)
`is.na(x)`	Is missing
`!is.na(x)`	Is not missing

Your turn: Use filter() and logical tests to show:
1. The data for China
2. All data for countries in Africa
3. All cases (rows) where life expectancy is greater than 80

You can also use multiple conditions, and these will extract rows that meet every test. By default, if you separate the tests with a comma, R will consider this an “and” test and find rows that are both Jordan and greater than 2000.

filter(gapminder, country == "Jordan", year > 2000)


## # A tibble: 2 x 6
##   country continent  year lifeExp     pop gdpPercap
##   <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
## 1 Jordan  Asia       2002    71.3 5307470     3845.
## 2 Jordan  Asia       2007    72.5 6053193     4519.

If you have any programming experience, you can also use the common operators for “and” with “&”, “or” with “|”, and “not” with “!”:

Operator	Meaning
`a & b`	and
`a \| b`	or
`!a`	not

Your turn: Use filter() and logical tests to show:
1. India before 1970
2. Countries where life expectancy in 2007 is below 60
3. Countries where life expectancy in 2007 is below 60 and are not in Africa

Beware of some common mistakes! You can’t collapse multiple tests into one. Instead, use two separate tests:

# This won't work!
filter(gapminder, 1960 < year < 1980)

# This will work
filter(gapminder, 1960 < year, year < 1980)

Also, you can avoid stringing together lots of tests by using the %in% operator, which checks to see if a value is in a list of values.

# This works, but is tedious-- what if you wanted to pick a dozen countries?
filter(gapminder, 
       country == "Mexico" |  country == "United States" | country == "Canada" )

# This is more concise and easier to add other countries later
filter(gapminder, 
       country %in% c("Mexico", "United States", "Canada" ))

Sort data with `arrange()`

The arrange() verb sorts data. By default it sorts in ascending order, from minimum to maximum value:

gapminder %>% arrange(lifeExp)


## # A tibble: 1,704 x 6
##    country      continent  year lifeExp     pop gdpPercap
##    <fct>        <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Rwanda       Africa     1992    23.6 7290203      737.
##  2 Afghanistan  Asia       1952    28.8 8425333      779.
##  3 Gambia       Africa     1952    30    284320      485.
##  4 Angola       Africa     1952    30.0 4232095     3521.
##  5 Sierra Leone Africa     1952    30.3 2143249      880.
##  6 Afghanistan  Asia       1957    30.3 9240934      821.
##  7 Cambodia     Asia       1977    31.2 6978607      525.
##  8 Mozambique   Africa     1952    31.3 6446316      469.
##  9 Sierra Leone Africa     1957    31.6 2295678     1004.
## 10 Burkina Faso Africa     1952    32.0 4469979      543.
## # ... with 1,694 more rows

You can sort in descending order (max to min) by using the desc() for the column/variable you want sorted:

gapminder %>% arrange(desc(lifeExp))


## # A tibble: 1,704 x 6
##    country          continent  year lifeExp       pop gdpPercap
##    <fct>            <fct>     <int>   <dbl>     <int>     <dbl>
##  1 Japan            Asia       2007    82.6 127467972    31656.
##  2 Hong Kong, China Asia       2007    82.2   6980412    39725.
##  3 Japan            Asia       2002    82   127065841    28605.
##  4 Iceland          Europe     2007    81.8    301931    36181.
##  5 Switzerland      Europe     2007    81.7   7554661    37506.
##  6 Hong Kong, China Asia       2002    81.5   6762476    30209.
##  7 Australia        Oceania    2007    81.2  20434176    34435.
##  8 Spain            Europe     2007    80.9  40448191    28821.
##  9 Sweden           Europe     2007    80.9   9031088    33860.
## 10 Israel           Asia       2007    80.7   6426679    25523.
## # ... with 1,694 more rows

You can sort by multiple columns by specifying them in a comma separated list. For example, we can sort by continent first and then sort by lifeExp (life expectancy) in descending order within each continent:

gapminder %>% 
  arrange(continent, desc(lifeExp))


## # A tibble: 1,704 x 6
##    country   continent  year lifeExp      pop gdpPercap
##    <fct>     <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Reunion   Africa     2007    76.4   798094     7670.
##  2 Reunion   Africa     2002    75.7   743981     6316.
##  3 Reunion   Africa     1997    74.8   684810     6072.
##  4 Libya     Africa     2007    74.0  6036914    12057.
##  5 Tunisia   Africa     2007    73.9 10276158     7093.
##  6 Reunion   Africa     1992    73.6   622191     6101.
##  7 Tunisia   Africa     2002    73.0  9770575     5723.
##  8 Mauritius Africa     2007    72.8  1250882    10957.
##  9 Libya     Africa     2002    72.7  5368585     9535.
## 10 Algeria   Africa     2007    72.3 33333216     6223.
## # ... with 1,694 more rows

Add new columns with `mutate()`

You create new columns, or variables, with the mutate() function. You can create a single new column of gdp in the gapminder tibble as follows:

mutate(gapminder, gdp = gdpPercap * pop)


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap          gdp
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.
## # ... with 1,694 more rows

And you can create multiple columns by including a comma-separated list of new columns to create:

mutate(gapminder, gdp = gdpPercap * pop,
                  pop_mill = round(pop / 1000000))


## # A tibble: 1,704 x 8
##    country     continent  year lifeExp      pop gdpPercap          gdp pop_mill
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>    <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.        8
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.        9
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.       10
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.       12
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.       13
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.       15
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.       13
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.       14
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.       16
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.       22
## # ... with 1,694 more rows

You can also run logical, conditional tests within mutate() using the ifelse() function. This works like the =IF function in Excel and it takes three arguments:

a logical test,
what happens if the test is true, and
what happens if the test is false:

ifelse(TEST, VALUE_IF_TRUE, VALUE_IF_FALSE)

We can create a new column that is a binary (TRUE/FALSE) indicator for whether year is after 1960:

mutate(gapminder, after_1960 = ifelse(year > 1960, TRUE, FALSE))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <lgl>     
##  1 Afghanistan Asia       1952    28.8  8425333      779. FALSE     
##  2 Afghanistan Asia       1957    30.3  9240934      821. FALSE     
##  3 Afghanistan Asia       1962    32.0 10267083      853. TRUE      
##  4 Afghanistan Asia       1967    34.0 11537966      836. TRUE      
##  5 Afghanistan Asia       1972    36.1 13079460      740. TRUE      
##  6 Afghanistan Asia       1977    38.4 14880372      786. TRUE      
##  7 Afghanistan Asia       1982    39.9 12881816      978. TRUE      
##  8 Afghanistan Asia       1987    40.8 13867957      852. TRUE      
##  9 Afghanistan Asia       1992    41.7 16317921      649. TRUE      
## 10 Afghanistan Asia       1997    41.8 22227415      635. TRUE      
## # ... with 1,694 more rows

We can also use text labels instead of TRUE and FALSE:

mutate(gapminder, 
       after_1960 = ifelse(year > 1960, "After 1960", "Before 1960"))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960 
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <chr>      
##  1 Afghanistan Asia       1952    28.8  8425333      779. Before 1960
##  2 Afghanistan Asia       1957    30.3  9240934      821. Before 1960
##  3 Afghanistan Asia       1962    32.0 10267083      853. After 1960 
##  4 Afghanistan Asia       1967    34.0 11537966      836. After 1960 
##  5 Afghanistan Asia       1972    36.1 13079460      740. After 1960 
##  6 Afghanistan Asia       1977    38.4 14880372      786. After 1960 
##  7 Afghanistan Asia       1982    39.9 12881816      978. After 1960 
##  8 Afghanistan Asia       1987    40.8 13867957      852. After 1960 
##  9 Afghanistan Asia       1992    41.7 16317921      649. After 1960 
## 10 Afghanistan Asia       1997    41.8 22227415      635. After 1960 
## # ... with 1,694 more rows

Your turn: Use mutate() to:
1. Add an africa column that is TRUE if the country is on the African continent
2. Add a column log_GDP for the logarithm of GDP per capita, using log(gdpPercap)
3. Add an africa_asia column that says “Africa or Asia” if the country is in Africa or Asia, and “Not Africa or Asia” if it’s not

Combine multiple verbs with pipes (`%>%`)

What if you want to include only rows from 2002 and make a new column with the logged GDP per capita? Doing this requires both filter() and mutate(), so we need to find a way to use both at once.

One solution is to use intermediate data frames for each step:

gapminder_2002_filtered <- filter(gapminder, year == 2002)

gapminder_2002_logged <- mutate(gapminder_2002_filtered, log_gdpPercap = log(gdpPercap))

That works fine, but your environment panel will start getting full of lots of intermediate data frames.

Another solution is to nest the functions inside each other. Remember that all dplyr functions return data frames, so you can feed the results of one into another:

filter(mutate(gapminder, log_gdpPercap = log(gdpPercap)), 
       year == 2002)

That works too, but it gets really complicated once you have even more functions, and it’s hard to keep track of which function’s arguments go where. I’d avoid doing this entirely.

One really nice solution is to use the pipe operator, or %>%. The pipe takes an object on the left and passes it as the first argument of the function on the right.

# gapminder will automatically get placed in the _____ spot
gapminder %>% filter(_____, country == "Jordan")

These two lines of code do the same thing:

filter(gapminder, country == "Jordan")

gapminder %>% filter(country == "Jordan")

Using pipes, you always start with a data frame, pass it to one verb to do one thing, then pass the output of that verb (a dataframe) to the next verb that will do something else, and so on. When reading any code with a %>%, it’s easiest to read the %>% as “and then”. This would read:

Take the gapminder dataset and then filter it so that it only has rows from 2002 and then add a new column (mutate) with the logged GDP per capita

gapminder %>% 
  filter(year == 2002) %>% 
  mutate(log_gdpPercap = log(gdpPercap))

Summarise data by groups with `group_by() %>% summarise()`

The summarise() verb takes an entire frame and collapses all of the rows in a single number as it calculates summary information about it. For instance, the following code will start with the entire gapminder data, calculate average life expectancy, and return just a single value, namely avarage life expectnacy among all countries and all years :

gapminder %>% summarize(mean_life = mean(lifeExp))


## # A tibble: 1 x 1
##   mean_life
##       <dbl>
## 1      59.5

You can also make multiple summary variables, just like mutate(), and it will return a column for each:

gapminder %>% summarize(mean_life = mean(lifeExp),
                        sd_life = sd(lifeExp),
                        min_life = min(lifeExp),
                        max_life = max(lifeExp)
                        )


## # A tibble: 1 x 4
##   mean_life sd_life min_life max_life
##       <dbl>   <dbl>    <dbl>    <dbl>
## 1      59.5    12.9     23.6     82.6

Your turn: Use summarise() to calculate:
1. The first (minimum) year in the gapminder dataset
2. The last (maximum) year in the dataset
3. The number of rows in the dataset (use the dplyr cheatsheet)
4. The number of distinct countries in the dataset (use the dplyr cheatsheet)
5. Use filter() and summarise() to calculate the median, minimum, and maximum life expectancy on the African continent in 2007

Again, remember that summarise() on its own summarises the entire dataset, so you only get numbers in a single row. These values can be what you want, e.g., averages, standard deviations, and min/max values for the entire dataset. If you group your data into separate subgroups with group_by(), you can use summarise() to calculate summary statistics for each group.

The group_by() function puts rows into groups based on values in a column. If you run:

gapminder %>% group_by(continent)


## # A tibble: 1,704 x 6
## # Groups:   continent [5]
##    country     continent  year lifeExp      pop gdpPercap
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.
##  2 Afghanistan Asia       1957    30.3  9240934      821.
##  3 Afghanistan Asia       1962    32.0 10267083      853.
##  4 Afghanistan Asia       1967    34.0 11537966      836.
##  5 Afghanistan Asia       1972    36.1 13079460      740.
##  6 Afghanistan Asia       1977    38.4 14880372      786.
##  7 Afghanistan Asia       1982    39.9 12881816      978.
##  8 Afghanistan Asia       1987    40.8 13867957      852.
##  9 Afghanistan Asia       1992    41.7 16317921      649.
## 10 Afghanistan Asia       1997    41.8 22227415      635.
## # ... with 1,694 more rows

…you won’t see anything different! R has put the dataset into separate invisible groups behind the scenes, but you haven’t done anything with those groups, so nothing has really happened. If you do things with those groups with summarise(), though, group_by() becomes much more useful.

For instance, this will take the gapminder data frame, group it by continent, and then summarize it by calculating the number of distinct countries in each group. It will return one row for each group, so there should be a row for each continent:

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country))

## # A tibble: 5 x 2
##   continent n_countries
##   <fct>           <int>
## 1 Africa             52
## 2 Americas           25
## 3 Asia               33
## 4 Europe             30
## 5 Oceania             2

You can calculate multiple summary statistics, as before:

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country),
            avg_life_exp = mean(lifeExp))

## # A tibble: 5 x 3
##   continent n_countries avg_life_exp
##   <fct>           <int>        <dbl>
## 1 Africa             52         48.9
## 2 Americas           25         64.7
## 3 Asia               33         60.1
## 4 Europe             30         71.9
## 5 Oceania             2         74.3

Your turn:
1. Calculate summary statistics for life expectancy for each continent. Calculate minimum, maximum, median, mean, and standard deviation, and total count (n)
2. Do the same, but for the year 2007 only

Finally, you can group by multiple columns and R will create subgroups for every combination of the groups and return the number of rows of combinations. For instance, we can calculate the average life expectancy by both year and continent and we’ll get 60 rows, since there are 5 continents and 12 years (5 × 12 = 60):

gapminder %>% 
  group_by(continent, year) %>% 
  summarize(avg_life_exp = mean(lifeExp))

## # A tibble: 60 x 3
## # Groups:   continent [5]
##    continent  year avg_life_exp
##    <fct>     <int>        <dbl>
##  1 Africa     1952         39.1
##  2 Africa     1957         41.3
##  3 Africa     1962         43.3
##  4 Africa     1967         45.3
##  5 Africa     1972         47.5
##  6 Africa     1977         49.6
##  7 Africa     1982         51.6
##  8 Africa     1987         53.3
##  9 Africa     1992         53.6
## 10 Africa     1997         53.6
## # ... with 50 more rows

A common mistake I have seen is that people use the summarise() function before any group_by(). Rememebr that if you summarise() first, you collapse the entire dataframe into a single row, so there is no group_by() that can be done on a single row of data!!

Further resources

dplyr and tidyr cheat sheet for examples.

Reshape Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Understand the concept of a wide and a long table format and for which purpose those formats are useful.
2. Understand what key-value pairs are.
3. Reshape a dataframe from long to wide format and back with the tidyr::pivot_longer() and tidyr::pivot_wider() commands.

Overview

It is often said that the vast majority of data analysis is spent on cleaning and preparing data. This is something that must be repeated many times over the course of analysis as new problems come to light or new data is collected.

Most people are used to analyze data in a spreadsheet or tabular format. For instance, if we wanted to study climate change, we can find data on the Combined Land-Surface Air and Sea-Surface Water Temperature Anomalies in the Northern Hemisphere at NASA’s Goddard Institute for Space Studies. The tabular data of temperature anomalies can be found here and part of that data set is shown below:

A lot of these tabular shape spreadsheets were designed for efficient data entry and not necessarily to undertake any kind of statistical analysis. The principles of tidy data provide a standard way to organise data values within a dataset. A standard makes initial data cleaning easier because you don’t need to start from scratch and reinvent the wheel every time.

Tidy data is a specific way of organising data in a consistent manner and structuring datasets to facilitate analysis with the tidyverse. The tidy data standard has been designed to facilitate exploratory data analysis; tidy datasets and tidy tools help make data analysis easier, allowing you to focus on the interesting domain problem, not on the logistics of cleaning data.

Before we proceed, a few definitions taken from Garret Grolemund and the vignette(“tidy-data”)

Variable: A quantity, quality, or property that you can measure.
Observation: A set of values that display the relationship between variables. To be an observation, values need to be measured under similar conditions, usually measured on the same observational unit at the same time.
Value: The state of a variable that you observe when you measure it.

There are three rules which make a dataset tidy:

Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

Figure 12.1 from R for Data Science

A tidy dataset is a long dataset, where each variable appears in one column, and each observation has its own row. The weather anomalies dataset is a wide dataset; the three variables are date (or year and month if you wanted to keep them separate), and delta (the actual temperature difference).

We will often need to reshape our datasets and should have a way to go:

from wide format to long (tidy) format using tidyr::gather() or tidyr::pivot_longer()
from long (tidy) to wide format using tidyr::spread() or tidyr::pivot_wider()

In a set of wonderful animations from Garrick Aden-Buie, this is the process of coverting from long format to wide and back

Let us review the basic tasks for tidying data using the R for Data Science gapminder subset.

table1

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

Note that in this data frame, each variable is in its own column (country, year, cases, and population), each observation is in its own row (i.e. each row is a different country-year pairing), and each value has its own cell.

`pivot_longer` or `gather` data

Gathering entails bringing a variable spread across multiple columns into a single column. For example, this version of table1 is not tidy because the year variable is in wide format, spread across multiple columns:

table4a

## # A tibble: 3 x 3
##   country     `1999` `2000`
## * <chr>        <int>  <int>
## 1 Afghanistan    745   2666
## 2 Brazil       37737  80488
## 3 China       212258 213766

The variables (columns) that a tidy dataframe would have would be country, year, and cases. We can use the pivot_longer or gather() function from the tidyr package to reshape the data frame and make this tidy. To do this we need three pieces of information:

The names of the columns that represent the values, not variables. Here, those are 1999 and 2000.
The key, or the name of the variable whose values form the column names. Here that is year.
The value, or the name of the variable whose values are spread over the cells. Here that is cases.

Notice that we create the names for key and value - they do not already exist in the data frame.

We implement this using the pivot_longer() or gather() function. pivot_longer() requires the newest version of tidyr

Once you have installed the newest version of tidyr, then you can use either pivot_longer() or gather()

table4a %>% 
  pivot_longer(cols=c(`1999`, `2000`), 
               names_to = "year", 
               values_to = "cases")

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Afghanistan 2000    2666
## 3 Brazil      1999   37737
## 4 Brazil      2000   80488
## 5 China       1999  212258
## 6 China       2000  213766

table4a %>% 
  gather(`1999`, `2000`, 
         key = year, 
         value = cases)

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Brazil      1999   37737
## 3 China       1999  212258
## 4 Afghanistan 2000    2666
## 5 Brazil      2000   80488
## 6 China       2000  213766

This operation would be called reshaping data wide to long.

`pivot_wider` or `spread` data

If we wanted to make a long table into a wide one, we use pivot_wider; spreading brings an observation spread across multiple rows into a single row. It is the reverse of gathering, or taking a wide dataset and making it long. For instance, take table2:

table2

## # A tibble: 12 x 4
##    country      year type            count
##    <chr>       <int> <chr>           <int>
##  1 Afghanistan  1999 cases             745
##  2 Afghanistan  1999 population   19987071
##  3 Afghanistan  2000 cases            2666
##  4 Afghanistan  2000 population   20595360
##  5 Brazil       1999 cases           37737
##  6 Brazil       1999 population  172006362
##  7 Brazil       2000 cases           80488
##  8 Brazil       2000 population  174504898
##  9 China        1999 cases          212258
## 10 China        1999 population 1272915272
## 11 China        2000 cases          213766
## 12 China        2000 population 1280428583

It violates the tidy data principle because each observation (unit of analysis is a country-year pairing) is split across multiple rows. To tidy the data frame, we need to know:

The key column, or the column that contains variable names. Here, it is type.
The value column, or the column that contains values for multiple variables. Here it is count.

Notice that unlike for gathering, when spreading the key and value columns are already defined in the data frame. We do not create the names ourselves, only identify them in the existing data frame.

table2 %>%
  pivot_wider(names_from = type, values_from = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

table2 %>%
  spread(key = type, value = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

This operation would be called reshaping data long to wide.

Separating

Separating splits multiple variables stored in a single column into multiple columns. For example in table3, the rate column contains both cases and population:

table3

## # A tibble: 6 x 3
##   country      year rate             
## * <chr>       <int> <chr>            
## 1 Afghanistan  1999 745/19987071     
## 2 Afghanistan  2000 2666/20595360    
## 3 Brazil       1999 37737/172006362  
## 4 Brazil       2000 80488/174504898  
## 5 China        1999 212258/1272915272
## 6 China        2000 213766/1280428583

This is a bad idea as you lose information. Tidy data principles require each column to contain a single variable. We can use separate() to split the column into two new columns:

table3 %>% 
  separate(rate, into = c("cases", "population"))

## # A tibble: 6 x 4
##   country      year cases  population
##   <chr>       <int> <chr>  <chr>     
## 1 Afghanistan  1999 745    19987071  
## 2 Afghanistan  2000 2666   20595360  
## 3 Brazil       1999 37737  172006362 
## 4 Brazil       2000 80488  174504898 
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

Uniting

Uniting is the inverse of separating - when a variable is stored in multiple columns, uniting brings the variable back into a single column. table5 splits the year variable into two columns:

table5

## # A tibble: 6 x 4
##   country     century year  rate             
## * <chr>       <chr>   <chr> <chr>            
## 1 Afghanistan 19      99    745/19987071     
## 2 Afghanistan 20      00    2666/20595360    
## 3 Brazil      19      99    37737/172006362  
## 4 Brazil      20      00    80488/174504898  
## 5 China       19      99    212258/1272915272
## 6 China       20      00    213766/1280428583

To bring them back, use the unite() function:

table5 %>% 
  unite(new, century, year)

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 19_99 745/19987071     
## 2 Afghanistan 20_00 2666/20595360    
## 3 Brazil      19_99 37737/172006362  
## 4 Brazil      20_00 80488/174504898  
## 5 China       19_99 212258/1272915272
## 6 China       20_00 213766/1280428583

# remove underscore
table5 %>% 
  unite(new, century, year, sep = "")

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 1999  745/19987071     
## 2 Afghanistan 2000  2666/20595360    
## 3 Brazil      1999  37737/172006362  
## 4 Brazil      2000  80488/174504898  
## 5 China       1999  212258/1272915272
## 6 China       2000  213766/1280428583

If we wanted to make gapminder a tabular, wide dataframe, we would use pivot_wider()

gapminder_life_exp_wide  <- gapminder %>% 
  select(country, continent, 
         lifeExp, year) %>% 
  pivot_wider(names_from = year, values_from = lifeExp) 


  gapminder_life_exp_wide

## # A tibble: 142 x 14
##    country continent `1952` `1957` `1962` `1967` `1972` `1977` `1982` `1987`
##    <fct>   <fct>      <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
##  1 Afghan~ Asia        28.8   30.3   32.0   34.0   36.1   38.4   39.9   40.8
##  2 Albania Europe      55.2   59.3   64.8   66.2   67.7   68.9   70.4   72  
##  3 Algeria Africa      43.1   45.7   48.3   51.4   54.5   58.0   61.4   65.8
##  4 Angola  Africa      30.0   32.0   34     36.0   37.9   39.5   39.9   39.9
##  5 Argent~ Americas    62.5   64.4   65.1   65.6   67.1   68.5   69.9   70.8
##  6 Austra~ Oceania     69.1   70.3   70.9   71.1   71.9   73.5   74.7   76.3
##  7 Austria Europe      66.8   67.5   69.5   70.1   70.6   72.2   73.2   74.9
##  8 Bahrain Asia        50.9   53.8   56.9   59.9   63.3   65.6   69.1   70.8
##  9 Bangla~ Asia        37.5   39.3   41.2   43.5   45.3   46.9   50.0   52.8
## 10 Belgium Europe      68     69.2   70.2   70.9   71.4   72.8   73.9   75.4
## # ... with 132 more rows, and 4 more variables: `1992` <dbl>, `1997` <dbl>,
## #   `2002` <dbl>, `2007` <dbl>

Similarly, if we wanted to convert from the wide gapminder to the long one, we would use either gather or pivot_longer()

gapminder_life_exp_wide %>% 
  gather(key = "year", value = "lifeExp",
         -country, -continent) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Albania     Europe     1952    55.2
##  3 Algeria     Africa     1952    43.1
##  4 Angola      Africa     1952    30.0
##  5 Argentina   Americas   1952    62.5
##  6 Australia   Oceania    1952    69.1
##  7 Austria     Europe     1952    66.8
##  8 Bahrain     Asia       1952    50.9
##  9 Bangladesh  Asia       1952    37.5
## 10 Belgium     Europe     1952    68  
## # ... with 1,694 more rows

gapminder_life_exp_wide %>% 
  pivot_longer(
    cols = c(-country, -continent), #keep country and continent
    names_to = "year", 
    values_to = "lifeExp",
         ) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Afghanistan Asia       1957    30.3
##  3 Afghanistan Asia       1962    32.0
##  4 Afghanistan Asia       1967    34.0
##  5 Afghanistan Asia       1972    36.1
##  6 Afghanistan Asia       1977    38.4
##  7 Afghanistan Asia       1982    39.9
##  8 Afghanistan Asia       1987    40.8
##  9 Afghanistan Asia       1992    41.7
## 10 Afghanistan Asia       1997    41.8
## # ... with 1,694 more rows

RStudio primer on tidyr

Reshape Data

Recent versions of tidyr have renamed these core functions: gather() is now pivot_longer() and spread() is now pivot_wider(). The syntax for these pivot_*() functions is slightly different from what it was in gather() and spread(), so you can’t just replace the names. Even though, both gather() and spread() still work and won’t go away for a while, I think it’s worth learning the newer pivot_*() functions.

More resources

Pivoting data from columns to rows (and back!) in the tidyverse
Pivoting in tidyr
Hadley Wickham’s tidy data paper
Data Organization in Spreadsheets

Side-by-side regression tables

Tue, 28 Jul 2020 00:00:00 +0000

huxtable::huxreg()

It’s often helpful to put the results from regression models in a side-by-side table so you can compare coefficients across different model specifications. If you’re unfamiliar with these kinds of tables, check out this helpful guide to how to read them.

We will use the huxreg() function in the huxtable package, the Palmer Penguins dataset, and a few regression models on trying to explain body_mass_g.

library(palmerpenguins)

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)
model2 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm, data = penguins)
model3 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species, data = penguins)
model4 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species + sex, data = penguins)

`huxtable::huxreg()`

Installing

huxtable is published on CRAN, so use the “Packages” panel in RStudio to install huxtable, or use:

install.packages("huxtable", dependencies = TRUE)

To knit to Word, you also need the flextable package, so install that too from the “Packages” panel, or use:

install.packages("flextable", dependencies = TRUE)

Using huxtable::huxreg() to knit to HTML and Word works very well.

Basic usage

As you go about builiding your linear models, you save them as object with different names, e.g., model1, model2, etc. Once you have the models you want to compate, pass them to huxreg():

library(huxtable)

huxreg(model1, model2, model3, model4)

	(1)	(2)	(3)	(4)
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

You can add column names, to make the output table more user friendly.

huxreg(list("A" = model1, "B" = model2, "C" = model3, "D" = model4))

	A	B	C	D
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

Statistics to display, bold significant variables, add captions

You can choose what statistics to display; for instance, if we wanted to show number of observatiosn, R^2 and adjusted R2, and the residual SE, we pass them to the statistics variable. We can also bold those variables that are significant at, say 0.05, and choose not to use stars to denote level of significance by using stars = NULL.

huxreg(model1, model2, model3, 
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05, 
                 stars = NULL
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)
(Intercept)	-5780.831	-5736.897	-3904.387
	(305.815)	(307.959)	(529.257)
flipper_length_mm	49.686	48.145	27.429
	(1.518)	(2.011)	(3.176)
bill_length_mm		6.047	61.736
		(5.180)	(7.126)
speciesChinstrap			-748.562
			(81.534)
speciesGentoo			90.435
			(88.647)
#observations	342	342	342
R squared	0.759	0.760	0.822
Adj. R Squared	0.758	0.759	0.820
Residual SE	394.278	394.068	340.114

Warning 1: huxtable reformats all your tables

If your document creates any other tables (like with tidy()), huxtable automatically formats these tables in a fancy way. If you don’t want that, you can turn it off with this code—put it at the top of your document near where you load your libraries:

# Huxtable likes to automatically format *all* tables, which is annoying. 
# This turns that off.
options('huxtable.knit_print_df' = FALSE)

warning 2: Knitting to PDF is fragile

In order to knit to PDF, you need to install LaTeX, which you should have done when you installed tinytex. When using huxtable, before knitting to PDF for the first time on your computer, you need to run this in your console to install the LaTeX packages that R uses to knit huxtable tables to PDF:

huxtable::install_latex_dependencies()

If you’re using tinytex, you’ll also need to run this once on your computer:

tinytex::tlmgr_install("unicode-math")

Worked examples | Learning from Data: R programming

Exploratory Data Analysis for Modelling

Penguin Bill dimensions

Exploratory Data Analysis (EDA).

GGally::ggpairs() to get scatter plot + correlation matrix

Dummy variables

Modelling considerations of numerical variables

Penguin body mass vs flipper length

Penguin body mass vs bill depth

Penguin body mass and flipper size, faceted by sex

Simpson’s Paradox: Penguin bill length vs bill depth

Is Y heavily skewed? Does it need to be transformed?

Acknowledgements

Import data

Overview

Importing CSV files: read_csv()

Importing CSV files directly off the internet

Importing CSV files saved locally

Need for speed: Enter data.table::fread()

Other data formats

rio: a swiss-army knife for data input-output

Never work directly on the raw data

Other links

Sampling and confidence intervals (CI)

Summary statistics

Boxplots

Confidence Intervals (CI)

CIs for body mass

CIs for flipper length

t-Test assuming unequal variance

t-Test for body mass

t-Test for flipper length

t-Test assuming equal variance

How do we test whether the two groups have equal variance?

Bartlett test: Check equality of variances based on the mean

Levene test Check equality of variances based on the median

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

Acknowledgements

Testing for differences in mean values

Summary statistics

Boxplots

Confidence Intervals (CI)

CIs for body mass

CIs for flipper length

t-Test assuming unequal variance

t-Test for body mass

t-Test for flipper length

t-Test assuming equal variance

How do we test whether the two groups have equal variance?

Bartlett test: Check equality of variances based on the mean

Levene test Check equality of variances based on the median

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

Acknowledgements

Inpect data

Overview

dplyr::glimpse()

skimr::skim()

On you own

Simulation-based tests

There is only one test

Bootstrap simulation, using infer package

CI for median

Clean data

janitor package for cleaning variable names

Code that works is not necessarily good code

Other links

Linear Model Fitting

Fit a model using lm(Y ~ X1 + X2 +..., data = dataframe)

Look at model output

Write down the equation for model1

Plot scatterplot and residuals

Add more explanatory variables

Check collinearity

Summary model comparison table using huxtable::huxreg()

Fitting multiple regression models in one go

Simpson’s paradox

Model diagnostics

Linear Regression Assumptions: L-I-N-E

Regression diagnostic plots with ggfortify::autoplot()

Binary classification model

`GGally::ggpairs()` to get scatter plot + correlation matrix

Importing CSV files: `read_csv()`

Need for speed: Enter `data.table::fread()`

`rio`: a swiss-army knife for data input-output

`dplyr::glimpse()`

`skimr::skim()`

Bootstrap simulation, using `infer` package

`janitor` package for cleaning variable names

Fit a model using `lm(Y ~ X1 + X2 +..., data = dataframe)`

Summary model comparison table using `huxtable::huxreg()`

Regression diagnostic plots with `ggfortify::autoplot()`

Making plots interactive using `plotly`

Gapminder Animations - `transition_time()`

The `pipe` operator, or `%>%`

Key functions in `dplyr`

Pick columns with `select()`

Pick rows with `filter()`

Sort data with `arrange()`

Add new columns with `mutate()`

Combine multiple verbs with pipes (`%>%`)

Summarise data by groups with `group_by() %>% summarise()`

`pivot_longer` or `gather` data

`pivot_wider` or `spread` data

`huxtable::huxreg()`