Exercises | Learning from Data: R programming

Confidence Intervals

Sat, 25 Jul 2020 00:00:00 +0000

Exploratory Data Analysis for Modelling

Sat, 25 Jul 2020 00:00:00 +0000

R Syntax, Vectors, missing data

Sat, 25 Jul 2020 00:00:00 +0000

R Syntax

We can type commands in the command prompt and use R as a simple calculator. For instance, try typing 5 + 20, and hitting enter. When you do this, you’ve entered a command, and R will execute that command. However, it’s more interesting when we can create objects or variables and work with these beasts!

Assignmnent Operator `<-`

R treats everything (single numbers, lists, vectors, datasets) as objects. To create an object, we must use the assignment operator <-. For instance, if we had data on a student whose name is Alex, is 28 years old, and comes from Athens, we would create three objects, name, height, and city and assign the values of Alex, 28, and Athens respectively, we would type

name <- "Alex"
age <- 28
city <- "Athens"

The two objects have now been created; if we wanted to print out their values, we can use the print() function or just type the names of the objects.

print(name); print(age); print(city)

## [1] "Alex"

## [1] 28

## [1] "Athens"

name

## [1] "Alex"

age

## [1] 28

city

## [1] "Athens"

You can mentally read the command age <- 28 as object age becomes equal to the value 28. There is a keyboard shortcut Alt + - to get the assignment operator. We can do more interesting and useful things creating variables and assigning values to them. For instance, if we have the relevant dimensions and wanted to calculate the area and volume of a room, we could do it as follows:

room_length <- 5.63
room_width  <- 6.48
room_height <- 2.93
room_area <- room_length * room_width
room_volume <- room_length * room_width * room_height

room_area

## [1] 36.4824

room_volume

## [1] 106.8934

R is case sensitive

R is case sensitive and needs everything exactly as it was defined. age is different from AgE and Age. So if you type

age <- 28
AgE <- 34
Age <- 55

age; AgE; Age

## [1] 28

## [1] 34

## [1] 55

R will create three different objects.

Typos

R is a brilliant piece of software, but it cannot handle typos. Unlike Google’s search, “Did you mean…”, it takes it on faith that what you typed is exactly what you meant. For example, suppose that you forgot to hit the shift key when trying to type +, and as a result your command ended up being 5 = 20 rather than 5 + 20. Here’s what happens:

5 = 20

## Error in 5 = 20: invalid (do_set) left-hand side to assignment

R attempted to interpret 5 = 20 as a command, and spits out an error message because this makes no sense to it. Even more subtle is the fact that some typos won’t produce errors at all, because they happen to correspond to R commands. For instance, suppose that instead of 5 + 20, I mistakenly type command 5 - 20. Clearly, R has no way of knowing that you meant to add 20 to 5, not subtract 20 from 5, so what happens this time is this:

5 - 20

## [1] -15

In this case, R produces the right answer, but to the the wrong question.

R will always try to do exactly what you ask it to do. There is no autocorrect or equivalent to “Did you mean..” in R, and for good reason. When doing advanced stuff and even the simplest of statistics is pretty advanced in a lot of ways, it’s dangerous to let a mindless automaton like R try to overrule the human user. But because of this, it’s your responsibility to be careful. Always make sure you type exactly what you mean. When dealing with computers, it’s not enough to type approximately the right thing. In general, you absolutely must be precise in what you say to R … like all machines it is too stupid to be anything other than absurdly literal in its interpretation.

Comments

It is useful to put comments in your code, to make everything more readable. These comments could help others and you when you go back to your code in the future. R comments start with a hashtag sign #. Everything after the hashtag to the end of the line will be ignored by R. RStudio by default thinks that every line you write is a command; if you want to turn a line into a comment, place the cursor in the line and hit Ctrl + Shift + C in Windows or Cmd + Shift + C in a Mac.

# This line is a comment and will be ignored when run.
city # Text after the hashtag "#" is also ignored.

## [1] "Athens"

R knows you’re not finished

If you hit enter in a situation where it’s obvious to R that you haven’t actually finished typing the command, R is just smart enough to keep waiting. For example, if you wanted to calculate 15 - 4, and start by typing type 15 - and then press enter by mistake, R is smart enough to realise that you probably wanted to type in another number. So here’s what happens:

> 15 -
+

and there’s a blinking cursor next to the plus + sign. What this means is that R is still waiting for you to finish. It thinks you’re still typing your command, so it hasn’t tried to execute it yet. In other words, this plus sign is actually another command prompt. It’s different from the usual one (i.e., the > symbol) to remind you that R is going to add whatever you type now to what you typed last time. For example, if I then go on to type 4 and hit enter, what we get:

> 15 -
+ 4
[1] 11

And as far as R is concerned, this is exactly the same as if you had typed 15 - 4.

By the way, if after entering the 15 - you wanted to stop execution and cancel your command, just hit the escape key. R will return you to the normal command prompt (i.e. >) without attempting to execute the botched command.

Arithmetic Operations and Functions

R has the basic operators and you can use it as as simple calculator: addition is +, subtraction is -, multiplication is *, division is /, and ^ is the power operator:

2 + 3

## [1] 5

5 - 8

## [1] -3

13 * 21

## [1] 273

34 / 55

## [1] 0.6181818

(5 * 13)/4 - 7

## [1] 9.25

# ^ : to the power off
2^3

## [1] 8

# for exponentiation, you can also use **
2 ** 3

## [1] 8

# square root
sqrt(25)

## [1] 5

Besides the basic operations functions, you can use standard mathematical functions

Rounding -round(), floor(), ceiling(),
Logarithms and Exponentials -exp(), log(), log10(), log2()

# R knows pi = 3.1415926...

# round to 2 decimal places 
round(pi, digits = 2); round(pi,2)

## [1] 3.14

## [1] 3.14

#Round down to nearest integer
floor(pi)

## [1] 3

#Round up to nearest integer
ceiling(pi)

## [1] 4

Main Data types and Vectors

character: sometimes referred to as string data, tend to be surrounded by quotes <chr>
numeric: real or decimal numbers, sometimes referred to as “double” <dbl>
integer: a subset of numeric in which numbers are stored as integers <int>
factor: a categorical variables with different categories sorted alphabetically by default <fct>
logical: Boolean data (TRUE and FALSE) <lgl>

Your turn

Vectors

A vector is a collection of objects. There is a magical operator in R, c which we use to combine different elements.

# assign vector
ages <- c(20:30, 35, 50, 42, 72) 

# recall vector
ages

##  [1] 20 21 22 23 24 25 26 27 28 29 30 35 50 42 72

# how many things are in the vector 'ages'?
length(ages)

## [1] 15

# what type of object is 'ages?
class(ages)

## [1] "numeric"

R allows vectorized operations, so we can get the average, or median of ages by just typing

# performing functions with vectors
mean(ages)

## [1] 31.6

median(ages)

## [1] 27

We can also have a collection of strings, or characters

# vector of days of the week 
days <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")

In this case, each word is encased in quotation marks, indicating they are characters rather than object names.

Please answer the following questions about days:

How many values are in days?
What type of data (class) is days?
Overview of days

Manipulating vectors

# add a value to end of vector
ages <- c(ages, 90)

# add value at the beginning
ages <- c(30, ages)

# extracting second value
days[2]

## [1] "Tuesday"

# excluding (dropping) second value
days[-2]

## [1] "Monday"    "Wednesday" "Thursday"  "Friday"    "Saturday"  "Sunday"

# extracting first and third values
days[c(1, 3)]

## [1] "Monday"    "Wednesday"

Your turn

R tends to handle interpreting data types in the background of most operations. Usually it tries to coerce data to fit the general pattern of the data given to it.

What type of data is each of the following objects? Anything unusual?

num_char <- c(1, 2, 3, "a")
num_logical <- c(1, 2, 3, TRUE)
char_logical <- c("a", "b", "c", TRUE)
tricky <- c(1, 2, 3, "4")

Factors

days is a character vector so R internally sorts it alphabetically and it thinks that Friday should be first. If we wanted to make into a categorical variable, we use

days <- factor(days)

We can reorder, or relevel the factor, using fct_relevel from the tidyverse package forcats, or using levels from baseR.

days_sorted <- forcats::fct_relevel(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

days_sorted2 <- factor(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

Your turn

Missing data, or `NA`

# create a vector with missing data
times <- c(2, 4, 4, NA, 6)

NA is not a character

# calculate mean and max on vector with missing data
mean(times)

## [1] NA

max(times)

## [1] NA

# add argument to remove NA
mean(times, na.rm = TRUE)

## [1] 4

max(times, na.rm = TRUE)

## [1] 6

# remove incomplete cases
na.omit(times)

## [1] 2 4 4 6
## attr(,"na.action")
## [1] 4
## attr(,"class")
## [1] "omit"

Your turn

Online Quiz: Variable Types - Vectors

Visualise data

Sat, 25 Jul 2020 00:00:00 +0000

Introduction to `ggplot2` by visualising numeric data.

We will start with the gapminder data set. We look at its contents

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

Animating changes

Racing bars! We will create a simple bar graph showing the evolution of GDP per capita for the top 8 countries

IMDB movie ratings: Scatterplots and relationships

For this section, we will use a sample of movies released since 2000 with data from IMDB. We have data on movies from the following six genres:

Action
Adventure
Comedy
Drama
Animation
Documentary

imdb <- read_csv(here::here("data", "movies.csv"))
imdb_short <- imdb %>% 
  filter(genre %in% c("Action", "Adventure", "Comedy", "Drama", "Animation", "Documentary"),
         year >= 2000)

glimpse(imdb_short)

## Rows: 1,762
## Columns: 11
## $ title               <chr> "Avatar", "Jurassic World", "The Avengers", "Th...
## $ genre               <chr> "Action", "Action", "Action", "Action", "Action...
## $ director            <chr> "James Cameron", "Colin Trevorrow", "Joss Whedo...
## $ year                <dbl> 2009, 2015, 2012, 2008, 2015, 2012, 2004, 2013,...
## $ duration            <dbl> 178, 124, 173, 152, 141, 164, 93, 146, 151, 103...
## $ gross               <dbl> 760505847, 652177271, 623279547, 533316061, 458...
## $ budget              <dbl> 2.37e+08, 1.50e+08, 2.20e+08, 1.85e+08, 2.50e+0...
## $ cast_facebook_likes <dbl> 4834, 8458, 87697, 57802, 92000, 106759, 1148, ...
## $ votes               <dbl> 886204, 418214, 995415, 1676169, 462669, 114433...
## $ reviews             <dbl> 3777, 1934, 2425, 5312, 1752, 3514, 688, 1208, ...
## $ rating              <dbl> 7.9, 7.0, 8.1, 9.0, 7.5, 8.5, 7.2, 7.6, 7.3, 8....

IMDB movie ratings: Boxplots, violin plots

Let us consider the rating movies got according to their genre. How can we visualise the distribution of ratings?

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_violin()+
  theme_minimal()+
  theme(legend.position = "none")

Multiple panels using `facet_wrap()` and `facet_grid()`

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(year),
    cols= vars(genre)
  )

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(cut(budget, 3)),
    cols= vars(genre)
  )

Hypothesis Testing

Sat, 25 Jul 2020 00:00:00 +0000

Import, inspect, and clean data

Sat, 25 Jul 2020 00:00:00 +0000

Inspecting and Cleaning Data

Inspecting and Cleaning Data

Inspecting data using skimr::skim()

Manipulate data

Sat, 25 Jul 2020 00:00:00 +0000

Select variables in a dataset using `select()` and sort using `arrange()`

The dataframe movies has been loaded into memory. It contains a sample of movies from IMDB, and its contents are shown below:

Filter rows in a dataset using `filter()`

Remember that select() allows us to choose columns, or variables, whereas filter() chooses rows, or cases, that conform to certain criteria

`mutate()` to change the data type of a variable and create new variables

`group_by()/summarise()` to get summary statistics, including counts, means, etc., within categories.

Putting it all together

You can put together all of your dplyr knowledge to work four genres of movies, namely action, adventure, comedy and drama and create the following plot.

For these genres, you have to

Make sure you do not have multiple entries of the same movie; use distinct(movie, _keep.all=TRUE)
Calculate a normalised metric for rating, where you adjust the movie’s rating by the number of votes it received out of the total votes in its genre, normalised_rating = rating * (votes / total votes in genre)
Arrange movies, so higher normalised_rating appears first.
Categorise and colour movies according to their budget cost
- cheap (<20m, or <20e6 as e6 is R shorthand for 1 million, or $10^6$,
- moderate (20-120m), and
- expensive (>120m)
Convert cost column to a factor variable and re-level in the correct order (cheap, moderate, expensive)
Change the labels in the x- and y-axis, and give appropriate titles, subtitles, etc
use theme minimal

Some tips:

for sorting columns within a a ggplot, check out the reorder() in x argument
if you perform dplyr on original dataframe make sure to overwrite dataframe otherwise all changes are done on the fly and are not saved
consider freeing the scales of the facet wrap

Model Fitting

Sat, 25 Jul 2020 00:00:00 +0000

Questions on Descriptive and Inferential Statistics

Wed, 26 Aug 2020 00:00:00 +0000

The questions listed below are designed for discussion and preparation. When reviewing these questions, try to illustrate your points with specific examples/cases from what we have seen in class.

Why do we make distinctions between samples and populations in statistics?
Discuss the use of exploratory data analysis. Illustrate with an example.
Why do we care about the distribution of their data?
What are outliers? What impact do they have on how you describe your data?
What is a robust statistic? Why would you choose to use it?
What is variability? Discuss measures of variability. What are their strengths and weaknesses? In a operations setting, what is the relation of variability to the quality of the product/service?
Why do we standardise data? What does a Z-score tell you, i.e., how do you interpret one? How do you convert a raw score to a Z-score?
What is the Normal distribution? What is the Standard Normal distribution?
How do you use the Normal distribution table to find the percentage of the population that is expected to fall between two points or beyond or below one point in the distribution?
What does the Central Limit Theorem tell us?
Discuss the differences between descriptive and inferential statistics. Is one better than the other? Are they competitive or complementary? Illustrate the kind of situation in which each approach is appropriate.
What are the steps involved in hypothesis testing?
What does specifying the null hypothesis mean? What about the alternative hypothesis? What is the benefit of being so specific about the hypotheses?
With inferential statistics, the goal is to reject the null hypothesis. What does this mean? Do we conclude that the alternative hypothesis is correct? Why or why not?
Why is the standard error of the mean, based on many samples, going to be smaller than the standard deviation of a single sample? In explaining your answer, be sure to describe the interpretation of a standard error of the mean.
What types of error can occur when making decisions based on test of hypothesis? Be specific.
Why are observations that are more than 3 or less than -3 standard deviations from the mean often considered outliers by some researchers?
What does it mean if a researcher sets her $\alpha = 0.01$, and rejects the null hypothesis? How does this differ from setting the alpha at $\alpha = 0.05$ and rejecting the null? In which case is the researcher going to be most likely to reject the null hypothesis?
What is the difference between a one-tailed (directional) and a two-tailed (non-directional) test? When would you use each of them?
What is meant when a researcher says that a finding is statistically significant?

Handling dates/times

Mon, 27 Jul 2020 00:00:00 +0000

The lubridate package is one of the most useful packages to handle dates and times in R. Your data may contain dates as a string of characters (class <chr>), and you need to convert them to date objects before you can do any kind of analysis.

`lubridate` to convert strings to `Date` objects

Let us look at an example: We will define Christmas as a string in various formats and will then try to convert it to a Date object so we can manipulate it.

library(lubridate)

today <-  Sys.Date()
today

## [1] "2020-08-25"

class(today)

## [1] "Date"

date1 <-  "25-12-2020"
date2 <-  "12-25-2020"
date3 <-  "2000-12-25"
date4 <-  "Dec 25, 2020"

class(date1)

## [1] "character"

class(date4)

## [1] "character"

# dmy: day-month-year
xmas1 <- lubridate::dmy(date1)
class(xmas1)

## [1] "Date"

# mdy: month-day-year
xmas2 <- lubridate::mdy(date2)
class(xmas2)

## [1] "Date"

# ymd: year-month-date, ISO8601 standard
# https://en.wikipedia.org/wiki/ISO_8601
xmas3 <- ymd(date3)
class(xmas3)

## [1] "Date"

# mdy: month-day-year
xmas4 <- lubridate::mdy(date4)
class(xmas4)

## [1] "Date"

# once we have it as a Data object, we can do calculations...
xmas1 - today

## Time difference of 122 days

# ... but these calculations will not work if the date is a string (character) 
date1 - today

## Error in `-.Date`(date1, today): can only subtract from "Date" objects

On your own

We’ll use data from http://www.tfl.gov.uk to analyse usage of the London Bike Sharing scheme. This data has already been downloaded for you and exists in a CSV (Comma Separated Values), along with weather information.

bike <- read_csv(here::here("data", "londonBikes.csv"))

glimpse(bike)

## Rows: 3,439
## Columns: 14
## $ date          <chr> "01-01-11", "02-01-11", "03-01-11", "04-01-11", "05-0...
## $ bikes_hired   <dbl> 4555, 6250, 7262, 13430, 13757, 9595, 9294, 9338, 105...
## $ season        <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ max_temp      <dbl> 7.2, 4.0, 2.9, NA, 7.1, NA, 10.8, 10.4, 7.2, 8.9, 8.3...
## $ min_temp      <dbl> NA, NA, NA, 0.3, 3.8, NA, 1.0, NA, NA, -1.9, NA, 3.7,...
## $ avg_temp      <dbl> 5.6, 2.9, 1.4, 2.7, 5.6, 4.1, 6.1, 6.9, 3.1, 4.3, 5.8...
## $ avg_humidity  <dbl> 84, 79, 80, 87, 84, 92, 92, 82, 79, 87, 82, 89, 89, 8...
## $ avg_pressure  <dbl> 1025, 1028, 1024, 1013, 1000, 996, 999, 997, 1012, 10...
## $ avg_windspeed <dbl> 10, 8, 6, 6, 19, 5, 11, 23, 16, 14, 16, 16, 23, 24, 2...
## $ rainfall_mm   <dbl> 0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 11.4, 13.0, 1.0, 0.0, 7...
## $ rain          <lgl> TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, FAL...
## $ fog           <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE,...
## $ thunderstorm  <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALS...
## $ snow          <lgl> FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE,...

date is a character string, and is given as 01-01-2011, 02-01-2011, 03-01-2011, meaning 1st, 2nd, 3rd of January, etc. In other words, the format of the string is dmy, or day-month-year.

Model diagnostics

Sat, 25 Jul 2020 00:00:00 +0000

Reshape data

Sat, 25 Jul 2020 00:00:00 +0000

`pivot_longer()` - `pivot_wider` to make a wide dataset long and vice-versa

The United Nations have all sorts of data. For this example, we will work with data on tourist/visitor arrivals and tourism expenditure. The dataframe un_tourism_data has been loaded into memory, and contains data on tourist arrivals (in thousands) and tourism expenditure (in millions of US$). We would like to calculate spending per tourist and see how some of the top tourist destinations compare

You have to:

Convert un_tourism_data from long to wide format; you need to do this to create the new variable spending_per_tourist
clean column names
rename columns to “tourism_expenditure” and “tourist_arrivals”
remove rows where tourism expenditure or arrivals are NA
create a new column where you calculate spending per tourist (remember expenditure is in millions and arrival is in thousands)

You have successfully calculated spending per tourist. We are now faced with the challenge of producing a plot that looks like this

The best way to get this plot is to first reshape the dataframe from wide to long, and then apply your ggplot skills.

`left_join()` to join tables on columns

We have loaded into memory two dataframes countries and matches that contain matches played in various European football (soccer) leagues over a number of years. We want to join the two dataframes, so we can see the name, rather than an ID of the league. We also want to calculate the average number of goals per game in each league and plot those averages for all seasons.

`bind_rows()` to combine rows from two or more datasets

We have three distinct dataframes, england_matches, germany_matches, and italy_matches that contain data on each country. We need to combine these three datasets into one, and sort it in ascending order by date.

Exercises | Learning from Data: R programming

Confidence Intervals

Exploratory Data Analysis for Modelling

R Syntax, Vectors, missing data

R Syntax

Assignmnent Operator <-

R is case sensitive

Typos

Comments

R knows you’re not finished

Arithmetic Operations and Functions

Main Data types and Vectors

Your turn

Vectors

Manipulating vectors

Your turn

Factors

Your turn

Missing data, or NA

Your turn

Online Quiz: Variable Types - Vectors

Visualise data

Introduction to ggplot2 by visualising numeric data.

Scatter plots and multiple panels using facet_wrap()

Animating changes

IMDB movie ratings: Scatterplots and relationships

IMDB movie ratings: Boxplots, violin plots

Multiple panels using facet_wrap() and facet_grid()

Hypothesis Testing

Import, inspect, and clean data

Inspecting and Cleaning Data

Manipulate data

Select variables in a dataset using select() and sort using arrange()

Filter rows in a dataset using filter()

mutate() to change the data type of a variable and create new variables

group_by()/summarise() to get summary statistics, including counts, means, etc., within categories.

Putting it all together

Model Fitting

Questions on Descriptive and Inferential Statistics

Handling dates/times

lubridate to convert strings to Date objects

On your own

Model diagnostics

Reshape data

pivot_longer() - pivot_wider to make a wide dataset long and vice-versa

left_join() to join tables on columns

bind_rows() to combine rows from two or more datasets

Assignmnent Operator `<-`

Missing data, or `NA`

Introduction to `ggplot2` by visualising numeric data.

Scatter plots and multiple panels using `facet_wrap()`

Multiple panels using `facet_wrap()` and `facet_grid()`

Select variables in a dataset using `select()` and sort using `arrange()`

Filter rows in a dataset using `filter()`

`mutate()` to change the data type of a variable and create new variables

`group_by()/summarise()` to get summary statistics, including counts, means, etc., within categories.

`lubridate` to convert strings to `Date` objects

`pivot_longer()` - `pivot_wider` to make a wide dataset long and vice-versa

`left_join()` to join tables on columns

`bind_rows()` to combine rows from two or more datasets