Resources

You can download all the data sets, R scripts, practice questions and their solutions from our GitHub repository.

Introduction

In this section, we will learn to visualize categorical data. We will look at the following type of plots:

• univariate bar plot
• bivariate bar plot
  • grouped
  • stacked
  • proportional
• mosaic plot
• pie chart
• donut chart

We will be using ggplot2 package throughout this article. So you should know the basics of data visualization with ggplot2. If you are new to or have never used ggplot2, do not worry. We have several tutorials and an ebook on ggplot2, you can go through them first and then come back to this article. Let us read the case study data before we start our visualization journey.

# read data
data <- readRDS('analytics.rds')

Bar Plot

Bar charts provide a visual representation of categorical data. The bars can be plotted either vertically or horizontally. The categories/groups appear along the horizontal X axis and the height of the bar represents a measured value.

ggplot(data) +
  geom_bar(aes(x = device), fill = "blue") +
  xlab("Device") + ylab("Count")

In the above example, the bars represent the count/frequency of the categories. If the bars represent continuous data, the value could be mean or sum of the variable being represented.

ggplot(data) +
  geom_bar(aes(x = device, fill = gender), position = "dodge") +
  xlab("Device") + ylab("Count")

ggplot(data) +
  geom_bar(aes(x = device, fill = gender)) +
  xlab("Device") + ylab("Count")

data %>% 
  select(device, gender) %>% 
  table() %>% 
  tibble::as_tibble() %>% 
  ggplot(aes(x = device, y = n, fill = gender)) +
  geom_bar(stat = "identity", position = "fill") +
  xlab("Device") + ylab("Gender")

Mosaic Plot

A mosaic plot is a graphical representation of a two way table or contingency table. It was introduced by Hartigan & Kleiner and is divided into rectangles. Proportions on horizontal axis represents the number of observations for each level of the X variable. The vertical length of each rectangle is proportional to the proportion of Y variable in each level of X variable.

ggplot(data = data) +
  geom_mosaic(aes(x = product(channel, device), fill = channel)) +
  xlab("Device") + ylab("Channel")

Pie Chart

Pie chart is a circular chart, divided into slices to show relevant sizes of data. It shows the distribution of the different levels of a categorical variable as a circle is divided into radial slices. Each level corresponds with a single slice of the circle and size indicates the proportion of the level. Use it when comparing each group’s contribution to the whole as opposed to comparing groups to each other.

data %>% 
  pull(device) %>%
  table() %>%
  pie()

data %>% 
  pull(device) %>% 
  table() %>% 
  pie3D(explode = 0.1)

data %>% 
  pull(device) %>% 
  fct_count() %>% 
  rename(device = f, count = n) %>% 
  ggplot() +
  geom_bar(aes(x = "", y = count, fill = device), width = 1, stat = "identity") +
  coord_polar("y", start = 0)

Donut Chart

Donut chart is a variation of the pie chart. It has a round hole in the middle which makes it look like a donut. The focus is on the length of the arcs and not the proportions of the slices. Blank spaces inside donut chart can be used to display information inside it.

data %>% 
  pull(device) %>% 
  fct_count() %>% 
  rename(device = f, count = n) %>% 
  ggdonutchart("count", label = "device", fill = "device", color = "white",
               palette = c("#00AFBB", "#E7B800", "#FC4E07"))

Summary

• Bar charts provide a visual representation of categorical data.
• Use grouped bar chart to make comparison against different categories of data.
• Use stacked bar chart while looking at cumulative data.
• Use proportional bar chart when you want to concurrently observe each of the several variables as they fluctuate.
• Use mosaic plot to discover associations between two variables.
• Use pie chart and donut chart when comparing each group’s contribution to the whole.

Your Turn…

Generate all the below plots:

1. Bar plot of channel

2. Display grouped bar plot of user_type by channel

3. Display stacked bar plot of channel by gender

4. Display proportional bar plot of channel by device

5. Display mosaic plot of device by channel

6. Display pie or donut chart of channel

Table of Contents

• Resources
• Introduction
• How to change value of levels?
• How to add or remove levels?
• How to change order of levels?
• Practice Questions

Resources

You can download all the data sets, R scripts, practice questions and their solutions from our GitHub repository.

Introduction

In this section, our focus will be on handling the levels of a categorical variable, and exploring the forcats package for the same. We will basically look at 3 key operations or transformations we would like to do when it comes to factors which are:

• change value of levels
• add or remove levels
• change order of levels

Before we start working with the value of the levels, let us read the case study data and take a quick look at some of the functions we used in the previous articles.

# read data
data <- readRDS('analytics.rds')

We will store the source of traffic as channel instead of referring to the column in the data.frame every time.

channel <- data$channel

Let us go back to the function we used for tabulating data, fct_count(). If you observe the result, it is in the same order as displayed by levels().

fct_count(channel)

## # A tibble: 8 x 2
##   f                   n
##   <fct>           <int>
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Display          3375
## 5 Organic Search 139668
## 6 Paid Search      4395
## 7 Referral        35615
## 8 Social           8031

If you want to sort the results by the count i.e. most common level comes at the top, use the sort argument.

fct_count(channel, sort = TRUE)

## # A tibble: 8 x 2
##   f                   n
##   <fct>           <int>
## 1 Organic Search 139668
## 2 Direct          39853
## 3 Referral        35615
## 4 Social           8031
## 5 Affiliates       7388
## 6 (Other)          6073
## 7 Paid Search      4395
## 8 Display          3375

If you want to view the proportion along with the count, set the prop argument to TRUE.

fct_count(channel, prop = TRUE)

## # A tibble: 8 x 3
##   f                   n      p
##   <fct>           <int>  <dbl>
## 1 (Other)          6073 0.0248
## 2 Affiliates       7388 0.0302
## 3 Direct          39853 0.163 
## 4 Display          3375 0.0138
## 5 Organic Search 139668 0.571 
## 6 Paid Search      4395 0.0180
## 7 Referral        35615 0.146 
## 8 Social           8031 0.0329

One of the important steps in data preparation/sanitization is to check if the levels are valid i.e. only levels which should be present in the data are actually present. fct_match() can be used to check validity of levels. It returns a logical vector if the level is present and an error if not.

table(fct_match(channel, "Social"))

## 
##  FALSE   TRUE 
## 236367   8031

Change Value of Levels

In this section, we will learn how to change the value of the levels. In order to keep it interesting, we will state an objective from our case study and then map it into a function from the forcats package.

fct_count(
  fct_collapse(
    channel,
    Search = c("Paid Search", "Organic Search")
  )
)

## # A tibble: 7 x 2
##   f               n
##   <fct>       <int>
## 1 (Other)      6073
## 2 Affiliates   7388
## 3 Direct      39853
## 4 Display      3375
## 5 Search     144063
## 6 Referral    35615
## 7 Social       8031

fct_count(
  fct_recode(
    channel,
    Search = "Paid Search",
    Search = "Organic Search"
  )
)

## # A tibble: 7 x 2
##   f               n
##   <fct>       <int>
## 1 (Other)      6073
## 2 Affiliates   7388
## 3 Direct      39853
## 4 Display      3375
## 5 Search     144063
## 6 Referral    35615
## 7 Social       8031

fct_count(fct_lump_min(channel, 5000))

## # A tibble: 7 x 2
##   f                   n
##   <fct>           <int>
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770

fct_count(fct_lump_n(channel, 3))

## # A tibble: 4 x 2
##   f                   n
##   <fct>           <int>
## 1 Direct          39853
## 2 Organic Search 139668
## 3 Referral        35615
## 4 Other           29262

fct_count(fct_lump_prop(channel, 0.02))

## # A tibble: 7 x 2
##   f                   n
##   <fct>           <int>
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770

fct_count(
  fct_other(
    channel, 
    keep = c("Organic Search", "Direct", "Referral"))
)

## # A tibble: 4 x 2
##   f                   n
##   <fct>           <int>
## 1 Direct          39853
## 2 Organic Search 139668
## 3 Referral        35615
## 4 Other           29262

fct_count(
  fct_other(
    channel, 
    drop = c("Display", "Paid Search")
  )
)

## # A tibble: 7 x 2
##   f                   n
##   <fct>           <int>
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770

# collapse
fct_count(
  fct_collapse(
  channel,
  Other = c("(Other)", "Affiliate", "Display", "Paid Search", "Social")
  )
)

## Warning: Unknown levels in `f`: Affiliate

## # A tibble: 5 x 2
##   f                   n
##   <fct>           <int>
## 1 Other           21874
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615

# recode
fct_count(
  fct_recode(
  channel,
  Other = "(Other)", 
  Other = "Affiliate", 
  Other = "Display", 
  Other = "Paid Search", 
  Other = "Social"
  )
)

## Warning: Unknown levels in `f`: Affiliate

## # A tibble: 5 x 2
##   f                   n
##   <fct>           <int>
## 1 Other           21874
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615

fct_count(fct_anon(channel, prefix = "ch_"))

## # A tibble: 8 x 2
##   f          n
##   <fct>  <int>
## 1 ch_1    6073
## 2 ch_2    8031
## 3 ch_3    4395
## 4 ch_4   39853
## 5 ch_5  139668
## 6 ch_6    3375
## 7 ch_7   35615
## 8 ch_8    7388

Add / Remove Levels

In this small section, we will learn to:

• add new levels
• drop levels
• make missing values explicit

levels(fct_expand(channel, "Blog"))

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"        
## [9] "Blog"

levels(fct_drop(fct_expand(channel, "Blog")))

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

fct_count(fct_explicit_na(data$gender))

## # A tibble: 3 x 2
##   f              n
##   <fct>      <int>
## 1 female     40565
## 2 male       61617
## 3 (Missing) 142216

Change Order of Levels

In this last section, we will learn how to change the order of the levels. We will look at the following scenarios from our case study:

We want to make

• Organic Search the first level
• Referral the third level
• Display the last level

levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_relevel(channel, "Organic Search"))

## [1] "Organic Search" "(Other)"        "Affiliates"     "Direct"        
## [5] "Display"        "Paid Search"    "Referral"       "Social"

levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_relevel(channel, "Referral", after = 2))

## [1] "(Other)"        "Affiliates"     "Referral"       "Direct"        
## [5] "Display"        "Organic Search" "Paid Search"    "Social"

levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_relevel(channel, "Display", after = Inf))

## [1] "(Other)"        "Affiliates"     "Direct"         "Organic Search"
## [5] "Paid Search"    "Referral"       "Social"         "Display"

# reorder levels
levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_infreq(channel))

## [1] "Organic Search" "Direct"         "Referral"       "Social"        
## [5] "Affiliates"     "(Other)"        "Paid Search"    "Display"

# reorder levels
levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_inorder(channel))

## [1] "Organic Search" "Direct"         "Referral"       "Affiliates"    
## [5] "(Other)"        "Social"         "Display"        "Paid Search"

# reorder levels
levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_rev(channel))

## [1] "Social"         "Referral"       "Paid Search"    "Organic Search"
## [5] "Display"        "Direct"         "Affiliates"     "(Other)"

# reorder levels
levels(channel)

## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"

levels(fct_shuffle(channel))

## [1] "Organic Search" "Referral"       "Display"        "(Other)"       
## [5] "Direct"         "Paid Search"    "Affiliates"     "Social"

Your Turn…

1. Display the count/frequency of the following variables in the descending order

   • device
   • landing_page
   • exit_page

2. Check if laptop is a level in the device column.

3. Combine the following levels in landing_page into Account

   • My Account
   • Register
   • Sign In
   • Your Info

4. Combine levels in landing_page that drive less than 1000 visits.

5. Get top 10 landing and exit pages.

6. Get landing pages that drive at least 5% of the total traffic to the website.

7. Retain only the following levels in the browser column:

   • Chrome
   • Firefox
   • Safari
   • Edge

8. Anonymize landing and exit page levels.

9. Make Home first level in the landing_page column.

10. Make Apparel second level in the landing_page column.

11. Make Specials last level in the landing_page column.

12. Order the levels in the browser by frequency:

13. Order the levels in landing page by appearance:

14. Shuffle the levels in os

15. Reverse the levels in browser

*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.

We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*

Email: support@rsquaredacademy.com

Table of Contents

• Resources
• Introduction
• Tables in R
• Practice Questions

Resources

You can download all the data sets, R scripts, practice questions and their solutions from our GitHub repository.

Introduction

Categorical data cannot be summarized in the same way as numeric data. It does not make sense to look at range, standard deviation etc. since data consists of a few distinct values only. So how do we summarize such data? We can look at

• count/frequency
• proportion
• cumulative frequency
• cross table
• contingency table etc.

In this section, we will explore the above ways of summarizing categorical data. We will also spend some time learning about tables as you will be using them extensively while working with categorical data. R has many packages for tabulating data and we list and explore all of them in the R scripts shared in the GitHub repository.

# read data
data <- readRDS('analytics.rds')

nlevels(data$device)

## [1] 3

levels(data$device)

## [1] "Desktop" "Mobile"  "Tablet"

unique(data$device)

## [1] Desktop Mobile  Tablet 
## Levels: Desktop Mobile Tablet

fct_unique(data$device)

## [1] Desktop Mobile  Tablet 
## Levels: Desktop Mobile Tablet

table(data$device)

## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634

fct_count(data$device)

## # A tibble: 3 x 2
##   f            n
##   <fct>    <int>
## 1 Desktop 177282
## 2 Mobile   63482
## 3 Tablet    3634

summary(data$device)

## Desktop  Mobile  Tablet 
##  177282   63482    3634

Tables

In the previous section, we used the table() function to tabulate categorical data. We will recreate the tabulation for device and store it in a new variable tab.

tab <- table(data$device)
tab

## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634

What does this function return? It is not a vector, list, data.frame or matrix. Let us use the class() function to check the class of the object returned by table(). It returns an object of the class table. This is a new type of object. Let us spend some time understanding tables as they are useful for organizing and summarizing categorical data. table is also the most used object when it comes to dealing with categorical data.

The table() function returns the counts of the categories but let us say we want to view the proportion or percentage instead of counts i.e. the proportion or percentage of traffic driven to our website by the different devices. The proportions() or prop.table() function comes in handy in such cases. It takes a table object as input (tab in our case).

prop.table(tab)

## 
##    Desktop     Mobile     Tablet 
## 0.72538237 0.25974844 0.01486919

proportions(tab)

## 
##    Desktop     Mobile     Tablet 
## 0.72538237 0.25974844 0.01486919

To get the percentages, multiply the output by 100. Use the round() function to round the decimal places according to your requirements.

proportions(tab) * 100

## 
##   Desktop    Mobile    Tablet 
## 72.538237 25.974844  1.486919

round(proportions(tab) * 100, 2)

## 
## Desktop  Mobile  Tablet 
##   72.54   25.97    1.49

So far, we have used table() to tabulate a single categorical variable. It can be used for a lot more than just tabulating data. We can examine the relationship between two categorical variables as well as create multidimensional tables. Let us look at the relationship between gender and device in our case study. Does gender affect the type of device used? To answer this, we will create a two way or cross table. In the table() function, we can specify multiple variables by separating them with a comma.

tab2 <- table(data$gender, data$device)
tab2

##         
##          Desktop Mobile Tablet
##   female   32803   7268    494
##   male     46418  14503    696
##   <NA>     98061  41711   2444

Keep in mind that the order of the variables matter. Rows represent the first variable while column represents the second.

table(data$device, data$gender)

##          
##           female  male  <NA>
##   Desktop  32803 46418 98061
##   Mobile    7268 14503 41711
##   Tablet     494   696  2444

The proportions() function works with two way tables as well.

proportions(tab2)

##         
##              Desktop      Mobile      Tablet
##   female 0.134219593 0.029738378 0.002021293
##   male   0.189927904 0.059341729 0.002847814
##   <NA>   0.401234871 0.170668336 0.010000082

proportions(tab2) * 100

##         
##             Desktop     Mobile     Tablet
##   female 13.4219593  2.9738378  0.2021293
##   male   18.9927904  5.9341729  0.2847814
##   <NA>   40.1234871 17.0668336  1.0000082

We would like to introduce another function at this point of time, margin.table(). What does this function do? It computes the marginal frequencies i.e. the sum of the rows or columns. It takes a table object as input. The margin argument allows us to specify whether we want the sum of rows or columns. 1 indicates rows and 2 indicates columns.

margin.table(tab2, 1) # sum of rows

## 
## female   male   <NA> 
##  40565  61617 142216

margin.table(tab2, 2) # sum of columns

## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634

If the margin argument is NULL (which it is by default), the function returns the sum of all cells of the table.

margin.table(tab2)

## [1] 244398

table() does not display row or column labels. It does display the group labels though. Let us revisit the output from tab2. You can observe that while it includes the group labels, the row and column labels are missing. The output from the dimnames() function shows the group labels of the variables but the row & column labels are absent.

dimnames(tab2)

## [[1]]
## [1] "female" "male"   NA      
## 
## [[2]]
## [1] "Desktop" "Mobile"  "Tablet"

names(tab2)

## NULL

names(dimnames(tab2)) 

## [1] "" ""

The output from names(dimnames(tab2)) is also empty. Let us add the variable names as the row & column labels to tab2.

names(dimnames(tab2)) <- c("Gender", "Device")
tab2

##         Device
## Gender   Desktop Mobile Tablet
##   female   32803   7268    494
##   male     46418  14503    696
##   <NA>     98061  41711   2444

Now look at the output from tab2 and you can observe the difference. The same is also visible when we run dimnames(tab2).

dimnames(tab2)

## $Gender
## [1] "female" "male"   NA      
## 
## $Device
## [1] "Desktop" "Mobile"  "Tablet"

To add margin totals to the table, use addmargins(). Like proportions() and margin.table(), it also takes a table object as the input.

addmargins(tab2)

##         Device
## Gender   Desktop Mobile Tablet    Sum
##   female   32803   7268    494  40565
##   male     46418  14503    696  61617
##   <NA>     98061  41711   2444 142216
##   Sum     177282  63482   3634 244398

rowSums() returns the row total while colSums() returns the column total. They are similar to margin.table().

rowSums(tab2)

## female   male   <NA> 
##  40565  61617 142216

colSums(tab2)

## Desktop  Mobile  Tablet 
##  177282   63482    3634

xtabs() is another way of creating multidimensional tables in R. In comparison to table(), it

• uses formula notation for input
• the data argument ensures variable names are referenced instead of using $ i.e. datagender)) table(gen)

  ## gen
  ## female   male 
  ##  40565  61617

  As you can see, table() excludes missing values while tabulating the data. In order to ensure that missing values are also counted, we can use the useNA argument. It can take two values:

  • ifany
  • always

  In the first case, it will show NA as a level and the count only if there are missing values in the data. In the second case, it will always show NA as a level irrespective of whether there are missing values in the data or not.

  table(gen, useNA = "ifany")

  ## gen
  ## female   male   <NA> 
  ##  40565  61617 142216

  table(data$device, useNA = "always")

  ## 
  ## Desktop  Mobile  Tablet    <NA> 
  ##  177282   63482    3634       0

  In this final section on tables, we will learn how to select/access the different parts of a table. We will use [ operator to select rows and columns of a table (it is similar to selecting data from a data.frame). Below are a few examples:

  • select first row

  tab2[1, ]           

  ## Desktop  Mobile  Tablet 
  ##   32803    7268     494

  • select first column

  tab2[, 1]           

  ## female   male   <NA> 
  ##  32803  46418  98061

  • select first two rows

  tab2[1:2, ]         

  ##         Device
  ## Gender   Desktop Mobile Tablet
  ##   female   32803   7268    494
  ##   male     46418  14503    696

  • select first two columns

  tab2[, 1:2]        

  ##         Device
  ## Gender   Desktop Mobile
  ##   female   32803   7268
  ##   male     46418  14503
  ##   <NA>     98061  41711

  • select nth row

  tab2[2, ]          

  ## Desktop  Mobile  Tablet 
  ##   46418   14503     696

  • select nth column

  tab2[, 2]          

  ## female   male   <NA> 
  ##   7268  14503  41711

  • select row by group label

  tab2["female", ]   

  ## Desktop  Mobile  Tablet 
  ##   32803    7268     494

  • select column by group label

  tab2[, "Mobile"]   

  ## female   male   <NA> 
  ##   7268  14503  41711

  Before we end this section, let us learn how to test if an object is of class table using is.table().

  is.table(tab2)

  ## [1] TRUE

  Next, we will look at different R packages for two way/contingency tables.

  Contingency Table

  For cross tables with output similar to SAS or SPSS, use any of the below:

  • CrossTable() from the gmodels package
  • ds_cross_table() from the descriptr package

  gmodels::CrossTable(data$device, data$gender)

  ## 
  ##  
  ##    Cell Contents
  ## |-------------------------|
  ## |                       N |
  ## | Chi-square contribution |
  ## |           N / Row Total |
  ## |           N / Col Total |
  ## |         N / Table Total |
  ## |-------------------------|
  ## 
  ##  
  ## Total Observations in Table:  102182 
  ## 
  ##  
  ##              | data$gender 
  ##  data$device |    female |      male | Row Total | 
  ## -------------|-----------|-----------|-----------|
  ##      Desktop |     32803 |     46418 |     79221 | 
  ##              |    58.228 |    38.334 |           | 
  ##              |     0.414 |     0.586 |     0.775 | 
  ##              |     0.809 |     0.753 |           | 
  ##              |     0.321 |     0.454 |           | 
  ## -------------|-----------|-----------|-----------|
  ##       Mobile |      7268 |     14503 |     21771 | 
  ##              |   218.694 |   143.975 |           | 
  ##              |     0.334 |     0.666 |     0.213 | 
  ##              |     0.179 |     0.235 |           | 
  ##              |     0.071 |     0.142 |           | 
  ## -------------|-----------|-----------|-----------|
  ##       Tablet |       494 |       696 |      1190 | 
  ##              |     0.986 |     0.649 |           | 
  ##              |     0.415 |     0.585 |     0.012 | 
  ##              |     0.012 |     0.011 |           | 
  ##              |     0.005 |     0.007 |           | 
  ## -------------|-----------|-----------|-----------|
  ## Column Total |     40565 |     61617 |    102182 | 
  ##              |     0.397 |     0.603 |           | 
  ## -------------|-----------|-----------|-----------|
  ## 
  ## 

  descriptr::ds_cross_table(data, device, gender)

  ##     Cell Contents
  ##  |---------------|
  ##  |     Frequency |
  ##  |       Percent |
  ##  |       Row Pct |
  ##  |       Col Pct |
  ##  |---------------|
  ## 
  ##  Total Observations:  244398 
  ## 
  ## ----------------------------------------------------------------------------
  ## |              |                          gender                           |
  ## ----------------------------------------------------------------------------
  ## |       device |       female |         male |           NA |    Row Total |
  ## ----------------------------------------------------------------------------
  ## |      Desktop |        32803 |        46418 |        98061 |       177282 |
  ## |              |        0.134 |         0.19 |        0.401 |              |
  ## |              |         0.19 |         0.26 |         0.55 |         0.73 |
  ## |              |         0.81 |         0.75 |         0.69 |              |
  ## ----------------------------------------------------------------------------
  ## |       Mobile |         7268 |        14503 |        41711 |        63482 |
  ## |              |         0.03 |        0.059 |        0.171 |              |
  ## |              |         0.11 |         0.23 |         0.66 |         0.26 |
  ## |              |         0.18 |         0.24 |         0.29 |              |
  ## ----------------------------------------------------------------------------
  ## |       Tablet |          494 |          696 |         2444 |         3634 |
  ## |              |        0.002 |        0.003 |         0.01 |              |
  ## |              |         0.14 |         0.19 |         0.67 |         0.01 |
  ## |              |         0.01 |         0.01 |         0.02 |              |
  ## ----------------------------------------------------------------------------
  ## | Column Total |        40565 |        61617 |       142216 |       244398 |
  ## |              |        0.166 |        0.252 |        0.582 |              |
  ## ----------------------------------------------------------------------------

  We list and explore different R packages for summarizing categorical data in our GitHub repository.

  Key Functions

  

Table of Contents

• Resources
• Introduction
• Case Study
• Factors

Resources

You can download all the data sets, R scripts, practice questions and their solutions from our GitHub repository.

Introduction

Before we begin our deep dive on categorical data, let us get a quick overview of different data types. Feel free to skip this section if you know the difference between nominal and ordinal data.

Case Study

As is the practice, throughout this series, we will work on a case study related to an e-commerce firm. As most of you would already be aware, a lot of data is captured when you go on the internet by the websites you browse as well as by third party cookies. Data collected is then used to display ads as well as to feed to recommendation algorithms.

The data used in the case study represents the basic information that is captured when users visit any website. It closely resembles real world data for an e-commerce store. We will try to generate insights about the visitors to be used by an imaginary marketing team for better targeting and promotion. The case study data set can be imported using the RStudio IDE or R code.

read_csv("analytics_raw.csv", 
         col_types = cols_only(device = col_factor(levels = c("Desktop", "Tablet", "Mobile")), 
                               gender = col_factor(levels = c("female", "male", "NA")), 
                               user_rating = col_factor(levels = c("1", "2", "3", "4", "5"),
                                                        ordered = TRUE)))

## New names:
## * `` -> ...1

## # A tibble: 244,398 x 3
##    device  gender user_rating
##    <fct>   <fct>  <ord>      
##  1 Desktop female 4          
##  2 Mobile  NA     5          
##  3 Desktop NA     4          
##  4 Desktop NA     5          
##  5 Desktop NA     4          
##  6 Mobile  NA     4          
##  7 Desktop NA     4          
##  8 Desktop NA     4          
##  9 Desktop female 5          
## 10 Desktop NA     4          
## # ... with 244,388 more rows

data <- readRDS('analytics.rds')
head(data)

## # A tibble: 6 x 19
##   device  os        browser user_type channel gender frequency recency page_depth
##   <fct>   <fct>     <fct>   <fct>     <fct>   <fct>      <dbl>   <dbl>      <dbl>
## 1 Desktop Windows   Chrome  New Visi~ Organi~ female         1       0          1
## 2 Mobile  iOS       Safari  Returnin~ Organi~ <NA>           3       1          1
## 3 Desktop Chrome OS Chrome  New Visi~ Direct  <NA>           1       0          5
## 4 Desktop Macintosh Chrome  Returnin~ Organi~ <NA>           2       0          1
## 5 Desktop Macintosh Chrome  Returnin~ Referr~ <NA>           5       8          1
## 6 Mobile  Android   Chrome  New Visi~ Organi~ <NA>           1       0          5
## # ... with 10 more variables: hour_of_day <chr>, age <dbl>, duration <dbl>,
## #   landing_page <fct>, exit_page <fct>, country <fct>, quantity <dbl>,
## #   revenue <dbl>, purchase_flag <lgl>, user_rating <dbl>

Factors

In this very important section, we will learn how R

• stores categorical data
• checks if given data is categorical
• converts other data types to factor
• handles missing values in categorical data
• specifies the orders of the categories/levels
• stores ordinal data

In R, categorical data is stored as factor. Before we explore the factor family of functions, let us generate the sample data we will use in this module. We will generate the device column from the case study data set using the sample() function. We provide the following inputs to generate the data:

• values from which the data must be generated
• the size of the sample
• indicate if the values must be repeated (TRUE/FALSE)

device <- sample(c("Desktop", "Mobile", "Tablet"), size = 25, replace = TRUE)
device

##  [1] "Mobile"  "Mobile"  "Tablet"  "Mobile"  "Mobile"  "Mobile"  "Tablet" 
##  [8] "Tablet"  "Desktop" "Mobile"  "Mobile"  "Mobile"  "Tablet"  "Mobile" 
## [15] "Desktop" "Tablet"  "Desktop" "Mobile"  "Mobile"  "Tablet"  "Desktop"
## [22] "Mobile"  "Tablet"  "Mobile"  "Mobile"

is.factor(device)

## [1] FALSE

as.factor(device)

##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet

as_factor(device)

##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Mobile Tablet Desktop

factor(device)

##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet

factor(device, levels = c("Desktop", "Mobile", "Tablet"))

##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet

factor(device, levels = c("Desktop", "Mobile"))

##  [1] Mobile  Mobile  <NA>    Mobile  Mobile  Mobile  <NA>    <NA>    Desktop
## [10] Mobile  Mobile  Mobile  <NA>    Mobile  Desktop <NA>    Desktop Mobile 
## [19] Mobile  <NA>    Desktop Mobile  <NA>    Mobile  Mobile 
## Levels: Desktop Mobile

factor(device, 
       levels = c("Desktop", "Mobile", "Tablet"),
       labels = c("Desk", "Mob", "Tab"))

##  [1] Mob  Mob  Tab  Mob  Mob  Mob  Tab  Tab  Desk Mob  Mob  Mob  Tab  Mob  Desk
## [16] Tab  Desk Mob  Mob  Tab  Desk Mob  Tab  Mob  Mob 
## Levels: Desk Mob Tab

# sample with missing values
device <- sample(c("Desktop", "Mobile", "Tablet", NA), size = 25, replace = TRUE)
device

##  [1] "Desktop" "Tablet"  "Tablet"  NA        "Mobile"  NA        "Desktop"
##  [8] "Desktop" "Tablet"  "Mobile"  "Tablet"  "Desktop" "Desktop" "Mobile" 
## [15] NA        "Desktop" "Mobile"  "Tablet"  "Mobile"  "Mobile"  "Tablet" 
## [22] NA        "Desktop" "Tablet"  "Mobile"

# store as categorical data
factor(device)

##  [1] Desktop Tablet  Tablet  <NA>    Mobile  <NA>    Desktop Desktop Tablet 
## [10] Mobile  Tablet  Desktop Desktop Mobile  <NA>    Desktop Mobile  Tablet 
## [19] Mobile  Mobile  Tablet  <NA>    Desktop Tablet  Mobile 
## Levels: Desktop Mobile Tablet

factor(device, exclude = NULL)

##  [1] Desktop Tablet  Tablet  <NA>    Mobile  <NA>    Desktop Desktop Tablet 
## [10] Mobile  Tablet  Desktop Desktop Mobile  <NA>    Desktop Mobile  Tablet 
## [19] Mobile  Mobile  Tablet  <NA>    Desktop Tablet  Mobile 
## Levels: Desktop Mobile Tablet <NA>

rating <- sample(c("Dislike", "Neutral", "Like"), size = 25, replace = TRUE)
rating

##  [1] "Like"    "Dislike" "Neutral" "Dislike" "Neutral" "Like"    "Neutral"
##  [8] "Dislike" "Dislike" "Dislike" "Dislike" "Neutral" "Dislike" "Neutral"
## [15] "Dislike" "Dislike" "Like"    "Neutral" "Dislike" "Neutral" "Like"   
## [22] "Neutral" "Dislike" "Like"    "Dislike"

is.ordered(rating)

## [1] FALSE

as.ordered(rating)

##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike < Like < Neutral

factor(rating, ordered = TRUE)

##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike < Like < Neutral

factor(rating, levels = c("Dislike", "Neutral", "Like"), ordered = TRUE)

##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike < Neutral < Like

ordered(rating, levels = c("Dislike", "Neutral", "Like"))

##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike < Neutral < Like

Table of Contents

• Resources
• Introduction
• Case Study
• Date & Time Classes
• Date Arithmetic
• Timezones & Daylight Savings
• Date & Time Formats
• Parse/Read Date & Time
• Date & Time Components
• Create, Update & Verify
• Intervals, Durations & Period
• Round & Rollback
• References

Resources

Below are the links to all the resources related to this tutorial:

• Slides
• Code & Data
• RStudio Cloud Project
• ebook

Introduction

Sys.Date()

## [1] "2020-06-10"

lubridate::today()

## [1] "2020-06-10"

Sys.time()

## [1] "2020-06-10 22:47:50 IST"

lubridate::now()

## [1] "2020-06-10 22:47:50 IST"

lubridate::now(tzone = "UTC")

## [1] "2020-06-10 17:17:50 UTC"

lubridate::am(now())

## [1] FALSE

lubridate::pm(now())

## [1] TRUE

Sys.Date()

## [1] "2020-06-10"

lubridate::leap_year(Sys.Date())

## [1] TRUE

Case Study

Throughout the tutorial, we will work on a case study related to transactions of an imaginary trading company. The data set includes information about invoice and payment dates.

transact <- readr::read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/transact.csv')

## # A tibble: 2,466 x 3
##    Invoice    Due        Payment   
##    <date>     <date>     <date>    
##  1 2013-01-02 2013-02-01 2013-01-15
##  2 2013-01-26 2013-02-25 2013-03-03
##  3 2013-07-03 2013-08-02 2013-07-08
##  4 2013-02-10 2013-03-12 2013-03-17
##  5 2012-10-25 2012-11-24 2012-11-28
##  6 2012-01-27 2012-02-26 2012-02-22
##  7 2013-08-13 2013-09-12 2013-09-09
##  8 2012-12-16 2013-01-15 2013-01-12
##  9 2012-05-14 2012-06-13 2012-07-01
## 10 2013-07-01 2013-07-31 2013-07-26
## # ... with 2,456 more rows

Date & Time Classes

In this section, we will look at two things. First, how to create date/time data in R, and second, how to convert other data types to date/time. Let us begin by creating the release date of R 3.6.2.

release_date <- 2019-12-12
release_date

## [1] 1995

Okay! Why do we see 1995 when we call the date? What is happening here? Let us quickly check the data type of release_date.

class(release_date)

## [1] "numeric"

The data type is numeric i.e. R has subtracted 12 twice from 2019 to return 1995. Clearly, the above method is not the right way to store date/time. Let us see if we can get some hints from the built-in R functions we used in the previous section. If you observe the output, all of them returned date/time wrapped in quotes. Hmmm… let us wrap our date in quotes and see what happens.

release_date <- "2019-12-12"
release_date

## [1] "2019-12-12"

Alright, now R does not do any arithmetic and returns the date as we specified. Great! Is this the right format to store date/time then? No. Why? What is the problem if date/time is saved as character/string? The problem is the nature or type of operations done on date/time are different when compared to string/character, number or logical values.

• how do we add/subtract dates?
• how do we extract components such as year, month, day etc.

To answer the above questions, we will first check the data type of Sys.Date() and now().

class(Sys.Date())

## [1] "Date"

class(lubridate::now())

## [1] "POSIXct" "POSIXt"

class(release_date)

## [1] "character"

As you can see from the above output, there are 3 different classes for storing date/time in R

• Date
• POSIXct
• POSIXlt

Let us explore each of the above classes one by one.

Sys.Date()

## [1] "2020-06-10"

unclass(Sys.Date())

## [1] 18423

release_date <- as.Date("2019-12-12")
release_date

## [1] "2019-12-12"

release_date <- lubridate::as_date("2019-12-12")
release_date

## [1] "2019-12-12"

release_date - as.Date("1970-01-01")

## Time difference of 18242 days

unclass(release_date)

## [1] 18242

lubridate::origin

## [1] "1970-01-01 UTC"

unclass(as.Date("1963-08-28"))

## [1] -2318

as.Date(18242, origin = "1970-01-01")

## [1] "2019-12-12"

as.Date(7285, origin = "2000-01-01")

## [1] "2019-12-12"

ISOdate(year  = 2019,
        month = 12,
        day   = 12,
        hour  = 8,
        min   = 5, 
        sec   = 3,
        tz    = "UTC")

## [1] "2019-12-12 08:05:03 UTC"

class(lubridate::now())

## [1] "POSIXct" "POSIXt"

unclass(lubridate::now())

## [1] 1591809475
## attr(,"tzone")
## [1] ""

release_date <- as.POSIXct("2019-12-12 08:05:03")
class(release_date)

## [1] "POSIXct" "POSIXt"

unclass(release_date) 

## [1] 1576118103
## attr(,"tzone")
## [1] ""

release_date <- as.POSIXlt("2019-12-12 08:05:03")
release_date

## [1] "2019-12-12 08:05:03 IST"

release_date <- as.POSIXlt("2019-12-12 08:05:03")
unclass(release_date)

## $sec
## [1] 3
## 
## $min
## [1] 5
## 
## $hour
## [1] 8
## 
## $mday
## [1] 12
## 
## $mon
## [1] 11
## 
## $year
## [1] 119
## 
## $wday
## [1] 4
## 
## $yday
## [1] 345
## 
## $isdst
## [1] 0
## 
## $zone
## [1] "IST"
## 
## $gmtoff
## [1] NA

release_date <- as.POSIXlt("2019-12-12 08:05:03")
unlist(release_date)

##    sec    min   hour   mday    mon   year   wday   yday  isdst   zone gmtoff 
##    "3"    "5"    "8"   "12"   "11"  "119"    "4"  "345"    "0"  "IST"     NA

## [1] 8

release_date$mon

## [1] 11

release_date$zone

## [1] "IST"

Resources

Below are the links to all the resources related to this tutorial:

• Slides
• Code & Data
• RStudio Cloud Project
• ebook

Introduction

Sys.Date()

## [1] "2021-02-03"

lubridate::today()

## [1] "2021-02-03"

Sys.time()

## [1] "2021-02-03 18:54:23 IST"

lubridate::now()

## [1] "2021-02-03 18:54:23 IST"

lubridate::now(tzone = "UTC")

## [1] "2021-02-03 13:24:23 UTC"

lubridate::am(now())

## [1] FALSE

lubridate::pm(now())

## [1] TRUE

Sys.Date()

## [1] "2021-02-03"

lubridate::leap_year(Sys.Date())

## [1] FALSE

Case Study

Throughout the tutorial, we will work on a case study related to transactions of an imaginary trading company. The data set includes information about invoice and payment dates.

transact <- readr::read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/transact.csv')

## # A tibble: 2,466 x 3
##    Invoice    Due        Payment   
##    <date>     <date>     <date>    
##  1 2013-01-02 2013-02-01 2013-01-15
##  2 2013-01-26 2013-02-25 2013-03-03
##  3 2013-07-03 2013-08-02 2013-07-08
##  4 2013-02-10 2013-03-12 2013-03-17
##  5 2012-10-25 2012-11-24 2012-11-28
##  6 2012-01-27 2012-02-26 2012-02-22
##  7 2013-08-13 2013-09-12 2013-09-09
##  8 2012-12-16 2013-01-15 2013-01-12
##  9 2012-05-14 2012-06-13 2012-07-01
## 10 2013-07-01 2013-07-31 2013-07-26
## # ... with 2,456 more rows

Introduction

This is the second post in the series R & Databases. You can find the links to the first post of this series below:

• Quick Guide: R & SQLite

In this post, we will learn to query data from a database using dplyr.

Libraries, Code & Data

We will use the following libraries in this post:

• DBI
• RSQLite
• dbplyr
• dplyr

All the data sets used in this post can be found here and code can be downloaded from here.

con <- dbConnect(RSQLite::SQLite(), ":memory:")

dplyr::copy_to(con, mtcars)

mtcars2 <- dplyr::tbl(con, "mtcars")
mtcars2

## # Source:   table<mtcars> [?? x 11]
## # Database: sqlite 3.30.1 [:memory:]
##      mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##    <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  21       6  160    110  3.9   2.62  16.5     0     1     4     4
##  2  21       6  160    110  3.9   2.88  17.0     0     1     4     4
##  3  22.8     4  108     93  3.85  2.32  18.6     1     1     4     1
##  4  21.4     6  258    110  3.08  3.22  19.4     1     0     3     1
##  5  18.7     8  360    175  3.15  3.44  17.0     0     0     3     2
##  6  18.1     6  225    105  2.76  3.46  20.2     1     0     3     1
##  7  14.3     8  360    245  3.21  3.57  15.8     0     0     3     4
##  8  24.4     4  147.    62  3.69  3.19  20       1     0     4     2
##  9  22.8     4  141.    95  3.92  3.15  22.9     1     0     4     2
## 10  19.2     6  168.   123  3.92  3.44  18.3     1     0     4     4
## # ... with more rows

select(mtcars2, mpg, cyl, drat)

## # Source:   lazy query [?? x 3]
## # Database: sqlite 3.30.1 [:memory:]
##      mpg   cyl  drat
##    <dbl> <dbl> <dbl>
##  1  21       6  3.9 
##  2  21       6  3.9 
##  3  22.8     4  3.85
##  4  21.4     6  3.08
##  5  18.7     8  3.15
##  6  18.1     6  2.76
##  7  14.3     8  3.21
##  8  24.4     4  3.69
##  9  22.8     4  3.92
## 10  19.2     6  3.92
## # ... with more rows

filter(mtcars2, mpg > 25)

## # Source:   lazy query [?? x 11]
## # Database: sqlite 3.30.1 [:memory:]
##     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb
##   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1  32.4     4  78.7    66  4.08  2.2   19.5     1     1     4     1
## 2  30.4     4  75.7    52  4.93  1.62  18.5     1     1     4     2
## 3  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1
## 4  27.3     4  79      66  4.08  1.94  18.9     1     1     4     1
## 5  26       4 120.     91  4.43  2.14  16.7     0     1     5     2
## 6  30.4     4  95.1   113  3.77  1.51  16.9     1     1     5     2

mtcars2 %>%
  group_by(cyl) %>%
  summarise(mileage = mean(mpg))

## Warning: Missing values are always removed in SQL.
## Use `mean(x, na.rm = TRUE)` to silence this warning
## This warning is displayed only once per session.

## # Source:   lazy query [?? x 2]
## # Database: sqlite 3.30.1 [:memory:]
##     cyl mileage
##   <dbl>   <dbl>
## 1     4    26.7
## 2     6    19.7
## 3     8    15.1

mileages <- 
  mtcars2 %>%
  group_by(cyl) %>%
  summarise(mileage = mean(mpg))

dplyr::show_query(mileages)
## <SQL>
## SELECT `cyl`, AVG(`mpg`) AS `mileage`
## FROM `mtcars`
## GROUP BY `cyl`

dplyr::explain(mileages)
## <SQL>
## SELECT `cyl`, AVG(`mpg`) AS `mileage`
## FROM `mtcars`
## GROUP BY `cyl`
## 
## <PLAN>
##   id parent notused                       detail
## 1  6      0       0            SCAN TABLE mtcars
## 2  8      0       0 USE TEMP B-TREE FOR GROUP BY

dplyr::collect(mileages)

## # A tibble: 3 x 2
##     cyl mileage
##   <dbl>   <dbl>
## 1     4    26.7
## 2     6    19.7
## 3     8    15.1

References

• https://dbplyr.tidyverse.org/

Up Next..

In the next post, we will learn basic SQL commands.

Introduction

This is the first post in the series R & Databases. You can find the links to the other two posts of this series below:

• Data Wrangling with dbplyr
• SQL for Data Science - Part 1
• SQL for Data Science - Part 2

In this post, we will learn to:

• connect to a SQLite database from R
• display database information
• list tables in the database
• query data
  • read entire table
  • read few rows
  • read data in batches
• create table in database
• overwrite table in database
• append data to table in database
• remove table from database
• generate SQL query
• close database connection

Libraries, Code & Data

We will use the following libraries in this post:

• DBI
• RSQLite

All the data sets used in this post can be found here and code can be downloaded from here.

con <- dbConnect(RSQLite::SQLite(), ":memory:")

summary(con)

##           Length            Class             Mode 
##                1 SQLiteConnection               S4

dbListTables(con)

## [1] "ecom"         "sqlite_stat1" "sqlite_stat4"

dbListFields(con, "ecom")

## [1] "referrer" "device"   "bouncers" "n_visit"  "n_pages"  "duration"

Querying Data

The main objectives of connecting to a database are to:

• query data from the tables already present
• create new tables
• overwrite existing tables
• delete existing tables

Let us begin with querying data. We can do this in the following ways:

• read an entire table at once
• read few rows from a table
• read data in batches

dbReadTable(con, 'ecom')

##    referrer device bouncers n_visit n_pages duration
## 1    google laptop        1      10       1      693
## 2     yahoo tablet        1       9       1      459
## 3    direct laptop        1       0       1      996
## 4      bing tablet        0       3      18      468
## 5     yahoo mobile        1       9       1      955
## 6     yahoo laptop        0       5       5      135
## 7     yahoo mobile        1      10       1       75
## 8    direct mobile        1      10       1      908
## 9      bing mobile        0       3      19      209
## 10   google mobile        1       6       1      208
## 11   direct laptop        1       9       1      738
## 12   direct tablet        0       6      12      132
## 13   direct mobile        0       9      14      406
## 14    yahoo tablet        0       5       8       80
## 15    yahoo mobile        0       7       1       19
## 16     bing laptop        1       1       1      995
## 17     bing tablet        0       5      16      368
## 18   google tablet        1       7       1      406
## 19   social tablet        0       7      10      290
## 20   social tablet        0       2       1       28

dbGetQuery(con, "select * from ecom limit 10")

##    referrer device bouncers n_visit n_pages duration
## 1    google laptop        1      10       1      693
## 2     yahoo tablet        1       9       1      459
## 3    direct laptop        1       0       1      996
## 4      bing tablet        0       3      18      468
## 5     yahoo mobile        1       9       1      955
## 6     yahoo laptop        0       5       5      135
## 7     yahoo mobile        1      10       1       75
## 8    direct mobile        1      10       1      908
## 9      bing mobile        0       3      19      209
## 10   google mobile        1       6       1      208

query <- dbSendQuery(con, 'select * from ecom')
result <- dbFetch(query, n = 15)
result

##    referrer device bouncers n_visit n_pages duration
## 1    google laptop        1      10       1      693
## 2     yahoo tablet        1       9       1      459
## 3    direct laptop        1       0       1      996
## 4      bing tablet        0       3      18      468
## 5     yahoo mobile        1       9       1      955
## 6     yahoo laptop        0       5       5      135
## 7     yahoo mobile        1      10       1       75
## 8    direct mobile        1      10       1      908
## 9      bing mobile        0       3      19      209
## 10   google mobile        1       6       1      208
## 11   direct laptop        1       9       1      738
## 12   direct tablet        0       6      12      132
## 13   direct mobile        0       9      14      406
## 14    yahoo tablet        0       5       8       80
## 15    yahoo mobile        0       7       1       19

Query

dbHasCompleted(query)

## [1] FALSE

dbGetInfo(query)

## $statement
## [1] "select * from ecom"
## 
## $row.count
## [1] 15
## 
## $rows.affected
## [1] 0
## 
## $has.completed
## [1] FALSE

dbGetStatement(query)

## [1] "select * from ecom"

dbGetRowCount(query)

## [1] 15

dbGetRowsAffected(query)

## [1] 0

dbColumnInfo(query)

##       name      type
## 1 referrer character
## 2   device character
## 3 bouncers   integer
## 4  n_visit    double
## 5  n_pages    double
## 6 duration    double

Create Table

So far we have explored querying data from an existing table. Now, let us turn our attention to creating new tables in the database.

x <- 1:10
y <- letters[1:10]
trial <- tibble::tibble(x, y)
dbWriteTable(con, "trial", trial)

## Warning: Closing open result set, pending rows

dbListTables(con)

## [1] "ecom"         "sqlite_stat1" "sqlite_stat4" "trial"

dbExistsTable(con, "trial")

## [1] TRUE

dbGetQuery(con, "select * from trial limit 5")

##   x y
## 1 1 a
## 2 2 b
## 3 3 c
## 4 4 d
## 5 5 e

x <- sample(100, 10)
y <- letters[11:20]
trial2 <- tibble::tibble(x, y)
dbWriteTable(con, "trial", trial2, overwrite = TRUE)

dbGetQuery(con, "select * from trial limit 5")

##    x y
## 1 48 k
## 2 58 l
## 3 85 m
## 4 99 n
## 5 78 o

x <- sample(100, 10)
y <- letters[5:14]
trial3 <- tibble::tibble(x, y)
dbWriteTable(con, "trial", trial3, append = TRUE)

dbReadTable(con, "trial")

##     x y
## 1  48 k
## 2  58 l
## 3  85 m
## 4  99 n
## 5  78 o
## 6   9 p
## 7   6 q
## 8  59 r
## 9  11 s
## 10 38 t
## 11 39 e
## 12 69 f
## 13 43 g
## 14 71 h
## 15 99 i
## 16 56 j
## 17 45 k
## 18 81 l
## 19 93 m
## 20 47 n

sqlAppendTable(con, "ecom", head(ecom))

## Warning: Do not rely on the default value of the row.names argument for
## sqlAppendTable(), it will change in the future.

## <SQL> INSERT INTO `ecom`
##   (`referrer`, `device`, `bouncers`, `n_visit`, `n_pages`, `duration`)
## VALUES
##   ('google', 'laptop', TRUE, 10, 1, 693),
##   ('yahoo', 'tablet', TRUE, 9, 1, 459),
##   ('direct', 'laptop', TRUE, 0, 1, 996),
##   ('bing', 'tablet', FALSE, 3, 18, 468),
##   ('yahoo', 'mobile', TRUE, 9, 1, 955),
##   ('yahoo', 'laptop', FALSE, 5, 5, 135)

Insert Rows

# use dbExecute
dbExecute(con,
  "INSERT into trial (x, y) VALUES (32, 'c'), (45, 'k'), (61, 'h')"
)
## [1] 3

# use dbSendStatement
dbSendStatement(con,
  "INSERT into trial (x, y) VALUES (25, 'm'), (54, 'l'), (16, 'y')"
)
## <SQLiteResult>
##   SQL  INSERT into trial (x, y) VALUES (25, 'm'), (54, 'l'), (16, 'y')
##   ROWS Fetched: 0 [complete]
##        Changed: 3

dbRemoveTable(con, "trial")

## Warning: Closing open result set, pending rows

SQLite Data Type

If you want to know the data type, use dbDataType().

dbDataType(RSQLite::SQLite(), "a")

## [1] "TEXT"

dbDataType(RSQLite::SQLite(), 1:5)

## [1] "INTEGER"

dbDataType(RSQLite::SQLite(), 1.5)

## [1] "REAL"

dbDisconnect(con)

References

• https://dbi.r-dbi.org/

Introduction

In this post, we will learn to work with categorical/qualitative data in R using forcats. Let us begin by installing and loading forcats and a set of other pacakges we will be using.

Libraries & Code

We will use the following packages:

• forcats
• dplyr
• magrittr
• ggplot2
• tibbe
• purrr
• and readr

The codes from here.

library(forcats)
library(tibble)
library(magrittr)
library(purrr)
library(dplyr)
library(ggplot2)
library(readr)

Case Study

We will use a case study to explore the various features of the forcats package. You can download the data for the case study from here or directly import the data using the readr package. We will do the following in this case study:

• compute the frequency of different referrers
• plot average number of pages browsed for different referrers
• collapse referrers with low sample size into a single group
• club traffic from social media websites into a new category
• group referrers with traffic below a threshold into a single category

ecom <- 
  read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/web.csv',
    col_types = cols_only(
      referrer = col_factor(levels = c("bing", "direct", "social", "yahoo", "google")),
      n_pages = col_double(), duration = col_double()
    )
  )

ecom

## # A tibble: 1,000 x 3
##    referrer n_pages duration
##    <fct>      <dbl>    <dbl>
##  1 google         1      693
##  2 yahoo          1      459
##  3 direct         1      996
##  4 bing          18      468
##  5 yahoo          1      955
##  6 yahoo          5      135
##  7 yahoo          1       75
##  8 direct         1      908
##  9 bing          19      209
## 10 google         1      208
## # ... with 990 more rows

referrers <- use_series(ecom, referrer)

Tabulate Referrers

Let us look at the traffic driven by different referrer types.

fct_count(referrers)

## # A tibble: 5 x 2
##   f          n
##   <fct>  <int>
## 1 bing     194
## 2 direct   191
## 3 social   200
## 4 yahoo    207
## 5 google   208

If you want to sort the output in descending order, use sort and set it to TRUE.

fct_count(referrers, sort = TRUE)

## # A tibble: 5 x 2
##   f          n
##   <fct>  <int>
## 1 google   208
## 2 yahoo    207
## 3 social   200
## 4 bing     194
## 5 direct   191

Use fct_unique to view the categories or levels of the referrer variable.

fct_unique(referrers)

## [1] bing   direct social yahoo  google
## Levels: bing direct social yahoo google

Reorder Referrers

We want to examine the average number of pages visited by each referrer type.

refer_summary <- 
  ecom %>%
  group_by(referrer) %>%
  summarise(
    page = mean(n_pages),
    tos = mean(duration),
    n = n()
  )

## `summarise()` ungrouping (override with `.groups` argument)

refer_summary

## # A tibble: 5 x 4
##   referrer  page   tos     n
## * <fct>    <dbl> <dbl> <int>
## 1 bing      6.13  368.   194
## 2 direct    6.38  358.   191
## 3 social    5.42  355.   200
## 4 yahoo     5.99  336.   207
## 5 google    5.73  360.   208

Let us plot the average number of pages visited by each referrer type.

refer_summary %>%
  ggplot() +
  geom_point(aes(page, referrer))

Use fct_reorder to reorder the referrer types by the average number of pages visited.

refer_summary %>%
  ggplot() +
  geom_point(aes(page, fct_reorder(referrer, page)))

Plot Referrer Frequency (Descending Order)

Since we want to plot the referrers in descending order of frequency, we will use fct_infreq() to reorder by frequency.

referrers %>%
  fct_infreq() %>%
  fct_unique()

## [1] google yahoo  social bing   direct
## Levels: google yahoo social bing direct

Now that we know how to reorder categories/levels by frequency, let us reorder the referrers by frequency and plot them.

ecom %>%
  mutate(
    ref = referrer %>% 
      fct_infreq()
  ) %>%
  ggplot(aes(ref)) +
  geom_bar()

Plot Referrer Frequency (Ascending Order)

Let us look at the categories of the referrer variable.

fct_unique(referrers)

## [1] bing   direct social yahoo  google
## Levels: bing direct social yahoo google

Since we want to plot the referrers in ascending order of frequency, we will use fct_rev() to reverse the order.

referrers %>%
  fct_rev() %>%
  fct_unique()

## [1] google yahoo  social direct bing  
## Levels: google yahoo social direct bing

Let us reorder the referrers by frequency first and then reverse the order before plotting their frequencies.

ecom %>%
  mutate(
    ref = referrer %>% 
      fct_infreq() %>% 
      fct_rev()
  ) %>%
  ggplot(aes(ref)) +
  geom_bar()

Case Study 2

In this case study, we will learn to:

• combine categories
• recategorize

The data set we will use has just one column traffics i.e. the source of traffic for a imaginary website.

traffic <- 
  read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/web_traffic.csv',
    col_types = list(
      col_factor(levels = c("affiliates", "bing", "direct", "facebook", 
        "yahoo", "google", "instagram", "twitter", "unknown")
      )
    )
  )

traffic

## # A tibble: 48,232 x 1
##    traffics
##    <fct>   
##  1 google  
##  2 google  
##  3 google  
##  4 google  
##  5 google  
##  6 google  
##  7 google  
##  8 google  
##  9 google  
## 10 google  
## # ... with 48,222 more rows

traffics <- use_series(traffic, traffics)

Tabulate Referrer

Let us compute the traffic driven by different referrers using fct_count.

fct_count(traffics)   

## # A tibble: 9 x 2
##   f              n
##   <fct>      <int>
## 1 affiliates  7641
## 2 bing        5893
## 3 direct      1350
## 4 facebook    8135
## 5 yahoo       4899
## 6 google      9229
## 7 instagram   3907
## 8 twitter     4521
## 9 unknown     2657

Collapse Referrer Categories

We want to group some of the referrers into 2 categories:

• social
• search

To group categories/levels, we will use fct_collapse().

traffics %>%
  fct_collapse(
    social = c("facebook", "twitter", "instagram"),
    search = c("google", "bing", "yahoo")
  ) %>% 
  fct_count()

## # A tibble: 5 x 2
##   f              n
##   <fct>      <int>
## 1 affiliates  7641
## 2 search     20021
## 3 direct      1350
## 4 social     16563
## 5 unknown     2657

The above result can be achieved using fct_recode() as shown below:

fct_recode(traffics, 
  search = "bing", 
  search = "yahoo", 
  search = "google",
  social = "facebook", 
  social = "twitter", 
  social = "instagram") %>%
  levels()

## [1] "affiliates" "search"     "direct"     "social"     "unknown"

Lump Infrequent Referrer Types

Let us group together referrer types that drive low traffic to the website. Use fct_lump() to lump together categories.

fct_count(traffics)
## # A tibble: 9 x 2
##   f              n
##   <fct>      <int>
## 1 affiliates  7641
## 2 bing        5893
## 3 direct      1350
## 4 facebook    8135
## 5 yahoo       4899
## 6 google      9229
## 7 instagram   3907
## 8 twitter     4521
## 9 unknown     2657

traffics %>% 
  fct_lump() %>% 
  table()
## .
## affiliates       bing   facebook      yahoo     google  instagram    twitter 
##       7641       5893       8135       4899       9229       3907       4521 
##    unknown      Other 
##       2657       1350

Retain top 3 referrers

We want to retain the top 3 referrers and combine the rest of them into a single category.

## # A tibble: 9 x 2
##   f              n
##   <fct>      <int>
## 1 google      9229
## 2 facebook    8135
## 3 affiliates  7641
## 4 bing        5893
## 5 yahoo       4899
## 6 twitter     4521
## 7 instagram   3907
## 8 unknown     2657
## 9 direct      1350

Use fct_lump() and set the argument n to 3 indicating we want to retain top 3 categories and combine the rest.

traffics %>% 
  fct_lump(n = 3) %>% 
  table()

## .
## affiliates   facebook     google      Other 
##       7641       8135       9229      23227

Lump Referrer Types with less than 10% traffic

Let us combine referrers that drive less than 10% traffic to the website.

## # A tibble: 9 x 3
##   f              n percent
##   <fct>      <int>   <dbl>
## 1 affiliates  7641   15.8 
## 2 bing        5893   12.2 
## 3 direct      1350    2.8 
## 4 facebook    8135   16.9 
## 5 yahoo       4899   10.2 
## 6 google      9229   19.1 
## 7 instagram   3907    8.1 
## 8 twitter     4521    9.37
## 9 unknown     2657    5.51

Since we are looking at proportion of traffic driven to the website and not the actual numbers, we use the prop argument and set it to 0.1, indicating that we want to retain only those categories which have a proportion of more than 10% and combine the rest.

traffics %>%
  fct_lump(prop = 0.1) %>% 
  table()

## .
## affiliates       bing   facebook      yahoo     google      Other 
##       7641       5893       8135       4899       9229      12435

Retain 3 Referrer Types with lowest traffic

What if we want to retain 3 referrers which drive the lowest traffic to the website and combine the rest?

## # A tibble: 9 x 2
##   f              n
##   <fct>      <int>
## 1 direct      1350
## 2 unknown     2657
## 3 instagram   3907
## 4 twitter     4521
## 5 yahoo       4899
## 6 bing        5893
## 7 affiliates  7641
## 8 facebook    8135
## 9 google      9229

We will still use the n argument but instead of specifying 3, we now specify -3.

traffics %>% 
  fct_lump(n = -3) %>% 
  table()

## .
##    direct instagram   unknown     Other 
##      1350      3907      2657     40318

Retain 3 Referrer Types with less than 10% traffic

Let us see how to retain referrers that drive less than 10 % traffic to the website and combine the rest into a single group.

## # A tibble: 9 x 3
##   f              n percent
##   <fct>      <int>   <dbl>
## 1 affiliates  7641   15.8 
## 2 bing        5893   12.2 
## 3 direct      1350    2.8 
## 4 facebook    8135   16.9 
## 5 yahoo       4899   10.2 
## 6 google      9229   19.1 
## 7 instagram   3907    8.1 
## 8 twitter     4521    9.37
## 9 unknown     2657    5.51

Instead of setting prop to 0.1, we will set it to -0.1.

traffics %>% 
  fct_lump(prop = -0.1) %>% 
  table()

## .
##    direct instagram   twitter   unknown     Other 
##      1350      3907      4521      2657     35797

Replace Levels

Let us assume we want to retain a couple of important categories and group the rest into a single category. In the below example, we retain google and yahoo while grouping the rest as others using fct_other().

fct_other(traffics, keep = c("google", "yahoo")) %>%
  levels()

## [1] "yahoo"  "google" "Other"

Drop Levels

What if you want to drop a couple of categories instead of grouping them? Use the drop argument in fct_other() and specify the categories to be dropped. In the below example, we drop the following referrer categories:

• instagram
• twitter

fct_other(traffics, drop = c("instagram", "twitter")) %>%
  levels()

## [1] "affiliates" "bing"       "direct"     "facebook"   "yahoo"     
## [6] "google"     "unknown"    "Other"

Reorder Levels

The categories can be reordered using fct_relevel(). In the above example, we reorder the categories to ensure google appears first. Similarly in the below example, we reorder the levels to ensure twitter appears first irrespective of its frequency or order of appearance in the data.

fct_relevel(traffics, "twitter") %>%
  levels()

## [1] "twitter"    "affiliates" "bing"       "direct"     "facebook"  
## [6] "yahoo"      "google"     "instagram"  "unknown"

If the category needs to appear at a particular position, use the after argument and specify the position after which it should appear. For example, if google should be the third category, we would specify after = 2 i.e. google should come after the 2nd position (i.e. third position).

fct_relevel(traffics, "google", after = 2) %>%
  levels()

## [1] "affiliates" "bing"       "google"     "direct"     "facebook"  
## [6] "yahoo"      "instagram"  "twitter"    "unknown"

If the category should appear last, supply the value Inf (infinity) to the after argument as shown below.

fct_relevel(traffics, "facebook", after = Inf) %>%
  levels()

## [1] "affiliates" "bing"       "direct"     "yahoo"      "google"    
## [6] "instagram"  "twitter"    "unknown"    "facebook"

Case Study 3

In this case study, we deal with categorical data which is ordered and cyclical. It contains response to an imaginary survey.

response_data <- 
  read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/response.csv',
    col_types = list(col_factor(levels = c("like", "like somewhat", "neutral", 
      "dislike somewhat", "dislike"), ordered = TRUE)
    )
  )

responses <- use_series(response_data, response)
levels(responses)

## [1] "like"             "like somewhat"    "neutral"          "dislike somewhat"
## [5] "dislike"

Shift Levels

To shift the levels, we use fct_shift(). Use the n argument to indicate the direction of the shift. If n is positive, the levels are shifted to the left else to the right. In the below example, we shift the levels to the left by 2 positions.

fct_shift(responses, 2) %>%
  levels()

## [1] "neutral"          "dislike somewhat" "dislike"          "like"            
## [5] "like somewhat"

To shift the levels to the right, supply a negative value to the n argument in fct_shift(). In the below example, we shift the levels to the right by 2 positions.

fct_shift(responses, -2) %>%
  levels()

## [1] "dislike somewhat" "dislike"          "like"             "like somewhat"   
## [5] "neutral"

References

• https://forcats.tidyverse.org/index.html
• http://r4ds.had.co.nz/factors.html

Introduction

In this post, we will learn to work with date/time data in R using lubridate, an R package that makes it easy to work with dates and time. Let us begin by installing and loading the pacakge.

Libraries, Code & Data

We will use the following packages:

• lubridate
• dplyr
• magrittr
• readr

The data sets can be downloaded from here and the codes from here.

library(lubridate)
library(dplyr)
library(magrittr)
library(readr)

Quick Intro

origin

## [1] "1970-01-01 UTC"

now()

## [1] "2020-06-10 18:52:25 IST"

today()

## [1] "2020-06-10"

am(now())  

## [1] FALSE

pm(now())

## [1] TRUE

Case Study

transact <- read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/transact.csv')

## # A tibble: 2,466 x 3
##    Invoice    Due        Payment   
##    <date>     <date>     <date>    
##  1 2013-01-02 2013-02-01 2013-01-15
##  2 2013-01-26 2013-02-25 2013-03-03
##  3 2013-07-03 2013-08-02 2013-07-08
##  4 2013-02-10 2013-03-12 2013-03-17
##  5 2012-10-25 2012-11-24 2012-11-28
##  6 2012-01-27 2012-02-26 2012-02-22
##  7 2013-08-13 2013-09-12 2013-09-09
##  8 2012-12-16 2013-01-15 2013-01-12
##  9 2012-05-14 2012-06-13 2012-07-01
## 10 2013-07-01 2013-07-31 2013-07-26
## # ... with 2,456 more rows

Extract Date, Month & Year from Due Date

The first thing we will learn is to extract the date, month and year.

this_day <- as_date('2017-03-23')
day(this_day)

## [1] 23

month(this_day)

## [1] 3

year(this_day)

## [1] 2017

Let us now extract the date, month and year from the Due column.

transact %>%
  mutate(
    due_day   = day(Due),
    due_month = month(Due),
    due_year  = year(Due)
  )

## # A tibble: 2,466 x 6
##    Invoice    Due        Payment    due_day due_month due_year
##    <date>     <date>     <date>       <int>     <dbl>    <dbl>
##  1 2013-01-02 2013-02-01 2013-01-15       1         2     2013
##  2 2013-01-26 2013-02-25 2013-03-03      25         2     2013
##  3 2013-07-03 2013-08-02 2013-07-08       2         8     2013
##  4 2013-02-10 2013-03-12 2013-03-17      12         3     2013
##  5 2012-10-25 2012-11-24 2012-11-28      24        11     2012
##  6 2012-01-27 2012-02-26 2012-02-22      26         2     2012
##  7 2013-08-13 2013-09-12 2013-09-09      12         9     2013
##  8 2012-12-16 2013-01-15 2013-01-12      15         1     2013
##  9 2012-05-14 2012-06-13 2012-07-01      13         6     2012
## 10 2013-07-01 2013-07-31 2013-07-26      31         7     2013
## # ... with 2,456 more rows

Compute days to settle invoice

Time to do some arithmetic with the dates. Let us calculate the duration of a course by subtracting the course start date from the course end date.

course_start    <- as_date('2017-04-12')
course_end      <- as_date('2017-04-21')
course_duration <- course_end - course_start
course_duration
## Time difference of 9 days

Let us estimate the number of days to settle the invoice by subtracting the date of invoice from the date of payment.

transact %>%
  mutate(
    days_to_pay = Payment - Invoice
  )

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    days_to_pay
##    <date>     <date>     <date>     <drtn>     
##  1 2013-01-02 2013-02-01 2013-01-15 13 days    
##  2 2013-01-26 2013-02-25 2013-03-03 36 days    
##  3 2013-07-03 2013-08-02 2013-07-08  5 days    
##  4 2013-02-10 2013-03-12 2013-03-17 35 days    
##  5 2012-10-25 2012-11-24 2012-11-28 34 days    
##  6 2012-01-27 2012-02-26 2012-02-22 26 days    
##  7 2013-08-13 2013-09-12 2013-09-09 27 days    
##  8 2012-12-16 2013-01-15 2013-01-12 27 days    
##  9 2012-05-14 2012-06-13 2012-07-01 48 days    
## 10 2013-07-01 2013-07-31 2013-07-26 25 days    
## # ... with 2,456 more rows

Compute days over due

How many of the invoices were settled post the due date? We can find this by:

• subtracting the due date from the payment date
• counting the number of rows where delay < 0

transact %>%
  mutate(
    delay = Due - Payment
  ) %>%
  filter(delay < 0) %>%
  tally()

## # A tibble: 1 x 1
##       n
##   <int>
## 1   877

Is due year a leap year?

Just for fun, let us check if the due year happens to be a leap year.

transact %>%
  mutate(
    is_leap = leap_year(Due)
  )

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    is_leap
##    <date>     <date>     <date>     <lgl>  
##  1 2013-01-02 2013-02-01 2013-01-15 FALSE  
##  2 2013-01-26 2013-02-25 2013-03-03 FALSE  
##  3 2013-07-03 2013-08-02 2013-07-08 FALSE  
##  4 2013-02-10 2013-03-12 2013-03-17 FALSE  
##  5 2012-10-25 2012-11-24 2012-11-28 TRUE   
##  6 2012-01-27 2012-02-26 2012-02-22 TRUE   
##  7 2013-08-13 2013-09-12 2013-09-09 FALSE  
##  8 2012-12-16 2013-01-15 2013-01-12 FALSE  
##  9 2012-05-14 2012-06-13 2012-07-01 TRUE   
## 10 2013-07-01 2013-07-31 2013-07-26 FALSE  
## # ... with 2,456 more rows

If due day is February 29, is it a leap year?

Let us do some data sanitization. If the due day happens to be February 29, let us ensure that the due year is a leap year. Below are the steps to check if the due year is a leap year:

• we will extract the following from the due date:
  • day
  • month
  • year
• we will then create a new column is_leap which will have be set to TRUE if the year is a leap year else it will be set to FALSE
• filter all the payments due on 29th Feb
• select the following columns:
  • Due
  • is_leap

transact %>%
  mutate(
    due_day   = day(Due),
    due_month = month(Due),
    due_year  = year(Due),
    is_leap   = leap_year(due_year)
  ) %>%
  filter(due_month == 2 & due_day == 29) %>%
  select(Due, is_leap) 

## # A tibble: 4 x 2
##   Due        is_leap
##   <date>     <lgl>  
## 1 2012-02-29 TRUE   
## 2 2012-02-29 TRUE   
## 3 2012-02-29 TRUE   
## 4 2012-02-29 TRUE

Shift Date

Time to shift some dates. We can shift a date by days, weeks or months. Let us shift the course start date by:

• 2 days
• 3 weeks
• 1 year

course_start + days(2)
## [1] "2017-04-14"
course_start + weeks(3)
## [1] "2017-05-03"
course_start + years(1)
## [1] "2018-04-12"

Interval

Let us calculate the duration of the course using interval. If you observe carefully, the result is not the duration in days but an object of class interval. Now let us learn how we can use intervals.

interval(course_start, course_end)

## [1] 2017-04-12 UTC--2017-04-21 UTC

Intervals Overlap

Let us say you are planning a vacation and want to check if the vacation dates overlap with the course dates. You can do this by:

• creating vacation and course intervals
• use int_overlaps() to check if two intervals overlap. It returns TRUE if the intervals overlap else FALSE.

Let us use the vacation start and end dates to create vacation_interval and then check if it overlaps with course_interval.

vacation_start    <- as_date('2017-04-19')
vacation_end      <- as_date('2017-04-25')
course_interval   <- interval(course_start, course_end)
vacation_interval <- interval(vacation_start, vacation_end)
int_overlaps(course_interval, vacation_interval)
## [1] TRUE

How many invoices were settled within due date?

Let us use intervals to count the number of invoices that were settled within the due date. To do this, we will:

• create an interval for the invoice and due date
• create a new column due_next by incrementing the due date by 1 day
• another interval for due_next and the payment date
• if the intervals overlap, the payment was made within the due date

transact %>%
  mutate(
    inv_due_interval = interval(Invoice, Due),
    due_next         = Due + days(1),
    due_pay_interval = interval(due_next, Payment),
    overlaps         = int_overlaps(inv_due_interval, due_pay_interval)
  ) %>%
  select(Invoice, Due, Payment, overlaps)

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    overlaps
##    <date>     <date>     <date>     <lgl>   
##  1 2013-01-02 2013-02-01 2013-01-15 TRUE    
##  2 2013-01-26 2013-02-25 2013-03-03 FALSE   
##  3 2013-07-03 2013-08-02 2013-07-08 TRUE    
##  4 2013-02-10 2013-03-12 2013-03-17 FALSE   
##  5 2012-10-25 2012-11-24 2012-11-28 FALSE   
##  6 2012-01-27 2012-02-26 2012-02-22 TRUE    
##  7 2013-08-13 2013-09-12 2013-09-09 TRUE    
##  8 2012-12-16 2013-01-15 2013-01-12 TRUE    
##  9 2012-05-14 2012-06-13 2012-07-01 FALSE   
## 10 2013-07-01 2013-07-31 2013-07-26 TRUE    
## # ... with 2,456 more rows

Below we show another method to count the number of invoices paid within the due date. Instead of using days to change the due date, we use int_shift to shift it by 1 day.

transact %>%
  mutate(
    inv_due_interval = interval(Invoice, Due),
    due_pay_interval = interval(Due, Payment),  
    due_pay_next     = int_shift(due_pay_interval, by = days(1)),
    overlaps         = int_overlaps(inv_due_interval, due_pay_next)
  ) %>%
  select(Invoice, Due, Payment, overlaps)

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    overlaps
##    <date>     <date>     <date>     <lgl>   
##  1 2013-01-02 2013-02-01 2013-01-15 TRUE    
##  2 2013-01-26 2013-02-25 2013-03-03 FALSE   
##  3 2013-07-03 2013-08-02 2013-07-08 TRUE    
##  4 2013-02-10 2013-03-12 2013-03-17 FALSE   
##  5 2012-10-25 2012-11-24 2012-11-28 FALSE   
##  6 2012-01-27 2012-02-26 2012-02-22 TRUE    
##  7 2013-08-13 2013-09-12 2013-09-09 TRUE    
##  8 2012-12-16 2013-01-15 2013-01-12 TRUE    
##  9 2012-05-14 2012-06-13 2012-07-01 FALSE   
## 10 2013-07-01 2013-07-31 2013-07-26 TRUE    
## # ... with 2,456 more rows

You might be thinking why we incremented the due date by a day before creating the interval between the due day and the payment day. If we do not increment, both the intervals will share a common date i.e. the due date and they will always overlap as shown below:

transact %>%
  mutate(
    inv_due_interval = interval(Invoice, Due),
    due_pay_interval = interval(Due, Payment),
    overlaps         = int_overlaps(inv_due_interval, due_pay_interval)
  ) %>%
  select(Invoice, Due, Payment, overlaps)

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    overlaps
##    <date>     <date>     <date>     <lgl>   
##  1 2013-01-02 2013-02-01 2013-01-15 TRUE    
##  2 2013-01-26 2013-02-25 2013-03-03 TRUE    
##  3 2013-07-03 2013-08-02 2013-07-08 TRUE    
##  4 2013-02-10 2013-03-12 2013-03-17 TRUE    
##  5 2012-10-25 2012-11-24 2012-11-28 TRUE    
##  6 2012-01-27 2012-02-26 2012-02-22 TRUE    
##  7 2013-08-13 2013-09-12 2013-09-09 TRUE    
##  8 2012-12-16 2013-01-15 2013-01-12 TRUE    
##  9 2012-05-14 2012-06-13 2012-07-01 TRUE    
## 10 2013-07-01 2013-07-31 2013-07-26 TRUE    
## # ... with 2,456 more rows

Shift Interval

Intervals can be shifted too. In the below example, we shift the course interval by:

• 1 day
• 3 weeks
• 1 year

course_interval <- interval(course_start, course_end)
int_shift(course_interval, by = days(1))
## [1] 2017-04-13 UTC--2017-04-22 UTC
int_shift(course_interval, by = weeks(3))
## [1] 2017-05-03 UTC--2017-05-12 UTC
int_shift(course_interval, by = years(1))
## [1] 2018-04-12 UTC--2018-04-21 UTC

Within

Let us assume that we have to attend a conference in April 2017. Does it occur during the course duration? We can answer this using %within% which will return TRUE if a date falls within an interval.

conference <- as_date('2017-04-15')
conference %within% course_interval
## [1] TRUE

transact %>%
  mutate(
    inv_due_interval = interval(Invoice, Due),
    overlaps         = Payment %within% inv_due_interval
  ) %>%
  select(Due, Payment, overlaps)

## # A tibble: 2,466 x 3
##    Due        Payment    overlaps
##    <date>     <date>     <lgl>   
##  1 2013-02-01 2013-01-15 TRUE    
##  2 2013-02-25 2013-03-03 FALSE   
##  3 2013-08-02 2013-07-08 TRUE    
##  4 2013-03-12 2013-03-17 FALSE   
##  5 2012-11-24 2012-11-28 FALSE   
##  6 2012-02-26 2012-02-22 TRUE    
##  7 2013-09-12 2013-09-09 TRUE    
##  8 2013-01-15 2013-01-12 TRUE    
##  9 2012-06-13 2012-07-01 FALSE   
## 10 2013-07-31 2013-07-26 TRUE    
## # ... with 2,456 more rows

Quarter

Let us check the quarter and the semester in which the course starts.

course_start
## [1] "2017-04-12"
quarter(course_start)
## [1] 2
quarter(course_start, with_year = TRUE)
## [1] 2017.2
semester(course_start)  
## [1] 1

Let us count the invoices due for each quarter.

transact %>%
  mutate(
    quarter_due = quarter(Due)
  ) %>%
  count(quarter_due)

## # A tibble: 4 x 2
##   quarter_due     n
## *       <int> <int>
## 1           1   521
## 2           2   661
## 3           3   618
## 4           4   666

transact %>%
  mutate(
    Quarter = quarter(Due, with_year = TRUE)
  )

## # A tibble: 2,466 x 4
##    Invoice    Due        Payment    Quarter
##    <date>     <date>     <date>       <dbl>
##  1 2013-01-02 2013-02-01 2013-01-15   2013.
##  2 2013-01-26 2013-02-25 2013-03-03   2013.
##  3 2013-07-03 2013-08-02 2013-07-08   2013.
##  4 2013-02-10 2013-03-12 2013-03-17   2013.
##  5 2012-10-25 2012-11-24 2012-11-28   2012.
##  6 2012-01-27 2012-02-26 2012-02-22   2012.
##  7 2013-08-13 2013-09-12 2013-09-09   2013.
##  8 2012-12-16 2013-01-15 2013-01-12   2013.
##  9 2012-05-14 2012-06-13 2012-07-01   2012.
## 10 2013-07-01 2013-07-31 2013-07-26   2013.
## # ... with 2,456 more rows

course_interval / dseconds()
## [1] 777600
course_interval / dminutes()
## [1] 12960
course_interval / dhours()
## [1] 216
course_interval / dweeks()
## [1] 1.285714
course_interval / dyears()
## [1] 0.02464066

time_length(course_interval, unit = "seconds")
## [1] 777600
time_length(course_interval, unit = "minutes")
## [1] 12960
time_length(course_interval, unit = "hours")
## [1] 216

as.period(course_interval, unit = "seconds")
## [1] "777600S"
as.period(course_interval, unit = "minutes")
## [1] "12960M 0S"
as.period(course_interval, unit = "hours")
## [1] "216H 0M 0S"

References

• https://lubridate.tidyverse.org/
• http://r4ds.had.co.nz/dates-and-times.html

Introduction

In this post, we will learn to work with string data in R using stringr. As we did in the other posts, we will use a case study to explore the various features of the stringr package. Let us begin by installing and loading stringr and a set of other pacakges we will be using.

Libraries, Code & Data

We will use the following libraries:

• stringr
• dplyr
• magrittr
• tibble
• purrr
• and readr

The data sets can be downloaded from here and the codes from here.

library(stringr)
library(tibble)
library(magrittr)
library(purrr)
library(dplyr)
library(readr)

Case Study

• extract domain name from random email ids
• extract image type from url
• extract image dimension from url
• extract extension from domain name
• extract http protocol from url
• extract file type from url

mockstring <- read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/mock_strings.csv')
mockstring

## # A tibble: 1,000 x 12
##       id image_url domain imageurl email filename phone address url   full_name
##    <dbl> <chr>     <chr>  <chr>    <chr> <chr>    <chr> <chr>   <chr> <chr>    
##  1     1 https://~ addto~ http://~ mnew~ PedeMal~ 66-(~ 8 Anha~ http~ Mufi Ruit
##  2     2 https://~ gmpg.~ http://~ mdan~ Loborti~ 351-~ 697 Ea~ http~ Leese Fu~
##  3     3 https://~ samsu~ http://~ hgir~ CongueD~ 33-(~ 89 Dot~ http~ Blakelee~
##  4     4 https://~ spoti~ http://~ pmcm~ Eleifen~ 86-(~ 98135 ~ http~ Terencio~
##  5     5 https://~ wunde~ http://~ dris~ PurusPh~ 223-~ 7814 P~ http~ Debee Mc~
##  6     6 https://~ alexa~ http://~ cphl~ Element~ 420-~ 4897 L~ http~ Fran Pai~
##  7     7 https://~ googl~ http://~ kdod~ Mattis.~ 1-(7~ 53541 ~ http~ Frasco B~
##  8     8 https://~ ed.gov http://~ vhou~ PurusEu~ 62-(~ 4819 H~ http~ Car Pont~
##  9     9 https://~ jigsy~ http://~ rdik~ JustoEt~ 1-(6~ 68096 ~ http~ Tades Ch~
## 10    10 https://~ jugem~ http://~ tdud~ Ante.ti~ 30-(~ 9595 S~ http~ Wilton K~
## # ... with 990 more rows, and 2 more variables: currency <chr>, passwords <chr>

Overview

Before we start with the case study, let us take a quick tour of stringr and introduce ourselves to some of the functions we will be using later in the case study. One of the columns in the case study data is email. It contains random email ids. We want to ensure that the email ids adher to a particular format .i.e

• they contain @
• they contain only one @

Let us first detect if the email ids contain @. Since the data set has 1000 rows, we will use a smaller sample in the examples.

mockdata <- slice(mockstring, 1:10)
mockdata

## # A tibble: 10 x 12
##       id image_url domain imageurl email filename phone address url   full_name
##    <dbl> <chr>     <chr>  <chr>    <chr> <chr>    <chr> <chr>   <chr> <chr>    
##  1     1 https://~ addto~ http://~ mnew~ PedeMal~ 66-(~ 8 Anha~ http~ Mufi Ruit
##  2     2 https://~ gmpg.~ http://~ mdan~ Loborti~ 351-~ 697 Ea~ http~ Leese Fu~
##  3     3 https://~ samsu~ http://~ hgir~ CongueD~ 33-(~ 89 Dot~ http~ Blakelee~
##  4     4 https://~ spoti~ http://~ pmcm~ Eleifen~ 86-(~ 98135 ~ http~ Terencio~
##  5     5 https://~ wunde~ http://~ dris~ PurusPh~ 223-~ 7814 P~ http~ Debee Mc~
##  6     6 https://~ alexa~ http://~ cphl~ Element~ 420-~ 4897 L~ http~ Fran Pai~
##  7     7 https://~ googl~ http://~ kdod~ Mattis.~ 1-(7~ 53541 ~ http~ Frasco B~
##  8     8 https://~ ed.gov http://~ vhou~ PurusEu~ 62-(~ 4819 H~ http~ Car Pont~
##  9     9 https://~ jigsy~ http://~ rdik~ JustoEt~ 1-(6~ 68096 ~ http~ Tades Ch~
## 10    10 https://~ jugem~ http://~ tdud~ Ante.ti~ 30-(~ 9595 S~ http~ Wilton K~
## # ... with 2 more variables: currency <chr>, passwords <chr>

Use str_detect() to detect @ and str_count() to count the number of times @ appears in the email ids.

# detect @
str_detect(mockdata$email, pattern = "@")

##  [1] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE

# count @
str_count(mockdata$email, pattern = "@")

##  [1] 1 1 1 1 1 1 1 1 1 1

We can use str_c() to concatenate strings. Let us add the string email id: before each email id in the data set.

str_c("email id:", mockdata$email)

##  [1] "email id:mnewburn0@fastcompany.com"   
##  [2] "email id:mdankersley1@digg.com"       
##  [3] "email id:hgirhard2@altervista.org"    
##  [4] "email id:pmcmenamy3@sciencedirect.com"
##  [5] "email id:drisbrough4@bandcamp.com"    
##  [6] "email id:cphlippi5@surveymonkey.com"  
##  [7] "email id:kdodswell6@un.org"           
##  [8] "email id:vhourihane7@ovh.net"         
##  [9] "email id:rdike8@timesonline.co.uk"    
## [10] "email id:tdudbridge9@clickbank.net"

If we want to split a string into two parts using a particular pattern, we use str_split(). Let us split the domain name and extension from the domain column in the data. The domain name and extension are separated by . and we will use it to split the domain column. Since . is a special character, we will use two slashes to escape the special character.

str_split(mockdata$domain, pattern = "\\.")

## [[1]]
## [1] "addtoany" "com"     
## 
## [[2]]
## [1] "gmpg" "org" 
## 
## [[3]]
## [1] "samsung" "com"    
## 
## [[4]]
## [1] "spotify" "com"    
## 
## [[5]]
## [1] "wunderground" "com"         
## 
## [[6]]
## [1] "alexa" "com"  
## 
## [[7]]
## [1] "google" "it"    
## 
## [[8]]
## [1] "ed"  "gov"
## 
## [[9]]
## [1] "jigsy" "com"  
## 
## [[10]]
## [1] "jugem" "jp"

We can truncate a string using str_trunc(). The default truncation happens at the beggining of the string but we can truncate the central part or the end of the string as well.

str_trunc(mockdata$email, width = 10)

##  [1] "mnewbur..." "mdanker..." "hgirhar..." "pmcmena..." "drisbro..."
##  [6] "cphlipp..." "kdodswe..." "vhourih..." "rdike8@..." "tdudbri..."

str_trunc(mockdata$email, width = 10, side = "left")

##  [1] "...any.com" "...igg.com" "...sta.org" "...ect.com" "...amp.com"
##  [6] "...key.com" "...@un.org" "...ovh.net" "...e.co.uk" "...ank.net"

str_trunc(mockdata$email, width = 10, side = "center")

##  [1] "mnew...com" "mdan...com" "hgir...org" "pmcm...com" "dris...com"
##  [6] "cphl...com" "kdod...org" "vhou...net" "rdik....uk" "tdud...net"

Strings can be sorted using str_sort(). Let us quickly sort the emails in both ascending and descending orders.

str_sort(mockdata$email)

##  [1] "cphlippi5@surveymonkey.com"   "drisbrough4@bandcamp.com"    
##  [3] "hgirhard2@altervista.org"     "kdodswell6@un.org"           
##  [5] "mdankersley1@digg.com"        "mnewburn0@fastcompany.com"   
##  [7] "pmcmenamy3@sciencedirect.com" "rdike8@timesonline.co.uk"    
##  [9] "tdudbridge9@clickbank.net"    "vhourihane7@ovh.net"

str_sort(mockdata$email, decreasing = TRUE)

##  [1] "vhourihane7@ovh.net"          "tdudbridge9@clickbank.net"   
##  [3] "rdike8@timesonline.co.uk"     "pmcmenamy3@sciencedirect.com"
##  [5] "mnewburn0@fastcompany.com"    "mdankersley1@digg.com"       
##  [7] "kdodswell6@un.org"            "hgirhard2@altervista.org"    
##  [9] "drisbrough4@bandcamp.com"     "cphlippi5@surveymonkey.com"

The case of a string can be changed to upper, lower or title case as shown below.

str_to_upper(mockdata$full_name)

##  [1] "MUFI RUIT"          "LEESE FURMAGIER"    "BLAKELEE WILSHIRE" 
##  [4] "TERENCIO MCILLRICK" "DEBEE MCERLAINE"    "FRAN PAINTEN"      
##  [7] "FRASCO BOWICH"      "CAR PONTEN"         "TADES CHECCUCCI"   
## [10] "WILTON KEMMEY"

str_to_lower(mockdata$full_name)

##  [1] "mufi ruit"          "leese furmagier"    "blakelee wilshire" 
##  [4] "terencio mcillrick" "debee mcerlaine"    "fran painten"      
##  [7] "frasco bowich"      "car ponten"         "tades checcucci"   
## [10] "wilton kemmey"

Parts of a string can be replaced using str_replace(). In the address column of the data set, let us replace:

• Street with ST
• Road with RD

str_replace(mockdata$address, "Street", "ST")

##  [1] "8 Anhalt Crossing"          "697 East Avenue"           
##  [3] "89 Dottie Circle"           "98135 Blue Bill Park Drive"
##  [5] "7814 Pennsylvania ST"       "4897 Little Fleur Drive"   
##  [7] "53541 Morrow Center"        "4819 Hermina Parkway"      
##  [9] "68096 Monument Park"        "9595 Spaight Avenue"

str_replace(mockdata$address, "Road", "RD")

##  [1] "8 Anhalt Crossing"          "697 East Avenue"           
##  [3] "89 Dottie Circle"           "98135 Blue Bill Park Drive"
##  [5] "7814 Pennsylvania Street"   "4897 Little Fleur Drive"   
##  [7] "53541 Morrow Center"        "4819 Hermina Parkway"      
##  [9] "68096 Monument Park"        "9595 Spaight Avenue"

We can extract parts of the string that match a particular pattern using str_extract().

str_extract(mockdata$email, pattern = "org")

##  [1] NA    NA    "org" NA    NA    NA    "org" NA    NA    NA

Before we extract, we need to know whether the string contains text that match our pattern. Use str_match() to see if the pattern is present in the string.

str_match(mockdata$email, pattern = "org")

##       [,1] 
##  [1,] NA   
##  [2,] NA   
##  [3,] "org"
##  [4,] NA   
##  [5,] NA   
##  [6,] NA   
##  [7,] "org"
##  [8,] NA   
##  [9,] NA   
## [10,] NA

If we are dealing with a character vector and know that the pattern we are looking at is present in the vector, we might want to know the index of the strings in which it is present. Use str_which() to identify the index of the strings that match our pattern.

str_which(mockdata$email, pattern = "org")

## [1] 3 7

Another objective might be to locate the position of the pattern we are looking for in the string. For example, if we want to know the position of @ in the email ids, we can use str_locate().

str_locate(mockdata$email, pattern = "@")

##       start end
##  [1,]    10  10
##  [2,]    13  13
##  [3,]    10  10
##  [4,]    11  11
##  [5,]    12  12
##  [6,]    10  10
##  [7,]    11  11
##  [8,]    12  12
##  [9,]     7   7
## [10,]    12  12

The length of the string can be computed using str_length(). Let us ensure that the length of the strings in the password column is 16.

str_length(mockdata$passwords)

##  [1] 16 16 16 16 16 16 16 16 16 16

We can extract parts of a string by specifying the starting and ending position using str_sub(). Let us extract the currency type from the currency column.

str_sub(mockdata$currency, start = 1, end = 1)

##  [1] "¥" "$" "\200" "\200" "\200" "¥" "$" "¥" "\200" "\200"

One final function that we will look at before the case study is word(). It extracts word(s) from sentences. We do not have any sentences in the data set, but let us use it to extract the first and last name from the full_name column.

word(mockdata$full_name, 1)

##  [1] "Mufi"     "Leese"    "Blakelee" "Terencio" "Debee"    "Fran"    
##  [7] "Frasco"   "Car"      "Tades"    "Wilton"

word(mockdata$full_name, 2)

##  [1] "Ruit"      "Furmagier" "Wilshire"  "McIllrick" "McErlaine" "Painten"  
##  [7] "Bowich"    "Ponten"    "Checcucci" "Kemmey"

Alright, now let us apply what we have learned so far to our case study.

Extract domain name from email ids

emails <- 
  mockstring %>%
  pull(email) %>%
  head()

emails

## [1] "mnewburn0@fastcompany.com"    "mdankersley1@digg.com"       
## [3] "hgirhard2@altervista.org"     "pmcmenamy3@sciencedirect.com"
## [5] "drisbrough4@bandcamp.com"     "cphlippi5@surveymonkey.com"

 str_split(emails, pattern = '@')

## [[1]]
## [1] "mnewburn0"       "fastcompany.com"
## 
## [[2]]
## [1] "mdankersley1" "digg.com"    
## 
## [[3]]
## [1] "hgirhard2"      "altervista.org"
## 
## [[4]]
## [1] "pmcmenamy3"        "sciencedirect.com"
## 
## [[5]]
## [1] "drisbrough4"  "bandcamp.com"
## 
## [[6]]
## [1] "cphlippi5"        "surveymonkey.com"

emails %>%
  str_split(pattern = '@') %>%
  map_chr(2)

## [1] "fastcompany.com"   "digg.com"          "altervista.org"   
## [4] "sciencedirect.com" "bandcamp.com"      "surveymonkey.com"

emails %>%
  str_split(pattern = '@') %>%
  map_chr(2) %>%
  str_split(pattern = '\\.') 

## [[1]]
## [1] "fastcompany" "com"        
## 
## [[2]]
## [1] "digg" "com" 
## 
## [[3]]
## [1] "altervista" "org"       
## 
## [[4]]
## [1] "sciencedirect" "com"          
## 
## [[5]]
## [1] "bandcamp" "com"     
## 
## [[6]]
## [1] "surveymonkey" "com"

emails %>%
  str_split(pattern = '@') %>%
  map_chr(2) %>%
  str_split(pattern = '\\.') %>%
  map_chr(extract(1))

## [1] "fastcompany"   "digg"          "altervista"    "sciencedirect"
## [5] "bandcamp"      "surveymonkey"

Extract Domain Extension

The below code extracts the domain extension instead of the domain name.

emails %>%
  str_split(pattern = '@') %>%
  map_chr(2) %>%
  str_split(pattern = '\\.', simplify = TRUE) %>%
  extract(, 2)

## [1] "com" "com" "org" "com" "com" "com"

Extract image type from URL

img <- 
  mockstring %>%
  pull(imageurl) %>%
  head()

img

## [1] "http://dummyimage.com/130x183.jpg/dddddd/000000"
## [2] "http://dummyimage.com/106x217.bmp/dddddd/000000"
## [3] "http://dummyimage.com/146x127.bmp/cc0000/ffffff"
## [4] "http://dummyimage.com/181x194.png/5fa2dd/ffffff"
## [5] "http://dummyimage.com/220x123.jpg/ff4444/ffffff"
## [6] "http://dummyimage.com/118x176.bmp/dddddd/000000"

str_split(img, pattern = '\\.')

## [[1]]
## [1] "http://dummyimage" "com/130x183"       "jpg/dddddd/000000"
## 
## [[2]]
## [1] "http://dummyimage" "com/106x217"       "bmp/dddddd/000000"
## 
## [[3]]
## [1] "http://dummyimage" "com/146x127"       "bmp/cc0000/ffffff"
## 
## [[4]]
## [1] "http://dummyimage" "com/181x194"       "png/5fa2dd/ffffff"
## 
## [[5]]
## [1] "http://dummyimage" "com/220x123"       "jpg/ff4444/ffffff"
## 
## [[6]]
## [1] "http://dummyimage" "com/118x176"       "bmp/dddddd/000000"

img %>%
  str_split(pattern = '\\.') %>%
  map_chr(extract(3))

## [1] "jpg/dddddd/000000" "bmp/dddddd/000000" "bmp/cc0000/ffffff"
## [4] "png/5fa2dd/ffffff" "jpg/ff4444/ffffff" "bmp/dddddd/000000"

img %>%
  str_split(pattern = '\\.') %>%
  map_chr(extract(3)) %>%
  str_sub(start = 1, end = 3)

## [1] "jpg" "bmp" "bmp" "png" "jpg" "bmp"

img %>%
  str_split(pattern = '\\.') %>%
  map_chr(extract(3)) %>%
  str_split(pattern = '/') %>%
  map_chr(extract(1))

## [1] "jpg" "bmp" "bmp" "png" "jpg" "bmp"

Extract Image Dimesion from URL

str_locate(img, pattern = "[0-9]") 

##      start end
## [1,]    23  23
## [2,]    23  23
## [3,]    23  23
## [4,]    23  23
## [5,]    23  23
## [6,]    23  23

str_sub(img, start = 23) 

## [1] "130x183.jpg/dddddd/000000" "106x217.bmp/dddddd/000000"
## [3] "146x127.bmp/cc0000/ffffff" "181x194.png/5fa2dd/ffffff"
## [5] "220x123.jpg/ff4444/ffffff" "118x176.bmp/dddddd/000000"

img %>%
  str_sub(start = 23) %>%
  str_split(pattern = '\\.') 

## [[1]]
## [1] "130x183"           "jpg/dddddd/000000"
## 
## [[2]]
## [1] "106x217"           "bmp/dddddd/000000"
## 
## [[3]]
## [1] "146x127"           "bmp/cc0000/ffffff"
## 
## [[4]]
## [1] "181x194"           "png/5fa2dd/ffffff"
## 
## [[5]]
## [1] "220x123"           "jpg/ff4444/ffffff"
## 
## [[6]]
## [1] "118x176"           "bmp/dddddd/000000"

img %>%
  str_sub(start = 23) %>%
  str_split(pattern = '\\.') %>%
  map_chr(extract(1))

## [1] "130x183" "106x217" "146x127" "181x194" "220x123" "118x176"

Extract HTTP Protocol from URL

url1 <- 
  mockstring %>%
  pull(url) %>%
  first()

url1

## [1] "https://engadget.com/nascetur/ridiculus/mus/vivamus/vestibulum.jsp?eu=est&tincidunt=risus&in=auctor&leo=sed&maecenas=tristique&pulvinar=in&lobortis=tempus&est=sit&phasellus=amet&sit=sem&amet=fusce&erat=consequat&nulla=nulla&tempus=nisl&vivamus=nunc&in=nisl&felis=duis&eu=bibendum&sapien=felis&cursus=sed&vestibulum=interdum&proin=venenatis&eu=turpis&mi=enim&nulla=blandit&ac=mi&enim=in&in=porttitor&tempor=pede&turpis=justo&nec=eu&euismod=massa&scelerisque=donec&quam=dapibus&turpis=duis&adipiscing=at&lorem=velit&vitae=eu&mattis=est&nibh=congue&ligula=elementum&nec=in&sem=hac&duis=habitasse&aliquam=platea&convallis=dictumst&nunc=morbi&proin=vestibulum&at=velit&turpis=id&a=pretium&pede=iaculis&posuere=diam&nonummy=erat&integer=fermentum&non=justo&velit=nec&donec=condimentum&diam=neque&neque=sapien&vestibulum=placerat&eget=ante&vulputate=nulla&ut=justo&ultrices=aliquam&vel=quis&augue=turpis&vestibulum=eget&ante=elit&ipsum=sodales&primis=scelerisque&in=mauris&faucibus=sit&orci=amet&luctus=eros&et=suspendisse&ultrices=accumsan&posuere=tortor&cubilia=quis&curae=turpis&donec=sed&pharetra=ante&magna=vivamus&vestibulum=tortor&aliquet=duis&ultrices=mattis&erat=egestas&tortor=metus&sollicitudin=aenean&mi=fermentum&sit=donec"

str_split(url1, pattern = '://') 

## [[1]]
## [1] "https"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
## [2] "engadget.com/nascetur/ridiculus/mus/vivamus/vestibulum.jsp?eu=est&tincidunt=risus&in=auctor&leo=sed&maecenas=tristique&pulvinar=in&lobortis=tempus&est=sit&phasellus=amet&sit=sem&amet=fusce&erat=consequat&nulla=nulla&tempus=nisl&vivamus=nunc&in=nisl&felis=duis&eu=bibendum&sapien=felis&cursus=sed&vestibulum=interdum&proin=venenatis&eu=turpis&mi=enim&nulla=blandit&ac=mi&enim=in&in=porttitor&tempor=pede&turpis=justo&nec=eu&euismod=massa&scelerisque=donec&quam=dapibus&turpis=duis&adipiscing=at&lorem=velit&vitae=eu&mattis=est&nibh=congue&ligula=elementum&nec=in&sem=hac&duis=habitasse&aliquam=platea&convallis=dictumst&nunc=morbi&proin=vestibulum&at=velit&turpis=id&a=pretium&pede=iaculis&posuere=diam&nonummy=erat&integer=fermentum&non=justo&velit=nec&donec=condimentum&diam=neque&neque=sapien&vestibulum=placerat&eget=ante&vulputate=nulla&ut=justo&ultrices=aliquam&vel=quis&augue=turpis&vestibulum=eget&ante=elit&ipsum=sodales&primis=scelerisque&in=mauris&faucibus=sit&orci=amet&luctus=eros&et=suspendisse&ultrices=accumsan&posuere=tortor&cubilia=quis&curae=turpis&donec=sed&pharetra=ante&magna=vivamus&vestibulum=tortor&aliquet=duis&ultrices=mattis&erat=egestas&tortor=metus&sollicitudin=aenean&mi=fermentum&sit=donec"

url1 %>%
  str_split(pattern = '://') %>%
  map_chr(extract(1))

## [1] "https"

Extract file type

urls <-
  mockstring %>%
  use_series(url) %>%
  extract(1:3)

urls %>%
  str_locate_all(pattern = '\\.') %>%
  map_int(nrow) %>%
  is_greater_than(2) %>%
  sum()

## [1] 0

urls %>%
  str_locate_all(pattern = "[?]") %>%
  map_int(nrow) %>%
  is_greater_than(1) %>%
  sum()

## [1] 0

d <- 
  urls %>%
  str_locate_all(pattern = '\\.') %>%
  map_int(extract(2)) %>%
  add(1)

d  

## [1] 64 47 48

q <-  
  urls %>%
  str_locate_all(pattern = "[?]") %>%
  map_int(extract(1)) %>%
  subtract(1)

q

## [1] 66 50 51

str_sub(urls, start = d, end = q)

## [1] "jsp"  "json" "json"

References

• https://stringr.tidyverse.org/
• http://r4ds.had.co.nz/strings.html

Introduction

A tibble, or tbl_df, is a modern reimagining of the data.frame, keeping what time has proven to be effective, and throwing out what is not. Tibbles are data.frames that are lazy and surly: they do less (i.e. they don’t change variable names or types, and don’t do partial matching) and complain more (e.g. when a variable does not exist). This forces you to confront problems earlier, typically leading to cleaner, more expressive code. Tibbles also have an enhanced print method() which makes them easier to use with large datasets containing complex objects.

Source: https://tibble.tidyverse.org/

In this post, we will explore tibbles. To be more precise, we will learn:

• how tibbles are different from data frames?
• how to create tibbles?
• how to manipulate tibbles?

Libraries, Code & Data

We will use the following packages:

• tibble
• dplyr

The code can be found here.

library(tibble)
library(dplyr)

Creating tibbles

tibble can be created using any of the following:

• tibble()
• as_tibble()
• tribble()

Let us start with tibble().

tibble(x = letters,
       y = 1:26,
       z = sample(100, 26))

## # A tibble: 26 x 3
##    x         y     z
##    <chr> <int> <int>
##  1 a         1    71
##  2 b         2    56
##  3 c         3    59
##  4 d         4    14
##  5 e         5    58
##  6 f         6    60
##  7 g         7    16
##  8 h         8     9
##  9 i         9    66
## 10 j        10    48
## # ... with 16 more rows

We mentioned the column names followed by the data. If you do not specify the column names, tibble() will supply them. Ensure that the length of each column is same.

tibble features

• never changes input’s types

tibble() will never alter the input’s type. For example, if you supply a character vector it will not be converted to factor unlike data.frame where you need to set stringsAsFactors to FALSE.

tibble(x = letters,
       y = 1:26,
       z = sample(100, 26))

## # A tibble: 26 x 3
##    x         y     z
##    <chr> <int> <int>
##  1 a         1    62
##  2 b         2    13
##  3 c         3    75
##  4 d         4    17
##  5 e         5    82
##  6 f         6    83
##  7 g         7     9
##  8 h         8    76
##  9 i         9    19
## 10 j        10    97
## # ... with 16 more rows

• never adjusts variable names

tibble() will never modify the column names. In the below example, you can observe that while data.frame adds a ., tibble() retains the column names as is.

names(data.frame(`order value` = 10))

## [1] "order.value"

names(tibble(`order value` = 10))

## [1] "order value"

• never prints all rows

tibble() will never print all the rows and clutter your console. It will only print the first 10 rows and only as many columns that fit the width of the console.

x <- 1:100
y <- letters[1]
z <- sample(c(TRUE, FALSE), 100, replace = TRUE)
tibble(x, y, z)

## # A tibble: 100 x 3
##        x y     z    
##    <int> <chr> <lgl>
##  1     1 a     TRUE 
##  2     2 a     FALSE
##  3     3 a     TRUE 
##  4     4 a     TRUE 
##  5     5 a     FALSE
##  6     6 a     TRUE 
##  7     7 a     TRUE 
##  8     8 a     FALSE
##  9     9 a     TRUE 
## 10    10 a     FALSE
## # ... with 90 more rows

• never recycles vector of length greater than 1

Recycling vectors of length greater than 1 often leads to errors and as such tibble() will only recycle vectors of length 1.

x <- 1:100
y <- letters
z <- sample(c(TRUE, FALSE), 100, replace = TRUE)
tibble(x, y, z)
Error in overscope_eval_next(overscope, expr) : object 'y' not found

Membership Testing

We can test if an object is a tibble using is_tibble().

is_tibble(mtcars)

## [1] FALSE

is_tibble(as_tibble(mtcars))

## [1] TRUE

Tribble

Another way to create tibbles is using tribble():

• it is short for transposed tibbles
• it is customized for data entry in code
• column names start with ~
• and values are separated by commas

tribble(
  ~x, ~y, ~z,
  #--|--|----
  1, TRUE, 'a',
  2, FALSE, 'b'
)

## # A tibble: 2 x 3
##       x y     z    
##   <dbl> <lgl> <chr>
## 1     1 TRUE  a    
## 2     2 FALSE b

Column Names

Names of the columns in tibbles need not be valid R variable names. They can contain unusual characters like a space or a smiley but must be enclosed in ticks.

tibble(
  ` ` = 'space',
  `2` = 'integer',
  `:)` = 'smiley'
)

## # A tibble: 1 x 3
##   ` `   `2`     `:)`  
##   <chr> <chr>   <chr> 
## 1 space integer smiley

Add Rows

Let us add data related to Safari browser to the web traffic data using add_row().

browsers <- enframe(c(chrome = 40, firefox = 20, edge = 30))
browsers

## # A tibble: 3 x 2
##   name    value
##   <chr>   <dbl>
## 1 chrome     40
## 2 firefox    20
## 3 edge       30

add_row(browsers, name = 'safari', value = 10)

## # A tibble: 4 x 2
##   name    value
##   <chr>   <dbl>
## 1 chrome     40
## 2 firefox    20
## 3 edge       30
## 4 safari     10

If we want to add the data at a particular row, we can specify the row number using the .before argument. Let us add the data related to Safari browser in the second row instead of the last row.

add_row(browsers, name = 'safari', value = 10, .before = 2)

## # A tibble: 4 x 2
##   name    value
##   <chr>   <dbl>
## 1 chrome     40
## 2 safari     10
## 3 firefox    20
## 4 edge       30

Add Columns

add_column() adds a new column to tibbles.

browsers <- enframe(c(chrome = 40, firefox = 20, edge = 30, safari = 10))
add_column(browsers, visits = c(4000, 2000, 3000, 1000))

## # A tibble: 4 x 3
##   name    value visits
##   <chr>   <dbl>  <dbl>
## 1 chrome     40   4000
## 2 firefox    20   2000
## 3 edge       30   3000
## 4 safari     10   1000

Rownames

The tibble package provides a set of functions to deal with rownames. Remember, tibble does not have rownames unlike data.frame. To check whether a data set has rownames, use has_rownames().

has_rownames(mtcars)

## [1] TRUE

remove_rownames(mtcars)

##     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
## 1  21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
## 2  21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
## 3  22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
## 4  21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
## 5  18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
## 6  18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
## 7  14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
## 8  24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
## 9  22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
## 10 19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
## 11 17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
## 12 16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
## 13 17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3
## 14 15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3
## 15 10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
## 16 10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4
## 17 14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
## 18 32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
## 19 30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
## 20 33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
## 21 21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
## 22 15.5   8 318.0 150 2.76 3.520 16.87  0  0    3    2
## 23 15.2   8 304.0 150 3.15 3.435 17.30  0  0    3    2
## 24 13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
## 25 19.2   8 400.0 175 3.08 3.845 17.05  0  0    3    2
## 26 27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
## 27 26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
## 28 30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
## 29 15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
## 30 19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
## 31 15.0   8 301.0 335 3.54 3.570 14.60  0  1    5    8
## 32 21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2

head(rownames_to_column(mtcars))

##             rowname  mpg cyl disp  hp drat    wt  qsec vs am gear carb
## 1         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## 2     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## 4    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## 5 Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## 6           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

mtcars_tbl <- rownames_to_column(mtcars)
column_to_rownames(mtcars_tbl)

##                      mpg cyl  disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive      21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout   18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
## Valiant             18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
## Duster 360          14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
## Merc 240D           24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
## Merc 230            22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
## Merc 280            19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
## Merc 280C           17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
## Merc 450SE          16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
## Merc 450SL          17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3
## Merc 450SLC         15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3
## Cadillac Fleetwood  10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
## Lincoln Continental 10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4
## Chrysler Imperial   14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
## Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
## Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
## Toyota Corolla      33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
## Toyota Corona       21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
## Dodge Challenger    15.5   8 318.0 150 2.76 3.520 16.87  0  0    3    2
## AMC Javelin         15.2   8 304.0 150 3.15 3.435 17.30  0  0    3    2
## Camaro Z28          13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
## Pontiac Firebird    19.2   8 400.0 175 3.08 3.845 17.05  0  0    3    2
## Fiat X1-9           27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
## Porsche 914-2       26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
## Lotus Europa        30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
## Ford Pantera L      15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
## Ferrari Dino        19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
## Maserati Bora       15.0   8 301.0 335 3.54 3.570 14.60  0  1    5    8
## Volvo 142E          21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2

Glimpse

Use glimpse() to get an overview of the data.

glimpse(mtcars)

## Rows: 32
## Columns: 11
## $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19.2, 17...
## $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, 4, 4,...
## $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 140.8,...
## $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, 180, ...
## $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.92, 3....
## $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3.150,...
## $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 22.90,...
## $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1,...
## $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,...
## $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, 4, 3,...
## $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, 1, 1,...

Check Column

has_name() can be used to check if a tibble has a specific column.

has_name(mtcars, 'cyl')

## [1] TRUE

has_name(mtcars, 'gears')

## [1] FALSE

Summary

References

• https://tibble.tidyverse.org/
• http://r4ds.had.co.nz/tibbles.html

Introduction

In the previous post, we learnt to combine tables using dplyr. In this post, we will explore a set of helper functions in order to:

• extract unique rows
• rename columns
• sample data
• extract columns
• slice rows
• arrange rows
• compare tables
• extract/mutate data using predicate functions
• count observations for different levels of a variable

Libraries, Code & Data

We will use the following packages:

• dplyr
• readr

The data sets can be downloaded from here and the codes from here.

library(dplyr)
library(readr)

Case Study

Let us look at a case study (e-commerce data) and see how we can use dplyr helper functions to answer questions we have about and to modify/transform the underlying data set.

ecom <- 
  read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/web.csv',
    col_types = cols_only(device = col_factor(levels = c("laptop", "tablet", "mobile")),
      referrer = col_factor(levels = c("bing", "direct", "social", "yahoo", "google")),
      purchase = col_logical(), bouncers = col_logical(), duration = col_double(),
      n_visit = col_double(), n_pages = col_double()
    )
  )

ecom

## # A tibble: 1,000 x 7
##    referrer device bouncers n_visit n_pages duration purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
##  1 google   laptop TRUE          10       1      693 FALSE   
##  2 yahoo    tablet TRUE           9       1      459 FALSE   
##  3 direct   laptop TRUE           0       1      996 FALSE   
##  4 bing     tablet FALSE          3      18      468 TRUE    
##  5 yahoo    mobile TRUE           9       1      955 FALSE   
##  6 yahoo    laptop FALSE          5       5      135 FALSE   
##  7 yahoo    mobile TRUE          10       1       75 FALSE   
##  8 direct   mobile TRUE          10       1      908 FALSE   
##  9 bing     mobile FALSE          3      19      209 FALSE   
## 10 google   mobile TRUE           6       1      208 FALSE   
## # ... with 990 more rows

Data Sanitization

Let us ensure that the data is sanitized by checking the sources of traffic and devices used to visit the site. We will use distinct to examine the values in the referrer column

distinct(ecom, referrer)

## # A tibble: 5 x 1
##   referrer
##   <fct>   
## 1 google  
## 2 yahoo   
## 3 direct  
## 4 bing    
## 5 social

and the device column as well.

distinct(ecom, device)

## # A tibble: 3 x 1
##   device
##   <fct> 
## 1 laptop
## 2 tablet
## 3 mobile

Rename Columns

Columns can be renamed using rename().

rename(ecom, time_on_site = duration)

## # A tibble: 1,000 x 7
##    referrer device bouncers n_visit n_pages time_on_site purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>        <dbl> <lgl>   
##  1 google   laptop TRUE          10       1          693 FALSE   
##  2 yahoo    tablet TRUE           9       1          459 FALSE   
##  3 direct   laptop TRUE           0       1          996 FALSE   
##  4 bing     tablet FALSE          3      18          468 TRUE    
##  5 yahoo    mobile TRUE           9       1          955 FALSE   
##  6 yahoo    laptop FALSE          5       5          135 FALSE   
##  7 yahoo    mobile TRUE          10       1           75 FALSE   
##  8 direct   mobile TRUE          10       1          908 FALSE   
##  9 bing     mobile FALSE          3      19          209 FALSE   
## 10 google   mobile TRUE           6       1          208 FALSE   
## # ... with 990 more rows

Data Tabulation

Let us now look at the proportion or share of visits driven by different sources of traffic.

ecom %>%
  group_by(referrer) %>%
  tally()

## # A tibble: 5 x 2
##   referrer     n
## * <fct>    <int>
## 1 bing       194
## 2 direct     191
## 3 social     200
## 4 yahoo      207
## 5 google     208

We would also like to know the number of bouncers driven by the different sources of traffic.

ecom %>%
  group_by(referrer, bouncers) %>%
  tally()

## # A tibble: 10 x 3
## # Groups:   referrer [5]
##    referrer bouncers     n
##    <fct>    <lgl>    <int>
##  1 bing     FALSE      104
##  2 bing     TRUE        90
##  3 direct   FALSE       98
##  4 direct   TRUE        93
##  5 social   FALSE       93
##  6 social   TRUE       107
##  7 yahoo    FALSE      110
##  8 yahoo    TRUE        97
##  9 google   FALSE      101
## 10 google   TRUE       107

Let us look at how many conversions happen across different devices.

ecom %>%
  group_by(device, purchase) %>%
  tally() %>%
  filter(purchase)

## # A tibble: 3 x 3
## # Groups:   device [3]
##   device purchase     n
##   <fct>  <lgl>    <int>
## 1 laptop TRUE        31
## 2 tablet TRUE        36
## 3 mobile TRUE        36

Another way to extract the above information is by using count

ecom %>%
  count(referrer, purchase) %>%
  filter(purchase)

## # A tibble: 5 x 3
##   referrer purchase     n
##   <fct>    <lgl>    <int>
## 1 bing     TRUE        17
## 2 direct   TRUE        25
## 3 social   TRUE        20
## 4 yahoo    TRUE        22
## 5 google   TRUE        19

Sampling Data

dplyr offers sampling functions which allow us to specify either the number or percentage of observations. sample_n() allows sampling a specific number of observations.

sample_n(ecom, 700)

## # A tibble: 700 x 7
##    referrer device bouncers n_visit n_pages duration purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
##  1 yahoo    laptop FALSE          2       9      162 FALSE   
##  2 yahoo    tablet FALSE          2      14      364 TRUE    
##  3 social   laptop TRUE           1       1      111 FALSE   
##  4 direct   laptop TRUE           4       1      896 FALSE   
##  5 yahoo    mobile FALSE          5       8       80 FALSE   
##  6 social   laptop TRUE           0       1      720 FALSE   
##  7 bing     mobile TRUE           5       1      190 FALSE   
##  8 direct   mobile TRUE           2       1      501 FALSE   
##  9 yahoo    tablet TRUE           1       1      605 FALSE   
## 10 bing     laptop TRUE           1       1      169 FALSE   
## # ... with 690 more rows

We can combine the sampling functions with other dplyr functions as shown below where we sample observation after grouping them according to the source of traffic.

ecom %>%
  group_by(referrer) %>%
  sample_n(100)

## # A tibble: 500 x 7
## # Groups:   referrer [5]
##    referrer device bouncers n_visit n_pages duration purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
##  1 bing     mobile FALSE          3       8      120 FALSE   
##  2 bing     laptop FALSE          9      13      299 FALSE   
##  3 bing     tablet FALSE          2      17      510 FALSE   
##  4 bing     laptop TRUE           9       1      709 FALSE   
##  5 bing     tablet TRUE           0       1      845 FALSE   
##  6 bing     tablet TRUE           1       1      721 FALSE   
##  7 bing     tablet TRUE           0       1      425 FALSE   
##  8 bing     mobile FALSE          0       7      196 FALSE   
##  9 bing     tablet TRUE           4       1      493 FALSE   
## 10 bing     mobile TRUE           6       1      604 FALSE   
## # ... with 490 more rows

sample_frac() allows a specific percentage of observations.

sample_frac(ecom, size = 0.7)

## # A tibble: 700 x 7
##    referrer device bouncers n_visit n_pages duration purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
##  1 google   mobile TRUE           0       1      132 FALSE   
##  2 yahoo    tablet TRUE           7       1      889 FALSE   
##  3 bing     mobile FALSE          7       1       22 FALSE   
##  4 google   laptop TRUE           5       1      376 FALSE   
##  5 social   tablet FALSE         10      11      330 TRUE    
##  6 social   mobile FALSE          7       8      216 FALSE   
##  7 bing     mobile FALSE          1      12      168 FALSE   
##  8 bing     tablet TRUE           7       1      489 FALSE   
##  9 social   laptop TRUE           0       1      581 FALSE   
## 10 direct   laptop FALSE          6       6       96 FALSE   
## # ... with 690 more rows

Data Extraction

In the first post, we had observed that dplyr verbs always returned a tibble. What if you want to extract a specific column or a bunch of rows but not as a tibble?

Use pull to extract columns either by name or position. It will return a vector. In the below example, we extract the device column as a vector. I am using head in addition to limit the output printed.

ecom_mini <- sample_n(ecom, size = 10)

pull(ecom_mini, device)

##  [1] laptop mobile mobile laptop tablet laptop mobile laptop tablet tablet
## Levels: laptop tablet mobile

pull(ecom_mini, 1) 

##  [1] bing   google yahoo  direct bing   direct bing   bing   bing   bing  
## Levels: bing direct social yahoo google

pull(ecom_mini, -1) 

##  [1] FALSE FALSE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE

slice(ecom, 5:15)

## # A tibble: 11 x 7
##    referrer device bouncers n_visit n_pages duration purchase
##    <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
##  1 yahoo    mobile TRUE           9       1      955 FALSE   
##  2 yahoo    laptop FALSE          5       5      135 FALSE   
##  3 yahoo    mobile TRUE          10       1       75 FALSE   
##  4 direct   mobile TRUE          10       1      908 FALSE   
##  5 bing     mobile FALSE          3      19      209 FALSE   
##  6 google   mobile TRUE           6       1      208 FALSE   
##  7 direct   laptop TRUE           9       1      738 FALSE   
##  8 direct   tablet FALSE          6      12      132 FALSE   
##  9 direct   mobile FALSE          9      14      406 TRUE    
## 10 yahoo    tablet FALSE          5       8       80 FALSE   
## 11 yahoo    mobile FALSE          7       1       19 FALSE

slice(ecom, n())

## # A tibble: 1 x 7
##   referrer device bouncers n_visit n_pages duration purchase
##   <fct>    <fct>  <lgl>      <dbl>   <dbl>    <dbl> <lgl>   
## 1 google   mobile TRUE           9       1      269 FALSE

Between

between() allows us to test if the values in a column lie between two specific values. In the below example, we check how many visits browsed pages between 5 and 15.

ecom_sample <- sample_n(ecom, 30)
  
ecom_sample %>%
  pull(n_pages) %>%
  between(5, 15) 

##  [1] FALSE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE FALSE  TRUE  TRUE FALSE
## [13]  TRUE  TRUE FALSE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
## [25] FALSE FALSE FALSE FALSE FALSE FALSE

Case When

case_when() is an alternative to if else. It allows us to lay down the conditions clearly and makes the code more readable. In the below example, we create a new column repeat_visit from n_visit (the number of previous visits).

ecom %>%
  mutate(
    repeat_visit = case_when(
      n_visit > 0 ~ TRUE,
      TRUE ~ FALSE
    )
  ) %>%
  select(n_visit, repeat_visit) 

## # A tibble: 1,000 x 2
##    n_visit repeat_visit
##      <dbl> <lgl>       
##  1      10 TRUE        
##  2       9 TRUE        
##  3       0 FALSE       
##  4       3 TRUE        
##  5       9 TRUE        
##  6       5 TRUE        
##  7      10 TRUE        
##  8      10 TRUE        
##  9       3 TRUE        
## 10       6 TRUE        
## # ... with 990 more rows

References

• https://dplyr.tidyverse.org/
• http://r4ds.had.co.nz/transform.html

Introduction

In the previous post we learnt about dplyr verbs and used them to compute average order value for an online retail company data. In this post, we will learn to combine tables using different *_join functions provided in dplyr.

Libraries, Code & Data

We will use the following packages:

• dplyr
• readr

The data sets can be downloaded from here and the codes from here.

library(dplyr)
library(readr)
options(tibble.width = Inf)

Case Study

For our case study, we will use two data sets. The first one, order, contains details of orders placed by different customers. The second data set, customer contains details of each customer. The below table displays the details stored in each data set.

Let us import both the data sets using read_csv.

order <- read_delim('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/order.csv', delim = ';')
order

## # A tibble: 300 x 3
##       id order_date amount
##    <dbl> <chr>       <dbl>
##  1   368 7/2/2016     365.
##  2   286 11/2/2016   2064.
##  3    28 2/22/2017    432.
##  4   309 3/5/2017     480.
##  5     2 12/28/2016   235.
##  6    31 12/30/2016  2745.
##  7   179 12/21/2016  2358.
##  8   484 11/24/2016  1031.
##  9   115 9/9/2016    1218.
## 10   340 5/6/2017    1184.
## # ... with 290 more rows

customer <- read_delim('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/customer.csv', delim = ';')
customer

## # A tibble: 91 x 3
##       id first_name city      
##    <dbl> <chr>      <chr>     
##  1     1 Elbertine  California
##  2     2 Marcella   Colorado  
##  3     3 Daria      Florida   
##  4     4 Sherilyn   Distric...
##  5     5 Ketty      Texas     
##  6     6 Jethro     California
##  7     7 Jeremiah   California
##  8     8 Constancia Texas     
##  9     9 Muire      Idaho     
## 10    10 Abigail    Texas     
## # ... with 81 more rows

Example Data

We will use another data set to illustrate how the different joins work. You can view the example data sets below.

Inner Join

Inner join return all rows from Age where there are matching values in Height, and all columns from Age and Height. If there are multiple matches between Age and Height, all combination of the matches are returned.

inner_join(customer, order, by = "id")

## # A tibble: 55 x 5
##       id first_name city       order_date amount
##    <dbl> <chr>      <chr>      <chr>       <dbl>
##  1     2 Marcella   Colorado   12/28/2016   235.
##  2     2 Marcella   Colorado   8/31/2016   1150.
##  3     5 Ketty      Texas      1/17/2017    346.
##  4     6 Jethro     California 1/27/2017   2317.
##  5     7 Jeremiah   California 6/21/2016    136.
##  6     7 Jeremiah   California 2/13/2017   1407.
##  7     7 Jeremiah   California 7/8/2016    1914.
##  8     8 Constancia Texas      11/5/2016   2461.
##  9     8 Constancia Texas      5/19/2017   2714.
## 10     9 Muire      Idaho      12/28/2016   187.
## # ... with 45 more rows

Left Join

Left join return all rows from Age, and all columns from Age and Height. Rows in Age with no match in Height will have NA values in the new columns. If there are multiple matches between Age and Height, all combinations of the matches are returned.

left_join(customer, order, by = "id")

## # A tibble: 104 x 5
##       id first_name city       order_date amount
##    <dbl> <chr>      <chr>      <chr>       <dbl>
##  1     1 Elbertine  California <NA>          NA 
##  2     2 Marcella   Colorado   12/28/2016   235.
##  3     2 Marcella   Colorado   8/31/2016   1150.
##  4     3 Daria      Florida    <NA>          NA 
##  5     4 Sherilyn   Distric... <NA>          NA 
##  6     5 Ketty      Texas      1/17/2017    346.
##  7     6 Jethro     California 1/27/2017   2317.
##  8     7 Jeremiah   California 6/21/2016    136.
##  9     7 Jeremiah   California 2/13/2017   1407.
## 10     7 Jeremiah   California 7/8/2016    1914.
## # ... with 94 more rows

Right Join

Right join return all rows from Height, and all columns from Age and Height. Rows in Height with no match in Age will have NA values in the new columns. If there are multiple matches between Age and Height, all combinations of the matches are returned.

right_join(customer, order, by = "id")

## # A tibble: 300 x 5
##       id first_name city       order_date amount
##    <dbl> <chr>      <chr>      <chr>       <dbl>
##  1     2 Marcella   Colorado   12/28/2016   235.
##  2     2 Marcella   Colorado   8/31/2016   1150.
##  3     5 Ketty      Texas      1/17/2017    346.
##  4     6 Jethro     California 1/27/2017   2317.
##  5     7 Jeremiah   California 6/21/2016    136.
##  6     7 Jeremiah   California 2/13/2017   1407.
##  7     7 Jeremiah   California 7/8/2016    1914.
##  8     8 Constancia Texas      11/5/2016   2461.
##  9     8 Constancia Texas      5/19/2017   2714.
## 10     9 Muire      Idaho      12/28/2016   187.
## # ... with 290 more rows

Semi Join

Semi join return all rows from Age where there are matching values in Height, keeping just columns from Age. A semi join differs from an inner join because an inner join will return one row of Age for each matching row of Height, where a semi join will never duplicate rows of Age.

semi_join(customer, order, by = "id")

## # A tibble: 42 x 3
##       id first_name city      
##    <dbl> <chr>      <chr>     
##  1     2 Marcella   Colorado  
##  2     5 Ketty      Texas     
##  3     6 Jethro     California
##  4     7 Jeremiah   California
##  5     8 Constancia Texas     
##  6     9 Muire      Idaho     
##  7    15 Valentijn  California
##  8    16 Monique    Missouri  
##  9    20 Colette    Texas     
## 10    28 Avrit      Texas     
## # ... with 32 more rows

Anti Join

Anti join return all rows from Age where there are not matching values in Height, keeping just columns from Age.

anti_join(customer, order, by = "id")

## # A tibble: 49 x 3
##       id first_name city      
##    <dbl> <chr>      <chr>     
##  1     1 Elbertine  California
##  2     3 Daria      Florida   
##  3     4 Sherilyn   Distric...
##  4    10 Abigail    Texas     
##  5    11 Wynne      Georgia   
##  6    12 Pietra     Minnesota 
##  7    13 Bram       Iowa      
##  8    14 Rees       New York  
##  9    17 Orazio     Louisiana 
## 10    18 Mason      Texas     
## # ... with 39 more rows

Full Join

Full join return all rows and all columns from both Age and Height. Where there are not matching values, returns NA for the one missing.

full_join(customer, order, by = "id")

## # A tibble: 349 x 5
##       id first_name city       order_date amount
##    <dbl> <chr>      <chr>      <chr>       <dbl>
##  1     1 Elbertine  California <NA>          NA 
##  2     2 Marcella   Colorado   12/28/2016   235.
##  3     2 Marcella   Colorado   8/31/2016   1150.
##  4     3 Daria      Florida    <NA>          NA 
##  5     4 Sherilyn   Distric... <NA>          NA 
##  6     5 Ketty      Texas      1/17/2017    346.
##  7     6 Jethro     California 1/27/2017   2317.
##  8     7 Jeremiah   California 6/21/2016    136.
##  9     7 Jeremiah   California 2/13/2017   1407.
## 10     7 Jeremiah   California 7/8/2016    1914.
## # ... with 339 more rows

References

• https://dplyr.tidyverse.org/
• http://r4ds.had.co.nz/relational-data.html

Introduction

According to a survey by CrowdFlower, data scientists spend most of their time cleaning and manipulating data rather than mining or modeling them for insights. As such, it becomes important to have tools that make data manipulation faster and easier. In today’s post, we introduce you to dplyr, a grammar of data manipulation.

Libraries, Code & Data

We will use the following libraries:

• dplyr
• and readr

The data sets can be downloaded from here and the codes from here.

library(dplyr)
library(readr)

dplyr Verbs

dplyr provides a set of verbs that help us solve the most common data manipulation challenges while working with tabular data (dataframes, tibbles):

• select
• filter
• arrange
• mutate
• summarise

Data

ecom <- 
  read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/web.csv',
    col_types = cols_only(device = col_factor(levels = c("laptop", "tablet", "mobile")),
      referrer = col_factor(levels = c("bing", "direct", "social", "yahoo", "google")),
      purchase = col_logical(), n_pages = col_double(), n_visit = col_double(), 
      duration = col_double(), order_value = col_double(), order_items = col_double()
    )
  )

ecom

## # A tibble: 1,000 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 google   laptop      10       1      693 FALSE              0           0
##  2 yahoo    tablet       9       1      459 FALSE              0           0
##  3 direct   laptop       0       1      996 FALSE              0           0
##  4 bing     tablet       3      18      468 TRUE               6         434
##  5 yahoo    mobile       9       1      955 FALSE              0           0
##  6 yahoo    laptop       5       5      135 FALSE              0           0
##  7 yahoo    mobile      10       1       75 FALSE              0           0
##  8 direct   mobile      10       1      908 FALSE              0           0
##  9 bing     mobile       3      19      209 FALSE              0           0
## 10 google   mobile       6       1      208 FALSE              0           0
## # ... with 990 more rows

Case Study

We will use dplyr to answer the following:

• what is the average order value by device types?
• what is the average number of pages visited by purchasers and non-purchasers?
• what is the average time on site for purchasers vs non-purchasers?
• what is the average number of pages visited by purchasers and non-purchasers using mobile?

Average Order Value

AOV by Devices

ecom %>%
  filter(purchase) %>%
  select(device, order_value) %>%
  group_by(device) %>%
  summarise_all(funs(revenue = sum, orders = n())) %>%
  mutate(
    aov = revenue / orders
  ) %>%
  select(device, aov) %>%
  arrange(aov)

## Warning: `funs()` is deprecated as of dplyr 0.8.0.
## Please use a list of either functions or lambdas: 
## 
##   # Simple named list: 
##   list(mean = mean, median = median)
## 
##   # Auto named with `tibble::lst()`: 
##   tibble::lst(mean, median)
## 
##   # Using lambdas
##   list(~ mean(., trim = .2), ~ median(., na.rm = TRUE))
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_warnings()` to see where this warning was generated.

## # A tibble: 3 x 2
##   device   aov
##   <fct>  <dbl>
## 1 tablet 1426.
## 2 mobile 1431.
## 3 laptop 1824.

Syntax

Before we start exploring the dplyr verbs, let us look at their syntax:

• the first argument is always a data.frame or tibble
• the subsequent arguments provide the information required for the verbs to take action
• the name of columns in the data need not be surrounded by quotes

Filter Rows

In order to compute the AOV, we must first separate the purchasers from non-purchasers. We will do this by filtering the data related to purchasers using the filter() function. It allows us to filter rows that meet a specific criteria/condition. The first argument is the name of the data frame and the rest of the arguments are expressions for filtering the data. Let us look at a few examples:

The first example we will look at filters all visits from device mobile. As we had learnt in the previous section, the first argument is our data set ecom and the next argument is the condition for filtering rows.

filter(ecom, device == "mobile")

## # A tibble: 344 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 yahoo    mobile       9       1      955 FALSE              0           0
##  2 yahoo    mobile      10       1       75 FALSE              0           0
##  3 direct   mobile      10       1      908 FALSE              0           0
##  4 bing     mobile       3      19      209 FALSE              0           0
##  5 google   mobile       6       1      208 FALSE              0           0
##  6 direct   mobile       9      14      406 TRUE               3         651
##  7 yahoo    mobile       7       1       19 FALSE              7        2423
##  8 google   mobile       5       1      147 FALSE              0           0
##  9 bing     mobile       0       7      196 FALSE              4         237
## 10 google   mobile      10       1      338 FALSE              0           0
## # ... with 334 more rows

We can specify multiple filtering conditions as well. In the below example, we specify two filter conditions:

• visit from device mobile
• resulted in a purchase or conversion

filter(ecom, device == "mobile", purchase)

## # A tibble: 36 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 direct   mobile       9      14      406 TRUE               3         651
##  2 bing     mobile       4      20      440 TRUE               3         184
##  3 bing     mobile       3      18      288 TRUE               6         764
##  4 social   mobile      10      11      242 TRUE               4         287
##  5 yahoo    mobile       6      14      322 TRUE               3        1443
##  6 google   mobile       1      18      252 TRUE               3        2449
##  7 social   mobile       7      16      352 TRUE              10        2824
##  8 direct   mobile       4      18      324 TRUE               3        1670
##  9 social   mobile       1      20      520 TRUE               5        1021
## 10 yahoo    mobile       0      13      351 TRUE              10         288
## # ... with 26 more rows

Here is another example where we specify multiple conditions:

• visit from device tablet
• made a purchase
• browsed less than 15 pages

filter(ecom, device == "tablet", purchase, n_pages < 15)

## # A tibble: 12 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 social   tablet       7      10      290 TRUE               9        1304
##  2 yahoo    tablet       2      14      364 TRUE               6        1667
##  3 google   tablet       7      12      324 TRUE               2        1358
##  4 direct   tablet       3      12      324 TRUE              10        1257
##  5 yahoo    tablet       0      13      390 TRUE               5        1748
##  6 social   tablet       2      12      300 TRUE               2        2754
##  7 direct   tablet       6      13      338 TRUE               5         683
##  8 yahoo    tablet       2      10      280 TRUE               4         293
##  9 social   tablet      10      10      290 TRUE               9          37
## 10 direct   tablet       3      10      260 TRUE               7         980
## 11 google   tablet       9      14      308 TRUE               7        2436
## 12 social   tablet      10      11      330 TRUE               1        2171

filter(ecom, purchase)

## # A tibble: 103 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 bing     tablet       3      18      468 TRUE               6         434
##  2 direct   mobile       9      14      406 TRUE               3         651
##  3 bing     tablet       5      16      368 TRUE               6        1049
##  4 social   tablet       7      10      290 TRUE               9        1304
##  5 direct   tablet       2      19      342 TRUE               5         622
##  6 social   tablet       9      20      420 TRUE               7        1613
##  7 bing     mobile       4      20      440 TRUE               3         184
##  8 yahoo    tablet       2      16      480 TRUE               9         286
##  9 bing     mobile       3      18      288 TRUE               6         764
## 10 yahoo    tablet       2      14      364 TRUE               6        1667
## # ... with 93 more rows

Select Columns

After filtering the data, we need to select relevent variables to compute the AOV. Remember, we do not need all the columns in the data to compute a required metric (in our case, AOV). The select() function allows us to select a subset of columns. The first argument is the name of the data frame and the subsequent arguments specify the columns by name or position.

To select the device and duration columns, we specify the data set i.e.  ecom followed by the name of the columns.

select(ecom, device, duration)

## # A tibble: 1,000 x 2
##    device duration
##    <fct>     <dbl>
##  1 laptop      693
##  2 tablet      459
##  3 laptop      996
##  4 tablet      468
##  5 mobile      955
##  6 laptop      135
##  7 mobile       75
##  8 mobile      908
##  9 mobile      209
## 10 mobile      208
## # ... with 990 more rows

We can select a set of columns using :. In the below example, we select all the columns starting from referrer up to order_items. Remember that we can use : only when the columns are adjacent to each other in the data set.

select(ecom, referrer:order_items)

## # A tibble: 1,000 x 7
##    referrer device n_visit n_pages duration purchase order_items
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>
##  1 google   laptop      10       1      693 FALSE              0
##  2 yahoo    tablet       9       1      459 FALSE              0
##  3 direct   laptop       0       1      996 FALSE              0
##  4 bing     tablet       3      18      468 TRUE               6
##  5 yahoo    mobile       9       1      955 FALSE              0
##  6 yahoo    laptop       5       5      135 FALSE              0
##  7 yahoo    mobile      10       1       75 FALSE              0
##  8 direct   mobile      10       1      908 FALSE              0
##  9 bing     mobile       3      19      209 FALSE              0
## 10 google   mobile       6       1      208 FALSE              0
## # ... with 990 more rows

What if you want to select all columns except a few? Typing the name of many columns can be cumbersome and may also result in spelling errors. We may use : only if the columns are adjacent to each other but that may not always be the case. dplyr allows us to specify columns that need not be selected using -. In the below example, we select all columns except n_pages and duration. Notice the - before both of them.

select(ecom, -n_pages, -duration)

## # A tibble: 1,000 x 6
##    referrer device n_visit purchase order_items order_value
##    <fct>    <fct>    <dbl> <lgl>          <dbl>       <dbl>
##  1 google   laptop      10 FALSE              0           0
##  2 yahoo    tablet       9 FALSE              0           0
##  3 direct   laptop       0 FALSE              0           0
##  4 bing     tablet       3 TRUE               6         434
##  5 yahoo    mobile       9 FALSE              0           0
##  6 yahoo    laptop       5 FALSE              0           0
##  7 yahoo    mobile      10 FALSE              0           0
##  8 direct   mobile      10 FALSE              0           0
##  9 bing     mobile       3 FALSE              0           0
## 10 google   mobile       6 FALSE              0           0
## # ... with 990 more rows

select(ecom, device, order_value)

## # A tibble: 1,000 x 2
##    device order_value
##    <fct>        <dbl>
##  1 laptop           0
##  2 tablet           0
##  3 laptop           0
##  4 tablet         434
##  5 mobile           0
##  6 laptop           0
##  7 mobile           0
##  8 mobile           0
##  9 mobile           0
## 10 mobile           0
## # ... with 990 more rows

# filter all visits that resulted in a purchase
ecom1 <- filter(ecom, purchase)

# select the relevant columns
ecom2 <- select(ecom1, device, order_value)

# view data
ecom2

## # A tibble: 103 x 2
##    device order_value
##    <fct>        <dbl>
##  1 tablet         434
##  2 mobile         651
##  3 tablet        1049
##  4 tablet        1304
##  5 tablet         622
##  6 tablet        1613
##  7 mobile         184
##  8 tablet         286
##  9 mobile         764
## 10 tablet        1667
## # ... with 93 more rows

Grouping Data

We need to compute the total order value and total order items for each device in order to compute their AOV. To achieve this, we need to group the selected order_value and order_items by device type. group_by() allows us to group or split data based on particular (discrete) variable. The first argument is the name of the data set and the second argument is the name of the column based on which the data will be split.

To split the data by referrer type, we use group_by and specify the data set i.e. ecom and the column based on which to split the data i.e. referrer.

group_by(ecom, referrer)

## # A tibble: 1,000 x 8
## # Groups:   referrer [5]
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 google   laptop      10       1      693 FALSE              0           0
##  2 yahoo    tablet       9       1      459 FALSE              0           0
##  3 direct   laptop       0       1      996 FALSE              0           0
##  4 bing     tablet       3      18      468 TRUE               6         434
##  5 yahoo    mobile       9       1      955 FALSE              0           0
##  6 yahoo    laptop       5       5      135 FALSE              0           0
##  7 yahoo    mobile      10       1       75 FALSE              0           0
##  8 direct   mobile      10       1      908 FALSE              0           0
##  9 bing     mobile       3      19      209 FALSE              0           0
## 10 google   mobile       6       1      208 FALSE              0           0
## # ... with 990 more rows

# split ecom2 by device type
ecom3 <- group_by(ecom2, device)
ecom3

## # A tibble: 103 x 2
## # Groups:   device [3]
##    device order_value
##    <fct>        <dbl>
##  1 tablet         434
##  2 mobile         651
##  3 tablet        1049
##  4 tablet        1304
##  5 tablet         622
##  6 tablet        1613
##  7 mobile         184
##  8 tablet         286
##  9 mobile         764
## 10 tablet        1667
## # ... with 93 more rows

Summarise Data

The next step is to compute the total order value and total order items for each device. i.e. we need to reduce the order value and order items data to a single summary. We can achieve this using summarise(). As usual, the first argument is the name of a data set and the subsequent arguments are functions that can summarise data. For example, we can use min, max, sum, mean etc.

Let us compute the average number of pages browsed by referrer type:

• split data by referrer type
• compute the average number of pages using mean

# split data by referrer type
step_1 <- group_by(ecom, referrer)

# compute average number of pages
step_2 <- summarise(step_1, mean(n_pages))

## `summarise()` ungrouping (override with `.groups` argument)

step_2

## # A tibble: 5 x 2
##   referrer `mean(n_pages)`
## * <fct>              <dbl>
## 1 bing                6.13
## 2 direct              6.38
## 3 social              5.42
## 4 yahoo               5.99
## 5 google              5.73

Now let us compute both the mean and the median.

# split data by referrer type
step_1 <- group_by(ecom, referrer)

# compute average number of pages
step_2 <- summarise(step_1, mean(n_pages), median(n_pages))

## `summarise()` ungrouping (override with `.groups` argument)

step_2

## # A tibble: 5 x 3
##   referrer `mean(n_pages)` `median(n_pages)`
## * <fct>              <dbl>             <dbl>
## 1 bing                6.13                 1
## 2 direct              6.38                 1
## 3 social              5.42                 1
## 4 yahoo               5.99                 2
## 5 google              5.73                 1

Another way to achieve the above result is to use the summarise_all() function. How does that work? It generates the specified summary for all the columns in the data set except for the column based on which the data has been grouped or split. So we need to ensure that the data does not have any irrelevant columns.

• split data by referrer type
• select order_value and order_items
• compute the average number of pages by applying the mean function to all the columns

# select relevant columns
step_1 <- select(ecom, referrer, order_value)

# split data by referrer type
step_2 <- group_by(step_1, referrer)

# compute average number of pages
step_3 <- summarise_all(step_2, funs(mean))
step_3

## # A tibble: 5 x 2
##   referrer order_value
## * <fct>          <dbl>
## 1 bing            316.
## 2 direct          441.
## 3 social          380.
## 4 yahoo           470.
## 5 google          328.

Let us compute mean and median number of pages for each referre type using summarise_all.

# select relevant columns
step_1 <- select(ecom, referrer, order_value)

# split data by referrer type
step_2 <- group_by(step_1, referrer)

# compute mean and median number of pages
step_3 <- summarise_all(step_2, funs(mean, median))
step_3

## # A tibble: 5 x 3
##   referrer  mean median
## * <fct>    <dbl>  <dbl>
## 1 bing      316.      0
## 2 direct    441.      0
## 3 social    380.      0
## 4 yahoo     470.      0
## 5 google    328.      0

ecom4 <- summarise(ecom3, revenue = sum(order_value),
          orders = n())

## `summarise()` ungrouping (override with `.groups` argument)

ecom4

## # A tibble: 3 x 3
##   device revenue orders
## * <fct>    <dbl>  <int>
## 1 laptop   56531     31
## 2 tablet   51321     36
## 3 mobile   51504     36

ecom4 <- summarise_all(ecom3, funs(revenue = sum, orders = n()))
ecom4

## # A tibble: 3 x 3
##   device revenue orders
## * <fct>    <dbl>  <int>
## 1 laptop   56531     31
## 2 tablet   51321     36
## 3 mobile   51504     36

Create Columns

To create a new column, we will use mutate(). The first argument is the name of the data set and the subsequent arguments are expressions for creating new columns based out of existing columns.

Let us add a new column avg_page_time i.e. time on site divided by number of pages visited.

# select duration and n_pages from ecom
mutate_1 <- select(ecom, n_pages, duration)
mutate(mutate_1, avg_page_time = duration / n_pages)

## # A tibble: 1,000 x 3
##    n_pages duration avg_page_time
##      <dbl>    <dbl>         <dbl>
##  1       1      693           693
##  2       1      459           459
##  3       1      996           996
##  4      18      468            26
##  5       1      955           955
##  6       5      135            27
##  7       1       75            75
##  8       1      908           908
##  9      19      209            11
## 10       1      208           208
## # ... with 990 more rows

We can create new columns based on other columns created using mutate. Let us create another column sqrt_avg_page_time i.e. square root of the average time on page using avg_page_time.

mutate(mutate_1,
       avg_page_time = duration / n_pages,
       sqrt_avg_page_time = sqrt(avg_page_time))

## # A tibble: 1,000 x 4
##    n_pages duration avg_page_time sqrt_avg_page_time
##      <dbl>    <dbl>         <dbl>              <dbl>
##  1       1      693           693              26.3 
##  2       1      459           459              21.4 
##  3       1      996           996              31.6 
##  4      18      468            26               5.10
##  5       1      955           955              30.9 
##  6       5      135            27               5.20
##  7       1       75            75               8.66
##  8       1      908           908              30.1 
##  9      19      209            11               3.32
## 10       1      208           208              14.4 
## # ... with 990 more rows

ecom5 <- mutate(ecom4, aov = revenue / orders)
ecom5

## # A tibble: 3 x 4
##   device revenue orders   aov
## * <fct>    <dbl>  <int> <dbl>
## 1 laptop   56531     31 1824.
## 2 tablet   51321     36 1426.
## 3 mobile   51504     36 1431.

Select Columns

The last step is to select the relevant columns. We will select the device type and the corresponding aov while getting rid of other columns. Use select() to extract the relevant columns.

ecom6 <- select(ecom5, device, aov)
ecom6

## # A tibble: 3 x 2
##   device   aov
##   <fct>  <dbl>
## 1 laptop 1824.
## 2 tablet 1426.
## 3 mobile 1431.

Arrange Data

Arranging data in ascending or descending order is one of the most common tasks in data manipulation. We can use arrange to arrange data by different columns. Let us say we want to arrange data by the number of pages browsed.

arrange(ecom, n_pages)

## # A tibble: 1,000 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 google   laptop      10       1      693 FALSE              0           0
##  2 yahoo    tablet       9       1      459 FALSE              0           0
##  3 direct   laptop       0       1      996 FALSE              0           0
##  4 yahoo    mobile       9       1      955 FALSE              0           0
##  5 yahoo    mobile      10       1       75 FALSE              0           0
##  6 direct   mobile      10       1      908 FALSE              0           0
##  7 google   mobile       6       1      208 FALSE              0           0
##  8 direct   laptop       9       1      738 FALSE              0           0
##  9 yahoo    mobile       7       1       19 FALSE              7        2423
## 10 bing     laptop       1       1      995 FALSE              0           0
## # ... with 990 more rows

If we want to arrange the data in descending order, we can use desc(). Let us arrange the data in descending order.

arrange(ecom , desc(n_pages))

## # A tibble: 1,000 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 social   tablet       9      20      420 TRUE               7        1613
##  2 bing     mobile       4      20      440 TRUE               3         184
##  3 yahoo    tablet       0      20      200 FALSE              0           0
##  4 direct   tablet       6      20      580 TRUE               5        1155
##  5 social   mobile       1      20      520 TRUE               5        1021
##  6 google   mobile       8      20      300 TRUE               7        2091
##  7 social   laptop       4      20      200 FALSE              0           0
##  8 yahoo    mobile       3      20      480 FALSE              0           0
##  9 social   laptop      10      20      280 TRUE               1        2011
## 10 yahoo    mobile       2      20      240 FALSE              0           0
## # ... with 990 more rows

Data can be arranged by multiple variables as well. Let us arrange data first by number of visits and then by number of pages in a descending order.

arrange(ecom, n_visit, desc(n_pages))

## # A tibble: 1,000 x 8
##    referrer device n_visit n_pages duration purchase order_items order_value
##    <fct>    <fct>    <dbl>   <dbl>    <dbl> <lgl>          <dbl>       <dbl>
##  1 yahoo    tablet       0      20      200 FALSE              0           0
##  2 google   laptop       0      19      418 TRUE               2         996
##  3 bing     laptop       0      18      180 FALSE              0           0
##  4 yahoo    laptop       0      18      522 TRUE               8        1523
##  5 direct   tablet       0      18      252 FALSE              0           0
##  6 social   laptop       0      17      204 FALSE              0           0
##  7 bing     laptop       0      17      272 TRUE               9        1384
##  8 bing     mobile       0      16      272 FALSE              0           0
##  9 yahoo    mobile       0      15      255 FALSE              0           0
## 10 direct   laptop       0      15      255 FALSE              0           0
## # ... with 990 more rows

arrange(ecom6, aov)

## # A tibble: 3 x 2
##   device   aov
##   <fct>  <dbl>
## 1 tablet 1426.
## 2 mobile 1431.
## 3 laptop 1824.

AOV by Devices

Let us combine all the code from the above steps:

ecom1 <- filter(ecom, purchase)
ecom2 <- select(ecom1, device, order_value)
ecom3 <- group_by(ecom2, device)
ecom4 <- summarise_all(ecom3, funs(revenue = sum, orders = n()))
ecom5 <- mutate(ecom4, aov = revenue / orders)
ecom6 <- select(ecom5, device, aov)
ecom7 <- arrange(ecom6, aov)
ecom7

## # A tibble: 3 x 2
##   device   aov
##   <fct>  <dbl>
## 1 tablet 1426.
## 2 mobile 1431.
## 3 laptop 1824.

If you observe, at each step we create a new variable(data frame) and then use it as an input in the next step i.e. the output from one step becomes the input for the next. Can we achieve the final outcome i.e. ecom7 without creating the intermediate data (ecom1 - ecom6)? Yes, we can. We will use the %>% operator to chain the steps and get rid of the intermediate data.

ecom %>%
  filter(purchase) %>%
  select(device, order_value) %>%
  group_by(device) %>%
  summarise_all(funs(revenue = sum, orders = n())) %>%
  mutate(
    aov = revenue / orders
  ) %>%
  select(device, aov) %>%
  arrange(aov)

## # A tibble: 3 x 2
##   device   aov
##   <fct>  <dbl>
## 1 tablet 1426.
## 2 mobile 1431.
## 3 laptop 1824.

Your Turn

• what is the average number of pages visited by purchasers and non-purchasers?
• what is the average time on site for purchasers vs non-purchasers?
• what is the average number of pages visited by purchasers and non-purchasers using mobile?

References

• https://dplyr.tidyverse.org/
• http://r4ds.had.co.nz/transform.html