<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>Rsquared Academy Blog: R</title>
<link>https://blog.rsquaredacademy.com/index-r.html</link>
<atom:link href="https://blog.rsquaredacademy.com/index-r.xml" rel="self" type="application/rss+xml"/>
<description>R tutorials and R package posts from the Rsquared Academy Blog. This page&#39;s feed (index-r.xml) carries R content only.</description>
<generator>quarto-1.10.18</generator>
<lastBuildDate>Thu, 19 Nov 2026 00:00:00 GMT</lastBuildDate>
<item>
  <title>olsrr 0.7.1</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/olsrr-0-7-1/</link>
  <description><![CDATA[ 




<p><img src="https://blog.rsquaredacademy.com/img/olsrr-0-7-1.png" class="img-fluid" style="width:100.0%" alt="olsrr hex logo next to the tagline Tools for building and checking OLS regression models in R, the line 0.7.1, corrected test results, the package name olsrr and olsrr.rsquaredacademy.com"></p>
<style>
.ba-card{display:grid;grid-template-columns:1fr 1fr;gap:0;border:1px solid #d0d7de;border-radius:8px;overflow:hidden;margin:1rem 0 .5rem;font-size:.92rem}
.ba-card>div{padding:.75rem 1rem}
.ba-card .ba-before{background:#fff5f5;border-right:1px solid #d0d7de}
.ba-card .ba-after{background:#f3fbf5}
.ba-card h5{margin:0 0 .4rem;font-size:.8rem;letter-spacing:.04em;text-transform:uppercase;color:#57606a}
.ba-card code{font-size:.88rem}
.ba-card table{margin:0;width:100%}
.ba-card td{padding:.15rem .4rem;border:0}
.ba-card .chg{font-weight:700;background:#fff3b0;border-radius:3px;padding:0 .2rem}
.ba-call{font-size:.85rem;color:#57606a;margin:-.25rem 0 1rem}
.ba-note{font-size:.88rem;color:#57606a;border-left:3px solid #d0d7de;padding-left:.75rem;margin:.5rem 0 1.25rem}
@media (max-width:600px){.ba-card{grid-template-columns:1fr}.ba-card .ba-before{border-right:0;border-bottom:1px solid #d0d7de}}
</style>
<p><code>olsrr</code> 0.7.1 is on CRAN. <code>olsrr</code> provides tools for building and checking ordinary least squares regression models in R: regression summaries, variable selection, heteroskedasticity and normality tests, collinearity diagnostics, influence measures and residual plots, all built around <code>lm()</code> objects.</p>
<p>We don’t usually write about patch releases, but this one deserves a post:</p>
<ul>
<li>Several test and summary results change: Holm-adjusted Breusch-Pagan <img src="https://latex.codecogs.com/png.latex?p">-values, the Cramér–von Mises normality test, and the residual MSE/RMSE in model summaries. If you report these numbers, please re-run your analyses.</li>
<li>Every calculation and plot was checked against base R and reference packages (<code>car</code>, <code>lmtest</code> and <code>nortest</code>) for this release.</li>
<li>It is the first <code>olsrr</code> post since our <a href="../../posts/introducing-olsrr/index.html">introduction in 2019</a>, so it also covers what has been added since then.</li>
</ul>
<p><strong>Who should read what:</strong> if you report Holm-adjusted Breusch-Pagan <img src="https://latex.codecogs.com/png.latex?p">-values, Cramér–von Mises normality tests or MSE/RMSE from <code>olsrr</code>, read Behaviour changes first. Everyone else can skip to What’s new since 2019.</p>
<p>You can install it from CRAN with:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install.packages</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"olsrr"</span>)</span></code></pre></div></div>
</details>
</div>
<p>You can see a full list of changes in the <a href="https://github.com/rsquaredacademy/olsrr/releases/tag/v0.7.1">release notes</a>.</p>
<section id="behaviour-changes" class="level2">
<h2 class="anchored" data-anchor-id="behaviour-changes">Behaviour changes</h2>
<p>These fixes change numbers you may already have in reports. In each case 0.7.1 now agrees with the base R or reference implementation, so the new values are the ones to use.</p>
<div class="ba-note">
<p>How we checked: every number in this post was re-run on CRAN <code>olsrr</code> 0.7.0 and 0.7.1, and the release passed an independent audit against base R, <code>car</code>, <code>lmtest</code> and <code>nortest</code> before it was submitted.</p>
</div>
<section id="breusch-pagan-test-holm-adjusted-p-values" class="level3">
<h3 class="anchored" data-anchor-id="breusch-pagan-test-holm-adjusted-p-values">Breusch-Pagan test: Holm-adjusted p-values</h3>
<p>With <code>ols_test_breusch_pagan(..., rhs = TRUE, multiple = TRUE, p.adj = "holm")</code>, version 0.7.0 sorted the raw <img src="https://latex.codecogs.com/png.latex?p">-values and multiplied them by reversed ranks without the cumulative-maximum step of Holm’s procedure. A predictor could keep an adjusted <img src="https://latex.codecogs.com/png.latex?p">-value that was too small and be wrongly flagged at <img src="https://latex.codecogs.com/png.latex?%07lpha%20=%200.05">. Version 0.7.1 uses <code>stats::p.adjust(p, "holm")</code>. In this three-predictor example the raw <img src="https://latex.codecogs.com/png.latex?p">-values are <code>0.0465</code>, <code>1.07e-05</code> and <code>0.0410</code>:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>)</span>
<span id="cb2-2">n <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span></span>
<span id="cb2-3">z <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n)</span>
<span id="cb2-4">d <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x1 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x2 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x3 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n))</span>
<span id="cb2-5">d<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.55</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> z)</span>
<span id="cb2-6"></span>
<span id="cb2-7">m_bp <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> x2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> x3, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> d)</span>
<span id="cb2-8"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_test_breusch_pagan</span>(m_bp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rhs =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">multiple =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p.adj =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"holm"</span>)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>
 Breusch Pagan Test for Heteroskedasticity
 -----------------------------------------
 Ho: the variance is constant            
 Ha: the variance is not constant        

       Data         
 -------------------
 Response : y 
 Variables: x1 x2 x3 

           Test Summary (Holm's p values)           
 -------------------------------------------------
  Variable           chi2       df         p       
 -------------------------------------------------
  x1                3.962893     1    8.192284e-02 
  x2               19.381555     1    3.211077e-05 
  x3                4.177610     1    8.192284e-02 
 -------------------------------------------------
  simultaneous     21.479644     3    8.369002e-05 
 -------------------------------------------------</code></pre>
</div>
</div>
<div class="ba-card" aria-label="Before and after: ols_test_breusch_pagan(m_bp, rhs = TRUE, multiple = TRUE, p.adj = " holm")"="">
<div class="ba-before"><h5 class="anchored">olsrr 0.7.0</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>x1</td>
<td><span class="chg">0.0465</span></td>
</tr>
<tr class="even">
<td>x2</td>
<td>3.21e-05</td>
</tr>
<tr class="odd">
<td>x3</td>
<td>0.0819</td>
</tr>
</tbody>
</table>
</div>
<div class="ba-after"><h5 class="anchored">olsrr 0.7.1</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>x1</td>
<td><span class="chg">0.0819</span></td>
</tr>
<tr class="even">
<td>x2</td>
<td>3.21e-05</td>
</tr>
<tr class="odd">
<td>x3</td>
<td>0.0819</td>
</tr>
</tbody>
</table>
</div>
</div>
<p class="ba-call">Same call on both CRAN versions: <code>ols_test_breusch_pagan(m_bp, rhs = TRUE, multiple = TRUE, p.adj = "holm")</code>. Changed values are highlighted.</p>
<details>
<summary>
Code that produced both values
</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb4-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run once with each version, e.g. in two sessions:</span></span>
<span id="cb4-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># remotes::install_version("olsrr", "0.7.0")   # then</span></span>
<span id="cb4-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("olsrr")                    # 0.7.1</span></span>
<span id="cb4-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(olsrr)</span>
<span id="cb4-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">set.seed</span>(<span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">15</span>)</span>
<span id="cb4-6">n <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">60</span></span>
<span id="cb4-7">z <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n)</span>
<span id="cb4-8">d <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">data.frame</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x1 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x2 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n), <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">x3 =</span> z <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n))</span>
<span id="cb4-9">d<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>y <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">rnorm</span>(n) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">exp</span>(<span class="fl" style="color: #AD0000;
background-color: null;
font-style: inherit;">0.55</span> <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">*</span> z)</span>
<span id="cb4-10">m_bp <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> x1 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> x2 <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> x3, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> d)</span>
<span id="cb4-11"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_test_breusch_pagan</span>(m_bp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rhs =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">multiple =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">TRUE</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">p.adj =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"holm"</span>)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>p</span></code></pre></div></div>
</details>
<p>In 0.7.0, <code>x1</code> remained below <code>0.05</code> (<code>0.0465</code>) even though <code>x3</code> had a smaller raw <img src="https://latex.codecogs.com/png.latex?p">-value and adjusted to <code>0.0819</code>. In 0.7.1, <code>x1</code> is <code>0.0819</code>, matching <code>stats::p.adjust()</code>.</p>
<p><strong>What to do:</strong> re-run any Holm-adjusted Breusch-Pagan tests; the 0.7.1 values match <code>stats::p.adjust()</code>.</p>
</section>
<section id="normality-test-cramérvon-mises" class="level3">
<h3 class="anchored" data-anchor-id="normality-test-cramérvon-mises">Normality test: Cramér–von Mises</h3>
<p>In 0.7.0, <code>ols_test_normality()</code> compared the Cramér–von Mises statistic with a standard uniform distribution instead of a normal distribution with mean and standard deviation estimated from the residuals, so it rejected normality for almost any regression. Version 0.7.1 calls <code>nortest::cvm.test()</code>:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb5" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb5-1">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> disp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb5-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_test_normality</span>(model)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>-----------------------------------------------
       Test             Statistic       pvalue  
-----------------------------------------------
Shapiro-Wilk              0.9366         0.0600 
Kolmogorov-Smirnov        0.1152         0.7464 
Cramer-von Mises          0.0764         0.2230 
Anderson-Darling          0.5859         0.1188 
-----------------------------------------------</code></pre>
</div>
</div>
<div class="ba-card" aria-label="Before and after: ols_test_normality(lm(mpg ~ disp + hp + wt + qsec, mtcars))">
<div class="ba-before"><h5 class="anchored">olsrr 0.7.0</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>Cramér–von Mises W</td>
<td><span class="chg">2.8122</span></td>
</tr>
<tr class="even">
<td>p-value</td>
<td><span class="chg">0.0000</span></td>
</tr>
</tbody>
</table>
</div>
<div class="ba-after"><h5 class="anchored">olsrr 0.7.1</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>Cramér–von Mises W</td>
<td><span class="chg">0.0764</span></td>
</tr>
<tr class="even">
<td>p-value</td>
<td><span class="chg">0.2230</span></td>
</tr>
</tbody>
</table>
</div>
</div>
<p class="ba-call">Same call on both CRAN versions: <code>ols_test_normality(lm(mpg ~ disp + hp + wt + qsec, mtcars))</code>. Changed values are highlighted.</p>
<details>
<summary>
Code that produced both values
</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb7" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb7-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run once with each version, e.g. in two sessions:</span></span>
<span id="cb7-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># remotes::install_version("olsrr", "0.7.0")   # then</span></span>
<span id="cb7-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("olsrr")                    # 0.7.1</span></span>
<span id="cb7-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(olsrr)</span>
<span id="cb7-5">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> disp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb7-6"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_test_normality</span>(model)<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>cramer</span></code></pre></div></div>
</details>
<p>In 0.7.1, the Cramér–von Mises result (<code>p = 0.2230</code>) agrees with <code>nortest::cvm.test(residuals(model))</code> and aligns qualitatively with the Shapiro-Wilk (<code>0.0600</code>) and Anderson-Darling (<code>0.1188</code>) tests on the same residuals. (See What’s next about the Kolmogorov-Smirnov row.)</p>
<p><strong>What to do:</strong> treat earlier Cramér–von Mises rejections from <code>olsrr</code> as unreliable and re-run the test.</p>
</section>
<section id="residual-mse-and-rmse" class="level3">
<h3 class="anchored" data-anchor-id="residual-mse-and-rmse">Residual MSE and RMSE</h3>
<p>In 0.7.0, <code>ols_regress()</code> and <code>ols_model_performance()</code> divided the residual sum of squares by <img src="https://latex.codecogs.com/png.latex?n"> rather than by the residual degrees of freedom (<a href="https://github.com/rsquaredacademy/olsrr/issues/221">#221</a>, reported by <a href="https://github.com/viqule11">@viqule11</a>). For <code>lm(mpg ~ disp + hp + wt + qsec, data = mtcars)</code> (<img src="https://latex.codecogs.com/png.latex?n%20=%2032">, 27 residual degrees of freedom) that gave <code>MSE = 5.8011</code> and <code>RMSE = 2.4085</code>. Version 0.7.1 reports <code>MSE = 6.8754</code> and <code>RMSE = 2.6221</code>, matching the ANOVA residual mean square and <code>summary(model)$sigma</code>.</p>
<div class="ba-card" aria-label="Before and after: ols_regress(lm(mpg ~ disp + hp + wt + qsec, mtcars))">
<div class="ba-before"><h5 class="anchored">olsrr 0.7.0</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>MSE</td>
<td><span class="chg">5.8011</span></td>
</tr>
<tr class="even">
<td>RMSE</td>
<td><span class="chg">2.4085</span></td>
</tr>
</tbody>
</table>
</div>
<div class="ba-after"><h5 class="anchored">olsrr 0.7.1</h5>
<table class="caption-top table">
<tbody>
<tr class="odd">
<td>MSE</td>
<td><span class="chg">6.8754</span></td>
</tr>
<tr class="even">
<td>RMSE</td>
<td><span class="chg">2.6221</span></td>
</tr>
</tbody>
</table>
</div>
</div>
<p class="ba-call">Same call on both CRAN versions: <code>ols_regress(lm(mpg ~ disp + hp + wt + qsec, mtcars))</code>. Changed values are highlighted.</p>
<details>
<summary>
Code that produced both values
</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb8" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb8-1"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># Run once with each version, e.g. in two sessions:</span></span>
<span id="cb8-2"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># remotes::install_version("olsrr", "0.7.0")   # then</span></span>
<span id="cb8-3"><span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># install.packages("olsrr")                    # 0.7.1</span></span>
<span id="cb8-4"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">library</span>(olsrr)</span>
<span id="cb8-5">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> disp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb8-6">r <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_regress</span>(model)</span>
<span id="cb8-7"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">mse =</span> r<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>mse, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">rmse =</span> r<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>rmse)</span></code></pre></div></div>
</details>
<p><strong>What to do:</strong> expect slightly larger MSE and RMSE values; they now match base R.</p>
</section>
</section>
<section id="whats-new-since-2019" class="level2">
<h2 class="anchored" data-anchor-id="whats-new-since-2019">What’s new since 2019</h2>
<section id="stepwise-selection-with-locked-predictors" class="level3">
<h3 class="anchored" data-anchor-id="stepwise-selection-with-locked-predictors">Stepwise Selection with Locked Predictors</h3>
<p>Between versions 0.6.0 and 0.7.1, the variable selection family expanded to cover <img src="https://latex.codecogs.com/png.latex?p">-values, AIC, SBC (BIC), Sawa’s BIC (<code>sbic</code>), <img src="https://latex.codecogs.com/png.latex?R%5E2">, and adjusted <img src="https://latex.codecogs.com/png.latex?R%5E2">. All forward, backward, and both-direction selectors now accept <code>include</code> and <code>exclude</code> arguments (by name or column index) so baseline covariates or confounders stay in the model throughout selection.</p>
<p>On the <code>surgical</code> dataset, locking <code>age</code> and <code>alc_heavy</code> via <code>include</code> keeps both predictors in the base model (Step 0, <code>AIC = 795.956</code>) and through every subsequent addition and removal step, even though <code>age</code> itself has <img src="https://latex.codecogs.com/png.latex?p%20=%200.473"> in the final fit:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb9" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb9-1">m_surg <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(</span>
<span id="cb9-2">  y <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> bcs <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> pindex <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> enzyme_test <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> liver_test <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> age <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> gender <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> alc_mod <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> alc_heavy,</span>
<span id="cb9-3">  <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> surgical</span>
<span id="cb9-4">)</span>
<span id="cb9-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_step_both_aic</span>(m_surg, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">include =</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">c</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"age"</span>, <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"alc_heavy"</span>))</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>
                                Stepwise Summary                                
------------------------------------------------------------------------------
Step    Variable             AIC        SBC       SBIC        R2       Adj. R2 
------------------------------------------------------------------------------
 0      Base Model         795.956    803.912    637.696    0.17900    0.14680 
 1      liver_test (+)     763.110    773.055    606.421    0.56938    0.54354 
 2      enzyme_test (+)    752.416    764.350    596.755    0.65959    0.63180 
 3      pindex (+)         737.680    751.603    585.012    0.75030    0.72429 
 4      bcs (+)            732.494    748.406    581.938    0.78142    0.75351 
 5      liver_test (-)     732.339    746.262    580.934    0.77382    0.75026 
------------------------------------------------------------------------------

Final Model Output 
------------------

                           Model Summary                            
-------------------------------------------------------------------
R                         0.880       RMSE                 198.590 
R-Squared                 0.774       MSE                39438.163 
Adj. R-Squared            0.750       Coef. Var             28.286 
Pred R-Squared            0.688       AIC                  732.339 
MAE                     140.528       SBC                  746.262 
-------------------------------------------------------------------
 RMSE: Root Mean Square Error 
 MSE: Mean Square Error 
 MAE: Mean Absolute Error 
 AIC: Akaike Information Criteria 
 SBC: Schwarz Bayesian Criteria 

                                 ANOVA                                  
-----------------------------------------------------------------------
                   Sum of                                              
                  Squares        DF    Mean Square      F         Sig. 
-----------------------------------------------------------------------
Regression    6476488.730         5    1295297.746    32.844    0.0000 
Residual      1893031.807        48      39438.163                     
Total         8369520.537        53                                    
-----------------------------------------------------------------------

                                      Parameter Estimates                                        
------------------------------------------------------------------------------------------------
      model         Beta    Std. Error    Std. Beta      t        Sig         lower       upper 
------------------------------------------------------------------------------------------------
(Intercept)    -1237.653       225.517                 -5.488    0.000    -1691.085    -784.221 
        age       -1.787         2.473       -0.050    -0.723    0.473       -6.759       3.184 
  alc_heavy      306.655        73.188        0.303     4.190    0.000      159.500     453.809 
enzyme_test       11.244         1.314        0.601     8.555    0.000        8.601      13.887 
     pindex       10.062         1.633        0.428     6.163    0.000        6.779      13.344 
        bcs       81.587        17.870        0.329     4.566    0.000       45.657     117.516 
------------------------------------------------------------------------------------------------</code></pre>
</div>
</div>
</section>
<section id="ten-panel-diagnostic-grid" class="level3">
<h3 class="anchored" data-anchor-id="ten-panel-diagnostic-grid">Ten-Panel Diagnostic Grid</h3>
<p><code>ols_plot_diagnostics()</code> gathers ten residual, fit, leverage, and influence views for an <code>lm</code> object: residuals vs fitted values, deleted studentized residuals vs predicted values, studentized residuals vs leverage, normal Q-Q plot, actual vs fitted values, Cook’s D chart, centered fit spread, residual spread, residual histogram, and residual box plot.</p>
<p>Calling <code>ols_plot_diagnostics(model)</code> directly pages the panels in 2×2 grids (and in 0.7.1 draws cleanly inside scripts, functions, and loops as well as at the top level). Setting <code>print_plot = FALSE</code> returns the ten <code>ggplot</code> objects without drawing them first, so they can be arranged in a single layout:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb11" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb11-1">model <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> disp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb11-2">panels <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_plot_diagnostics</span>(model, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">print_plot =</span> <span class="cn" style="color: #8f5902;
background-color: null;
font-style: inherit;">FALSE</span>)</span>
<span id="cb11-3">gridExtra<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">grid.arrange</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">grobs =</span> panels, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">ncol =</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">2</span>, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">top =</span> <span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"Regression Diagnostics"</span>)</span></code></pre></div></div>
</details>
<div class="cell-output-display">
<div>
<figure class="figure">
<p><img src="https://blog.rsquaredacademy.com/posts/olsrr-0-7-1/index_files/figure-html/diag-grid-1.png" class="img-fluid figure-img" alt="Ten-panel regression diagnostic grid for mpg ~ disp + hp + wt + qsec on mtcars, arranged in five rows of two panels: Residual vs Fitted Values, Deleted Studentized Residual vs Predicted Values, Outlier and Leverage Diagnostics, Normal Q-Q Plot, Actual vs Fitted for mpg, Cook's D Chart, centered fit spread and residual spread plots, Residual Histogram with normal curve, and Residual Box Plot." width="864"></p>
</figure>
</div>
</div>
</div>
</section>
<section id="model-performance-and-multi-model-comparison" class="level3">
<h3 class="anchored" data-anchor-id="model-performance-and-multi-model-comparison">Model Performance and Multi-Model Comparison</h3>
<p>Added in 0.7.0 and refined in 0.7.1, <code>ols_model_performance()</code> reports fit metrics for a single model, while <code>ols_compare_model_performance()</code> places competing models side by side across <img src="https://latex.codecogs.com/png.latex?R%5E2">, adjusted <img src="https://latex.codecogs.com/png.latex?R%5E2">, predicted <img src="https://latex.codecogs.com/png.latex?R%5E2">, AIC, SBC, SBIC, RMSE, and residual standard error (<code>sigma</code>):</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb12" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb12-1">m1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb12-2">m2 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb12-3">m3 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> qsec <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> disp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> mtcars)</span>
<span id="cb12-4"></span>
<span id="cb12-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_compare_model_performance</span>(m1, m2, m3)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>  name        r2    r2_adj   r2_pred      aic      sbc     sbic     rmse
1   m1 0.8267855 0.8148396 0.7810871 156.6523 162.5153 66.43956 2.593412
2   m2 0.8347678 0.8170643 0.7819955 157.1426 164.4713 67.43259 2.577785
3   m3 0.8351443 0.8107212 0.7710297 159.0696 167.8640 70.04081 2.622095
     sigma
1 2.593412
2 2.577785
3 2.622095</code></pre>
</div>
</div>
</section>
<section id="weighted-linear-regression-in-ols_regress" class="level3">
<h3 class="anchored" data-anchor-id="weighted-linear-regression-in-ols_regress">Weighted Linear Regression in <code>ols_regress()</code></h3>
<p>In 0.7.1, <code>ols_regress()</code> supports weighted linear models fitted with <code>lm(..., weights = ...)</code>, matching base R’s <code>summary()</code>, <code>anova()</code>, and <code>confint()</code> for coefficient estimates, standard errors, <img src="https://latex.codecogs.com/png.latex?t">-statistics, confidence intervals, <img src="https://latex.codecogs.com/png.latex?R%5E2">, residual standard error, the ANOVA table, standardized betas, and weighted predicted <img src="https://latex.codecogs.com/png.latex?R%5E2">:</p>
<div class="cell">
<details open="" class="code-fold">
<summary>Code</summary>
<div class="code-copy-outer-scaffold"><div class="sourceCode cell-code" id="cb14" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb14-1">d_wt <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> mtcars</span>
<span id="cb14-2">d_wt<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">$</span>w <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">seq_len</span>(<span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">nrow</span>(d_wt)) <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">/</span> <span class="dv" style="color: #AD0000;
background-color: null;
font-style: inherit;">8</span></span>
<span id="cb14-3"></span>
<span id="cb14-4">m_w1 <span class="ot" style="color: #003B4F;
background-color: null;
font-style: inherit;">&lt;-</span> <span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">lm</span>(mpg <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">~</span> wt <span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">+</span> hp, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">data =</span> d_wt, <span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">weights =</span> w)</span>
<span id="cb14-5"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">ols_regress</span>(m_w1)</span></code></pre></div></div>
</details>
<div class="cell-output cell-output-stdout">
<pre><code>                         Model Summary                          
---------------------------------------------------------------
R                       0.917       RMSE                 3.876 
R-Squared               0.841       MSE                 15.027 
Adj. R-Squared          0.830       Coef. Var           19.295 
Pred R-Squared          0.786       AIC                167.360 
MAE                     1.953       SBC                173.223 
---------------------------------------------------------------
 RMSE: Root Mean Square Error 
 MSE: Mean Square Error 
 MAE: Mean Absolute Error 
 AIC: Akaike Information Criteria 
 SBC: Schwarz Bayesian Criteria 

                               ANOVA                                 
--------------------------------------------------------------------
                Sum of                                              
               Squares        DF    Mean Square      F         Sig. 
--------------------------------------------------------------------
Regression    2308.267         2       1154.134    76.806    0.0000 
Residual       435.769        29         15.027                     
Total         2744.036        31                                    
--------------------------------------------------------------------

                                  Parameter Estimates                                    
----------------------------------------------------------------------------------------
      model      Beta    Std. Error    Std. Beta      t        Sig      lower     upper 
----------------------------------------------------------------------------------------
(Intercept)    37.938         1.545                 24.560    0.000    34.779    41.098 
         wt    -4.102         0.613       -0.666    -6.691    0.000    -5.355    -2.848 
         hp    -0.030         0.008       -0.345    -3.678    0.001    -0.047    -0.013 
----------------------------------------------------------------------------------------</code></pre>
</div>
</div>
</section>
</section>
<section id="other-improvements" class="level2">
<h2 class="anchored" data-anchor-id="other-improvements">Other improvements</h2>
<ul>
<li><code>ols_press()</code> weights the squared deleted residuals for weighted models, so <code>ols_pred_rsq()</code> no longer moves towards 1 when the weights are rescaled. For the weighted model shown above, multiplying the weights by 10 changed predicted <img src="https://latex.codecogs.com/png.latex?R%5E2"> from <code>0.9077</code> to <code>0.9908</code> in 0.7.0; in 0.7.1 both are <code>0.7859</code>.</li>
<li><code>ols_plot_resid_hist()</code> draws the bars with the same breaks as the normal curve. Previously the curve could be drawn at about half its correct height (<code>0.54</code> of the expected peak for <code>lm(hwy ~ displ + cty + year, data = ggplot2::mpg)</code>).</li>
<li><code>ols_plot_comp_plus_resid()</code> keeps every predictor for single-predictor and no-intercept models; 0.7.0 dropped a column and plotted a single point.</li>
<li><code>ols_test_outlier()</code> caps Bonferroni-adjusted <img src="https://latex.codecogs.com/png.latex?p">-values at 1 (0.7.0 could report values such as <code>6.62</code>).</li>
<li>Information criteria count parameters with the model rank, which fixes factor predictors and rank-deficient models; <code>ols_aic(corrected = TRUE)</code> matches the AICc formula (<a href="https://github.com/rsquaredacademy/olsrr/issues/224">#224</a>, <a href="https://github.com/rsquaredacademy/olsrr/issues/170">#170</a>, reported by <a href="https://github.com/comatrion">@comatrion</a>), and Mallows’ <img src="https://latex.codecogs.com/png.latex?C_p"> is correct for factor predictors (<a href="https://github.com/rsquaredacademy/olsrr/issues/92">#92</a>, reported by <a href="https://github.com/cccfran">@cccfran</a>) and in <code>ols_step_all_possible()</code> (<a href="https://github.com/rsquaredacademy/olsrr/issues/215">#215</a>, reported by <a href="https://github.com/LSantanaZA">@LSantanaZA</a>).</li>
<li>Stepwise selection handles transformed and factor predictors in refits (<a href="https://github.com/rsquaredacademy/olsrr/issues/244">#244</a>, <a href="https://github.com/rsquaredacademy/olsrr/issues/246">#246</a>), forward selection on saturated models (<a href="https://github.com/rsquaredacademy/olsrr/issues/199">#199</a>, reported by <a href="https://github.com/hirscht">@hirscht</a>), ties (<a href="https://github.com/rsquaredacademy/olsrr/issues/206">#206</a>, reported by <a href="https://github.com/kaygmz">@kaygmz</a>) and aliased terms (<a href="https://github.com/rsquaredacademy/olsrr/issues/172">#172</a>, reported by <a href="https://github.com/yayyb">@yayyb</a>).</li>
<li>Models fitted with <code>na.action = na.exclude</code> or with weights are handled consistently across plots and tests; diagnostics whose formulas assume unweighted least squares now stop with a clear error for weighted fits (<a href="https://github.com/rsquaredacademy/olsrr/issues/236">#236</a>).</li>
</ul>
</section>
<section id="whats-next" class="level2">
<h2 class="anchored" data-anchor-id="whats-next">What’s next</h2>
<p>Two issues remain open and will be fixed in 0.7.2. The Kolmogorov-Smirnov row of <code>ols_test_normality()</code> is not Lilliefors-corrected, so its <img src="https://latex.codecogs.com/png.latex?p">-value is too large when the mean and standard deviation are estimated from the residuals (<a href="https://github.com/rsquaredacademy/olsrr/issues/254">#254</a>). Until then, use the Shapiro-Wilk, Cramér–von Mises or Anderson-Darling rows, or call <code>nortest::lillie.test(residuals(model))</code>. Two panels of <code>ols_plot_diagnostics()</code> also share the title “Residual Fit Spread Plot” (<a href="https://github.com/rsquaredacademy/olsrr/issues/253">#253</a>); the plotted values and axis labels are correct.</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>A big thank you to everyone who contributed to <code>olsrr</code> since 2019 by opening issues, reporting bugs and sending pull requests: <a href="https://github.com/AminHP">@AminHP</a>, <a href="https://github.com/aravindhebbali">@aravindhebbali</a>, <a href="https://github.com/ashimkapoor">@ashimkapoor</a>, <a href="https://github.com/bappa10085">@bappa10085</a>, <a href="https://github.com/ButlerMSDA">@ButlerMSDA</a>, <a href="https://github.com/CarlosJara72">@CarlosJara72</a>, <a href="https://github.com/comatrion">@comatrion</a>, <a href="https://github.com/davidreilly007">@davidreilly007</a>, <a href="https://github.com/DrMondesire">@DrMondesire</a>, <a href="https://github.com/grantbrown">@grantbrown</a>, <a href="https://github.com/hirscht">@hirscht</a>, <a href="https://github.com/hkincaid">@hkincaid</a>, <a href="https://github.com/iankelk">@iankelk</a>, <a href="https://github.com/jmberros">@jmberros</a>, <a href="https://github.com/jorgevallejo">@jorgevallejo</a>, <a href="https://github.com/jst04004">@jst04004</a>, <a href="https://github.com/jtheog">@jtheog</a>, <a href="https://github.com/kaygmz">@kaygmz</a>, <a href="https://github.com/linusjf">@linusjf</a>, <a href="https://github.com/LSantanaZA">@LSantanaZA</a>, <a href="https://github.com/modche">@modche</a>, <a href="https://github.com/Molina-Valero">@Molina-Valero</a>, <a href="https://github.com/mwm-cbus">@mwm-cbus</a>, <a href="https://github.com/njrockwood">@njrockwood</a>, <a href="https://github.com/paleolimbot">@paleolimbot</a>, <a href="https://github.com/PsychlytxTD">@PsychlytxTD</a>, <a href="https://github.com/qwertytam">@qwertytam</a>, <a href="https://github.com/sblanco74">@sblanco74</a>, <a href="https://github.com/sckott">@sckott</a>, <a href="https://github.com/SunsetBlvd">@SunsetBlvd</a>, <a href="https://github.com/tikuma-lsuhsc">@tikuma-lsuhsc</a>, <a href="https://github.com/vasili111">@vasili111</a>, <a href="https://github.com/viqule11">@viqule11</a>, <a href="https://github.com/williamlai2">@williamlai2</a>, and <a href="https://github.com/yayyb">@yayyb</a>.</p>


</section>

 ]]></description>
  <category>R</category>
  <category>packages</category>
  <category>regression</category>
  <guid>https://blog.rsquaredacademy.com/posts/olsrr-0-7-1/</guid>
  <pubDate>Thu, 19 Nov 2026 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/olsrr-0-7-1.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Introducing standby</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/introducing-standby-alerts-notifications-tooltips-loading-screens-shiny/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2023-05-30-standby.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<p>
<img src="https://blog.rsquaredacademy.com/img/github-standby.png" width="70%" style="display: block; margin: auto;">
</p>
<p>
We are excited to announce the <a href="https://standby.rsquaredacademy.com">standby</a> package. It allows you to easily create alerts, notifications, tooltips and loading screens in Shiny. The package was developed as part of our internal tools for developing shiny apps and we are glad to share it with the rest of the community.
</p>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">
Installation
</h2>
<pre class="r"><code># Install release version from CRAN
install.packages("standby")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/standby")</code></pre>
</section>
<section id="loading-screens" class="level2">
<h2 class="anchored" data-anchor-id="loading-screens">
Loading Screens
</h2>
<p>
To use spinners/loaders from <strong>standby</strong> in your Shiny application, include the following in the <strong>UI</strong> part of the app:
</p>
<ul>
<li>
Include the dependencies using the appropriate <code>use*</code> functions (<code>useSpinkit()</code> in the below example).
</li>
<li>
Wrap the target output using corresponding rendering function (<code>spinkit()</code> in the below example).
</li>
</ul>
<section id="example" class="level3">
<h3 class="anchored" data-anchor-id="example">
Example
</h3>
<pre class="r"><code>library(shiny)
library(standby)

ui &lt;- fluidPage(

  standby::useSpinkit(), # include dependencies
  fluidRow(
    standby::spinkit(plotOutput("plot1")), # wrap output inside loader
    actionButton("render", "Render")
  )

)

server &lt;- function(input, output, session) {

    output$plot1 &lt;- renderPlot({
      input$render
      Sys.sleep(3)
      hist(mtcars$mpg)
    })

}

shinyApp(ui, server)</code></pre>
</section>
<section id="details" class="level3">
<h3 class="anchored" data-anchor-id="details">
Details
</h3>
<p>
The below table displays the dependency and rendering functions along with references:
</p>
<table class="table">
<colgroup>
<col width="6%">
<col width="15%">
<col width="13%">
<col width="64%">
</colgroup>
<thead>
<tr class="header">
<th align="center">
Index
</th>
<th align="center">
Dependency
</th>
<th align="center">
Render
</th>
<th align="center">
Reference
</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="center">
1
</td>
<td align="center">
<code>useThreeDots()</code>
</td>
<td align="center">
<code>threeDots()</code>
</td>
<td align="center">
<a href="https://github.com/nzbin/three-dots">Three Dots</a>
</td>
</tr>
<tr class="even">
<td align="center">
2
</td>
<td align="center">
<code>useSpinkit()</code>
</td>
<td align="center">
<code>spinkit()</code>
</td>
<td align="center">
<a href="https://github.com/tobiasahlin/SpinKit">SpinKit</a>
</td>
</tr>
<tr class="odd">
<td align="center">
3
</td>
<td align="center">
<code>useVizLoad()</code>
</td>
<td align="center">
<code>vizLoad()</code>
</td>
<td align="center">
<a href="https://github.com/RIDICS/Loading-Visualization">Loading Visualization</a>
</td>
</tr>
<tr class="even">
<td align="center">
4
</td>
<td align="center">
<code>useSpinners()</code>
</td>
<td align="center">
<code>spinners()</code>
</td>
<td align="center">
<a href="https://github.com/lukehaas/css-loaders">Spinners</a>
</td>
</tr>
<tr class="odd">
<td align="center">
5
</td>
<td align="center">
<code>useLoaders()</code>
</td>
<td align="center">
<code>loaders()</code>
</td>
<td align="center">
<a href="https://github.com/raphaelfabeni/css-loader">Loaders</a>
</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="alerts-and-notifications" class="level2">
<h2 class="anchored" data-anchor-id="alerts-and-notifications">
Alerts and Notifications
</h2>
<p>
To use alerts or notifications from <strong>buzz</strong> in your Shiny application, follow the below steps:
</p>
<ul>
<li>
Include the dependencies in the <strong>UI</strong> part of the app using the appropriate <code>use*</code> functions (<code>useToast()</code> in the below example).
</li>
<li>
Include the corresponding rendering function in the <strong>Server</strong> part of the app (<code>toast()</code> in the below example).
</li>
</ul>
<section id="example-1" class="level3">
<h3 class="anchored" data-anchor-id="example-1">
Example
</h3>
<pre class="r"><code>library(shiny)
library(standby)
ui &lt;- fluidPage(
  useBootBox(), # include dependencies
  actionButton(inputId = "notify",
               label   = "Show Notification")
)
server &lt;- function(input, output, session) {
  observeEvent(input$pnotify, {
    bootBox(class = "rubberBand") # display the alert
  })
}
shinyApp(ui, server)</code></pre>
</section>
<section id="details-1" class="level3">
<h3 class="anchored" data-anchor-id="details-1">
Details
</h3>
<p>
The below table displays the dependency and rendering functions along with references:
</p>
<table class="table">
<colgroup>
<col width="5%">
<col width="13%">
<col width="17%">
<col width="63%">
</colgroup>
<thead>
<tr class="header">
<th align="center">
Index
</th>
<th align="center">
Dependency
</th>
<th align="center">
Render
</th>
<th align="center">
Reference
</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="center">
1
</td>
<td align="center">
<code>useAlertify()</code>
</td>
<td align="center">
<code>alertify_alert()</code>
</td>
<td align="center">
<a href="https://github.com/MohammadYounes/AlertifyJS">Alertify</a>
</td>
</tr>
<tr class="even">
<td align="center">
2
</td>
<td align="center">
<code>useAlertify()</code>
</td>
<td align="center">
<code>alertify_notify()</code>
</td>
<td align="center">
<a href="https://github.com/MohammadYounes/AlertifyJS">Alertify</a>
</td>
</tr>
<tr class="odd">
<td align="center">
3
</td>
<td align="center">
<code>useBootBox()</code>
</td>
<td align="center">
<code>bootBox()</code>
</td>
<td align="center">
<a href="https://github.com/bootboxjs/bootbox">BootBox</a>
</td>
</tr>
<tr class="even">
<td align="center">
4
</td>
<td align="center">
<code>useMicroTip()</code>
</td>
<td align="center">
<code>microTip()</code>
</td>
<td align="center">
<a href="https://github.com/ghosh/microtip">MicroTip</a>
</td>
</tr>
<tr class="odd">
<td align="center">
5
</td>
<td align="center">
<code>useNS()</code>
</td>
<td align="center">
<code>notice()</code>
</td>
<td align="center">
<a href="https://tympanus.net/Development/NotificationStyles/">Notification Styles</a>
</td>
</tr>
<tr class="even">
<td align="center">
6
</td>
<td align="center">
<code>useNotify()</code>
</td>
<td align="center">
<code>notify()</code>
</td>
<td align="center">
<a href="https://github.com/sciactive/pnotify">PNotify</a>
</td>
</tr>
<tr class="odd">
<td align="center">
7
</td>
<td align="center">
<code>useTingle()</code>
</td>
<td align="center">
<code>tingle()</code>
</td>
<td align="center">
<a href="https://github.com/robinparisi/tingle">Tingle</a>
</td>
</tr>
<tr class="even">
<td align="center">
8
</td>
<td align="center">
<code>useToast()</code>
</td>
<td align="center">
<code>toast()</code>
</td>
<td align="center">
<a href="https://github.com/marcelodolza/iziToast">iziToast</a>
</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<ul>
<li>
<a href="https://standby.rsquaredacademy.com">Documentation</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy/standby">GitHub</a>
</li>
</ul>
</section>
<section id="inspiration-acknowledgement" class="level2">
<h2 class="anchored" data-anchor-id="inspiration-acknowledgement">
Inspiration &amp; Acknowledgement
</h2>
<p>
<a href="https://standby.rsquaredacademy.com">standby</a> takes inspiration from the following wonderful packages:
</p>
<ul>
<li>
<a href="https://github.com/daattali/shinycssloaders">shinycssloaders</a> by <a href="https://github.com/daattali">Dean Attali</a>
</li>
<li>
<a href="https://github.com/JohnCoene/waiter">waiter</a> by <a href="https://john-coene.com/">John Coene</a>
</li>
</ul>
<p>
We found the following books extremely helpful while working on the package and in general for developing Shiny apps. We would like to express out heartful gratitude to the respective authors for making them available for free to the community.
</p>
<ul>
<li>
<a href="https://unleash-shiny.rinterface.com/index.html">Outstanding User Interfaces with Shiny</a> by<a href="https://github.com/DivadNojnarg">David Granjon</a>
</li>
<li>
<a href="https://book.javascript-for-r.com/">JavaScript for R</a> by <a href="https://john-coene.com/">John Coene</a>
</li>
</ul>
</section>
<section id="getting-help" class="level2">
<h2 class="anchored" data-anchor-id="getting-help">
Getting Help
</h2>
<p>
If you encounter a bug, please file a minimal reproducible example using <a href="https://reprex.tidyverse.org/index.html">reprex</a> on github. For questions and clarifications, use <a href="https://stackoverflow.com/">StackOverflow</a>.
</p>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
All feedback is welcome. Issues (bugs and feature requests) can be posted to <a href="https://github.com/rsquaredacademy/standby/issues">github tracker</a>. For help with code or other related questions, feel free to reach out to us at <a href="mailto:pkgs@rsquaredacademy.com" class="email">pkgs@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>packages</category>
  <guid>https://blog.rsquaredacademy.com/posts/introducing-standby-alerts-notifications-tooltips-loading-screens-shiny/</guid>
  <pubDate>Tue, 30 May 2023 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/github-standby.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Introducing yahoofinancer</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/introducing-yahoofinancer-fetch-data-from-yahoo-finance-api/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2022-11-04-yahoofinancer.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<p>
<img src="https://blog.rsquaredacademy.com/img/github-yfr.png" width="70%" style="display: block; margin: auto;">
</p>
<p>
We are excited to announce the <a href="https://yahoofinancer.rsquaredacademy.com">yahoofinancer</a> package. The yahoofinancer R package allows the retrieval of nearly all data visible via the Yahoo Finance front-end and aims to reduce the pre-processing steps needed in analyzing such data.
</p>
<p>
yahoofinancer is inspired by and a port of the Python package <a href="https://yahooquery.dpguthrie.com/">yahooquery</a>. yahoofinancer is <strong>not</strong> affiliated, endorsed, or vetted by Yahoo, Inc.&nbsp;It’s an open-source tool that uses Yahoo’s publicly available APIs, and is intended for research and educational purposes.
</p>
<p>
With yahoofinancer, you can fetch the following data related to:
</p>
<ul>
<li>
company
<ul>
<li>
quote
</li>
<li>
summary
</li>
<li>
statistics
</li>
<li>
historical data
</li>
<li>
profile
</li>
<li>
financials
</li>
<li>
analysis
</li>
<li>
options
</li>
</ul>
</li>
<li>
index
<ul>
<li>
summary
</li>
<li>
historical data
</li>
</ul>
</li>
<li>
currency
</li>
<li>
market summary
</li>
</ul>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">
Installation
</h2>
<pre class="r"><code># Install release version from CRAN
install.packages("yahoofinancer")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/yahoofinancer")</code></pre>
</section>
<section id="usage" class="level2">
<h2 class="anchored" data-anchor-id="usage">
Usage
</h2>
</section>
<section id="ticker" class="level2">
<h2 class="anchored" data-anchor-id="ticker">
Ticker
</h2>
<p>
The <code>Ticker</code> class is the main workhorse of the package and through it we can obtain most of the data related to a company.
</p>
<section id="instantiate-ticker-class" class="level4">
<h4 class="anchored" data-anchor-id="instantiate-ticker-class">
Instantiate Ticker Class
</h4>
<p>
We can instantiate the <code>Ticker</code> class by passing the company’s ticker symbol. For instance, to get data for <a href="https://finance.yahoo.com/quote/AAPL/">Apple Inc.</a>, pass <code>aapl</code> as the first argument to the <code>Ticker</code> class.
</p>
<pre class="r"><code>aapl &lt;- Ticker$new('aapl')</code></pre>
<p>
We can use the <code>validate()</code> function to validate a symbol.
</p>
</section>
<section id="regular-market-price" class="level4">
<h4 class="anchored" data-anchor-id="regular-market-price">
Regular Market Price
</h4>
<p>
Let us begin by getting the real time price of <a href="https://finance.yahoo.com/quote/AAPL/">Apple Inc.</a>.
</p>
<pre class="r"><code>aapl$quote$regularMarketPrice</code></pre>
<pre><code>## [1] 138.88</code></pre>
</section>
<section id="summary" class="level4">
<h4 class="anchored" data-anchor-id="summary">
Summary
</h4>
<p>
To retrieve data from the <a href="https://finance.yahoo.com/quote/AAPL/">Summary</a> tab, use the <code>summary_detail</code> property on the <code>Ticker</code> class.
</p>
<pre class="r"><code>aapl$summary_detail</code></pre>
<pre><code>## $maxAge
## [1] 1
## 
## $priceHint
## [1] 2
## 
## $previousClose
## [1] 145.03
## 
## $open
## [1] 142.06
## 
## $dayLow
## [1] 138.76
## 
## $dayHigh
## [1] 142.795
## 
## $regularMarketPreviousClose
## [1] 145.03
## 
## $regularMarketOpen
## [1] 142.06
## 
## $regularMarketDayLow
## [1] 138.76
## 
## $regularMarketDayHigh
## [1] 142.795
## 
## $dividendRate
## [1] 0.92
## 
## $dividendYield
## [1] 0.0063
## 
## $exDividendDate
## [1] 1667520000
## 
## $payoutRatio
## [1] 0.1473
## 
## $fiveYearAvgDividendYield
## [1] 1
## 
## $beta
## [1] 1.246644
## 
## $trailingPE
## [1] 23.74017
## 
## $forwardPE
## [1] 20.36364
## 
## $volume
## [1] 97918516
## 
## $regularMarketVolume
## [1] 97918516
## 
## $averageVolume
## [1] 85440921
## 
## $averageVolume10days
## [1] 96881500
## 
## $averageDailyVolume10Day
## [1] 96881500
## 
## $bid
## [1] 0
## 
## $ask
## [1] 138.63
## 
## $bidSize
## [1] 800
## 
## $askSize
## [1] 1300
## 
## $marketCap
## [1] 2.307158e+12
## 
## $fiftyTwoWeekLow
## [1] 129.04
## 
## $fiftyTwoWeekHigh
## [1] 182.94
## 
## $priceToSalesTrailing12Months
## [1] 5.85086
## 
## $fiftyDayAverage
## [1] 150.0812
## 
## $twoHundredDayAverage
## [1] 156.2758
## 
## $trailingAnnualDividendRate
## [1] 0.9
## 
## $trailingAnnualDividendYield
## [1] 0.006205612
## 
## $currency
## [1] "USD"
## 
## $tradeable
## [1] FALSE</code></pre>
</section>
<section id="statistics" class="level4">
<h4 class="anchored" data-anchor-id="statistics">
Statistics
</h4>
<p>
If you are looking to retrieve data from the <a href="https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL">Statistics</a> tab, use the following properties:
</p>
<section id="key-statistics" class="level5">
<h5 class="anchored" data-anchor-id="key-statistics">
Key Statistics
</h5>
<p>
The <code>key_stats</code> property returns KPIs for a given symbol. It is available raw
</p>
<pre class="r"><code>head(map(aapl$key_stats, 'raw'))</code></pre>
<pre><code>## $maxAge
## NULL
## 
## $priceHint
## [1] 2
## 
## $enterpriseValue
## [1] 2.293495e+12
## 
## $forwardPE
## [1] 20.36364
## 
## $profitMargins
## [1] 0.2531
## 
## $floatShares
## [1] 15891255395</code></pre>
<p>
or formatted
</p>
<pre class="r"><code>head(map(aapl$key_stats, 'fmt'))</code></pre>
<pre><code>## $maxAge
## NULL
## 
## $priceHint
## [1] "2"
## 
## $enterpriseValue
## [1] "2.29T"
## 
## $forwardPE
## [1] "20.36"
## 
## $profitMargins
## [1] "25.31%"
## 
## $floatShares
## [1] "15.89B"</code></pre>
<p>
The <code>valuation_measures</code> property retrieves valuation measures for most recent four quarters.
</p>
<pre class="r"><code>aapl$valuation_measures</code></pre>
<pre><code>##         date enterprise_value enterprise_value_ebitda_ratio
## 1 2021-09-30     2.384485e+12                  2.384485e+12
## 2 2021-12-31     2.963725e+12                  2.963725e+12
## 3 2022-03-31     2.888888e+12                  2.888888e+12
## 4 2022-06-30     2.269030e+12                  2.269030e+12
## 5 2022-09-30     2.274841e+12                  2.274841e+12
##   enterprise_value_revenue_ratio forward_pe_ratio   market_cap     pb_ratio
## 1                   2.384485e+12     2.384485e+12 2.384485e+12 2.384485e+12
## 2                   2.963725e+12     2.963725e+12 2.963725e+12 2.963725e+12
## 3                   2.888888e+12     2.888888e+12 2.888888e+12 2.888888e+12
## 4                   2.269030e+12     2.269030e+12 2.269030e+12 2.269030e+12
## 5                   2.274841e+12     2.274841e+12 2.274841e+12 2.274841e+12
##       pe_ratio    peg_ratio     ps_ratio
## 1 2.384485e+12 2.384485e+12 2.384485e+12
## 2 2.963725e+12 2.963725e+12 2.963725e+12
## 3 2.888888e+12 2.888888e+12 2.888888e+12
## 4 2.269030e+12 2.269030e+12 2.269030e+12
## 5 2.274841e+12 2.274841e+12 2.274841e+12</code></pre>
</section>
</section>
<section id="historical-data" class="level4">
<h4 class="anchored" data-anchor-id="historical-data">
Historical Data
</h4>
<p>
Use the <code>get_history()</code> method to retrieve historical pricing data for a given symbol. By default, it returns YTD data.
</p>
<pre class="r"><code>head(aapl$get_history())</code></pre>
<pre><code>##                  date    volume   high    low   open  close adj_close
## 1 2022-01-03 14:30:00 104487900 182.88 177.71 177.83 182.01  181.2599
## 2 2022-01-04 14:30:00  99310400 182.94 179.12 182.63 179.70  178.9594
## 3 2022-01-05 14:30:00  94537600 180.17 174.64 179.61 174.92  174.1992
## 4 2022-01-06 14:30:00  96904000 175.30 171.64 172.70 172.00  171.2912
## 5 2022-01-07 14:30:00  86709100 174.14 171.03 172.89 172.17  171.4605
## 6 2022-01-10 14:30:00 106765600 172.50 168.17 169.08 172.19  171.4804</code></pre>
<p>
To get data from a specific date upto the current date, use the <code>start</code> argument to supply the initial date. The time between data points can be specified using the <code>interval</code> argument.
</p>
<pre class="r"><code>aapl$get_history(start = '2022-10-20', interval = '1d')</code></pre>
<pre><code>##                   date    volume   high    low   open  close adj_close
## 1  2022-10-20 13:30:00  64522000 145.89 142.65 143.02 143.39    143.39
## 2  2022-10-21 13:30:00  86464700 147.85 142.65 142.87 147.27    147.27
## 3  2022-10-24 13:30:00  75981900 150.23 146.00 147.19 149.45    149.45
## 4  2022-10-25 13:30:00  74732300 152.49 149.36 150.09 152.34    152.34
## 5  2022-10-26 13:30:00  88194300 151.99 148.04 150.96 149.35    149.35
## 6  2022-10-27 13:30:00 109180200 149.05 144.13 148.07 144.80    144.80
## 7  2022-10-28 13:30:00 164762400 157.50 147.82 148.20 155.74    155.74
## 8  2022-10-31 13:30:00  97943200 154.24 151.92 153.16 153.34    153.34
## 9  2022-11-01 13:30:00  80379300 155.45 149.13 155.08 150.65    150.65
## 10 2022-11-02 13:30:00  93604600 152.17 145.00 148.95 145.03    145.03
## 11 2022-11-03 13:30:00  97572100 142.80 138.75 142.06 138.88    138.88</code></pre>
<p>
To retrieve data between two specific dates, use both the <code>start</code> and <code>end</code> argument. The dates should be either <code>String</code> or <code>Date</code> object in <code>yyyy-mm-dd</code> format.
</p>
<pre class="r"><code>aapl$get_history(start = '2022-10-01', end = '2022-10-14', interval = '1d')</code></pre>
<pre><code>##                  date    volume   high    low   open  close adj_close
## 1 2022-10-03 13:30:00 114311700 143.07 137.69 138.21 142.45    142.45
## 2 2022-10-04 13:30:00  87830100 146.22 144.26 145.03 146.10    146.10
## 3 2022-10-05 13:30:00  79471000 147.38 143.01 144.07 146.40    146.40
## 4 2022-10-06 13:30:00  68402200 147.54 145.22 145.81 145.43    145.43
## 5 2022-10-07 13:30:00  85859100 143.10 139.45 142.54 140.09    140.09
## 6 2022-10-10 13:30:00  74899000 141.89 138.57 140.42 140.42    140.42
## 7 2022-10-11 13:30:00  77033700 141.35 138.22 139.90 138.98    138.98
## 8 2022-10-12 13:30:00  70433700 140.36 138.16 139.13 138.34    138.34
## 9 2022-10-13 13:30:00 113224000 143.59 134.37 134.99 142.99    142.99</code></pre>
<p>
The <code>period</code> arguments allows the user to retrieve data for specific length of time.
</p>
<pre class="r"><code>aapl$get_history(period = '1mo', interval = '1d')</code></pre>
<pre><code>##                   date    volume   high    low   open  close adj_close
## 1  2022-10-04 13:30:00  87830100 146.22 144.26 145.03 146.10    146.10
## 2  2022-10-05 13:30:00  79471000 147.38 143.01 144.07 146.40    146.40
## 3  2022-10-06 13:30:00  68402200 147.54 145.22 145.81 145.43    145.43
## 4  2022-10-07 13:30:00  85859100 143.10 139.45 142.54 140.09    140.09
## 5  2022-10-10 13:30:00  74899000 141.89 138.57 140.42 140.42    140.42
## 6  2022-10-11 13:30:00  77033700 141.35 138.22 139.90 138.98    138.98
## 7  2022-10-12 13:30:00  70433700 140.36 138.16 139.13 138.34    138.34
## 8  2022-10-13 13:30:00 113224000 143.59 134.37 134.99 142.99    142.99
## 9  2022-10-14 13:30:00  88512300 144.52 138.19 144.31 138.38    138.38
## 10 2022-10-17 13:30:00  85250900 142.90 140.27 141.07 142.41    142.41
## 11 2022-10-18 13:30:00  99136600 146.70 140.61 145.49 143.75    143.75
## 12 2022-10-19 13:30:00  61758300 144.95 141.50 141.69 143.86    143.86
## 13 2022-10-20 13:30:00  64522000 145.89 142.65 143.02 143.39    143.39
## 14 2022-10-21 13:30:00  86464700 147.85 142.65 142.87 147.27    147.27
## 15 2022-10-24 13:30:00  75981900 150.23 146.00 147.19 149.45    149.45
## 16 2022-10-25 13:30:00  74732300 152.49 149.36 150.09 152.34    152.34
## 17 2022-10-26 13:30:00  88194300 151.99 148.04 150.96 149.35    149.35
## 18 2022-10-27 13:30:00 109180200 149.05 144.13 148.07 144.80    144.80
## 19 2022-10-28 13:30:00 164762400 157.50 147.82 148.20 155.74    155.74
## 20 2022-10-31 13:30:00  97943200 154.24 151.92 153.16 153.34    153.34
## 21 2022-11-01 13:30:00  80379300 155.45 149.13 155.08 150.65    150.65
## 22 2022-11-02 13:30:00  93604600 152.17 145.00 148.95 145.03    145.03
## 23 2022-11-03 13:30:00  97572100 142.80 138.75 142.06 138.88    138.88</code></pre>
<p>
Please view the <a href="https://yahoofinancer.rsquaredacademy.com/reference/ticker-class#arguments-5">documentation</a> to see the valid values for <code>interval</code> and <code>period</code>.
</p>
</section>
<section id="profile" class="level4">
<h4 class="anchored" data-anchor-id="profile">
Profile
</h4>
<p>
Retrieve data from the <a href="https://finance.yahoo.com/quote/AAPL/profile?p=AAPL">Profile</a> tab. The <code>summary_profile</code> property returns the business summary for a given symbol.
</p>
<pre class="r"><code>aapl$summary_profile</code></pre>
<pre><code>## $address1
## [1] "One Apple Park Way"
## 
## $city
## [1] "Cupertino"
## 
## $state
## [1] "CA"
## 
## $zip
## [1] "95014"
## 
## $country
## [1] "United States"
## 
## $phone
## [1] "408 996 1010"
## 
## $website
## [1] "https://www.apple.com"
## 
## $industry
## [1] "Consumer Electronics"
## 
## $sector
## [1] "Technology"
## 
## $longBusinessSummary
## [1] "Apple Inc. designs, manufactures, and markets smartphones, personal computers, tablets, wearables, and accessories worldwide. It also sells various related services. In addition, the company offers iPhone, a line of smartphones; Mac, a line of personal computers; iPad, a line of multi-purpose tablets; and wearables, home, and accessories comprising AirPods, Apple TV, Apple Watch, Beats products, and HomePod. Further, it provides AppleCare support and cloud services store services; and operates various platforms, including the App Store that allow customers to discover and download applications and digital content, such as books, music, video, games, and podcasts. Additionally, the company offers various services, such as Apple Arcade, a game subscription service; Apple Fitness+, a personalized fitness service; Apple Music, which offers users a curated listening experience with on-demand radio stations; Apple News+, a subscription news and magazine service; Apple TV+, which offers exclusive original content; Apple Card, a co-branded credit card; and Apple Pay, a cashless payment service, as well as licenses its intellectual property. The company serves consumers, and small and mid-sized businesses; and the education, enterprise, and government markets. It distributes third-party applications for its products through the App Store. The company also sells its products through its retail and online stores, and direct sales force; and third-party cellular network carriers, wholesalers, retailers, and resellers. Apple Inc. was incorporated in 1977 and is headquartered in Cupertino, California."
## 
## $fullTimeEmployees
## [1] 164000
## 
## $companyOfficers
## list()
## 
## $maxAge
## [1] 86400</code></pre>
<p>
The <code>company_officers</code> property retrieves top executives for given symbol and their total pay package.
</p>
<pre class="r"><code>aapl$company_officers</code></pre>
<pre><code>##                       name age                                         title
## 1      Mr. Timothy D. Cook  60                                CEO &amp; Director
## 2        Mr. Luca  Maestri  57                                  CFO &amp; Sr. VP
## 3  Mr. Jeffrey E. Williams  57                       Chief Operating Officer
## 4   Ms. Katherine L. Adams  57                   Sr. VP, Gen. Counsel &amp; Sec.
## 5     Ms. Deirdre  O'Brien  54                     Sr. VP of People &amp; Retail
## 6         Mr. Chris  Kondo  NA              Sr. Director of Corp. Accounting
## 7        Mr. James  Wilson  NA                      Chief Technology Officer
## 8          Ms. Mary  Demby  NA                     Chief Information Officer
## 9        Ms. Nancy  Paxton  NA Sr. Director of Investor Relations &amp; Treasury
## 10       Mr. Greg  Joswiak  NA                 Sr. VP of Worldwide Marketing
##    year_born fiscal_year total_pay exercised_value unexercised_value
## 1       1961        2021  16386559               0                 0
## 2       1964        2021   5018883               0                 0
## 3       1964        2021   5017437               0                 0
## 4       1964        2021   5014533               0                 0
## 5       1967        2021   5061191               0                 0
## 6         NA          NA        NA               0                 0
## 7         NA          NA        NA               0                 0
## 8         NA          NA        NA               0                 0
## 9         NA          NA        NA               0                 0
## 10        NA          NA        NA               0                 0</code></pre>
</section>
<section id="financials" class="level4">
<h4 class="anchored" data-anchor-id="financials">
Financials
</h4>
<p>
The <code>get_balance_sheet()</code> method retrieves balance sheet data for most recent four quarters or most recent four years.
</p>
<pre class="r"><code>aapl$get_balance_sheet('annual')</code></pre>
<pre><code>## # A tibble: 4 x 24
##   end_date          cash short~1 net_r~2 inven~3 other~4 total~5 long_~6 prope~7
##   &lt;date&gt;           &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;
## 1 2022-09-24 23646000000 2.47e10 6.09e10  4.95e9 2.12e10 1.35e11 1.21e11 8.42e10
## 2 2021-09-25 34940000000 2.77e10 5.15e10  6.58e9 1.41e10 1.35e11 1.28e11 4.95e10
## 3 2020-09-26 38016000000 5.29e10 3.74e10  4.06e9 1.13e10 1.44e11 1.01e11 4.53e10
## 4 2019-09-28 48844000000 5.17e10 4.58e10  4.11e9 1.24e10 1.63e11 1.05e11 3.74e10
## # ... with 15 more variables: other_assets &lt;dbl&gt;, total_assets &lt;dbl&gt;,
## #   accounts_payable &lt;dbl&gt;, short_long_term_debt &lt;dbl&gt;,
## #   other_current_liab &lt;dbl&gt;, long_term_debt &lt;dbl&gt;, other_liab &lt;dbl&gt;,
## #   total_current_liabilities &lt;dbl&gt;, total_liab &lt;dbl&gt;, common_stock &lt;dbl&gt;,
## #   retained_earnings &lt;dbl&gt;, treasury_stock &lt;dbl&gt;,
## #   other_stockholder_equity &lt;dbl&gt;, total_stockholder_equity &lt;dbl&gt;,
## #   net_tangible_assets &lt;dbl&gt;, and abbreviated variable names ...</code></pre>
<p>
The <code>get_income_statement()</code> method retrieves income statement data for most recent four quarters or most recent four years.
</p>
<pre class="r"><code>aapl$get_income_statement('annual')</code></pre>
<pre><code>## # A tibble: 4 x 16
##   end_date   total_rev~1 cost_~2 gross~3 resea~4 selli~5 total~6 opera~7 total~8
##   &lt;date&gt;           &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;int&gt;
## 1 2022-09-24     3.94e11 2.24e11 1.71e11 2.63e10 2.51e10 2.75e11 1.19e11 -3.34e8
## 2 2021-09-25     3.66e11 2.13e11 1.53e11 2.19e10 2.20e10 2.57e11 1.09e11  2.58e8
## 3 2020-09-26     2.75e11 1.70e11 1.05e11 1.88e10 1.99e10 2.08e11 6.63e10  8.03e8
## 4 2019-09-28     2.60e11 1.62e11 9.84e10 1.62e10 1.82e10 1.96e11 6.39e10  1.81e9
## # ... with 7 more variables: ebit &lt;dbl&gt;, interest_expense &lt;dbl&gt;,
## #   income_before_tax &lt;dbl&gt;, income_tax_expense &lt;dbl&gt;,
## #   net_income_from_continuing_ops &lt;dbl&gt;, net_income &lt;dbl&gt;,
## #   net_income_applicable_to_common_shares &lt;dbl&gt;, and abbreviated variable
## #   names 1: total_revenue, 2: cost_of_revenue, 3: gross_profit,
## #   4: research_development, 5: selling_general_administrative,
## #   6: total_operating_expenses, 7: operating_income, ...</code></pre>
<p>
The <code>get_cash_flow()</code> method retrieves cash flow data for most recent four quarters or most recent four years.
</p>
<pre class="r"><code>aapl$get_cash_flow('annual')</code></pre>
<pre><code>## # A tibble: 4 x 20
##   end_date   net_inc~1 depre~2 chang~3 change~4 change~5 chang~6 chang~7 total~8
##   &lt;date&gt;         &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;    &lt;dbl&gt;    &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;   &lt;dbl&gt;
## 1 2022-09-24   9.98e10 1.11e10 1.00e10 -1.82e 9  9.93e 9  1.48e9 -8.39e9 1.22e11
## 2 2021-09-25   9.47e10 1.13e10 2.98e 9 -1.01e10  1.40e10 -2.64e9 -6.15e9 1.04e11
## 3 2020-09-26   5.74e10 1.11e10 6.52e 9  6.92e 9 -1.98e 9 -1.27e8  8.81e8 8.07e10
## 4 2019-09-28   5.53e10 1.25e10 5.08e 9  2.45e 8 -2.55e 9 -2.89e8 -8.96e8 6.94e10
## # ... with 11 more variables: capital_expenditures &lt;dbl&gt;, investments &lt;dbl&gt;,
## #   other_cashflows_from_investing_activities &lt;int&gt;,
## #   total_cashflows_from_investing_activities &lt;dbl&gt;, dividends_paid &lt;dbl&gt;,
## #   net_borrowings &lt;dbl&gt;, other_cashflows_from_financing_activities &lt;int&gt;,
## #   total_cash_from_financing_activities &lt;dbl&gt;, change_in_cash &lt;dbl&gt;,
## #   repurchase_of_stock &lt;dbl&gt;, issuance_of_stock &lt;int&gt;, and abbreviated
## #   variable names 1: net_income, 2: depreciation, 3: change_to_netincome, ...</code></pre>
</section>
<section id="analysis" class="level4">
<h4 class="anchored" data-anchor-id="analysis">
Analysis
</h4>
<section id="earnings-estimate" class="level5">
<h5 class="anchored" data-anchor-id="earnings-estimate">
Earnings Estimate
</h5>
<pre class="r"><code>aapl$earnings_trend$earnings_estimate</code></pre>
<pre><code>##         date period analyst avg_estimate low_estimate high_estimate
## 1 2022-12-31     0q      27         2.08         1.93          2.20
## 2 2023-03-31    +1q      26         1.47         1.34          1.61
## 3 2023-09-30     0y      33         6.26         5.40          6.87
## 4 2024-09-30    +1y      28         6.82         6.01          7.36
## 5       &lt;NA&gt;    +5y      NA           NA           NA            NA
## 6       &lt;NA&gt;    -5y      NA           NA           NA            NA
##   year_ago_eps
## 1         2.10
## 2         1.52
## 3         6.11
## 4         6.26
## 5           NA
## 6           NA</code></pre>
</section>
<section id="revenue-estimate" class="level5">
<h5 class="anchored" data-anchor-id="revenue-estimate">
Revenue Estimate
</h5>
<pre class="r"><code>aapl$earnings_trend$revenue_estimate</code></pre>
<pre><code>##         date period analyst avg_estimate low_estimate high_estimate
## 1 2022-12-31     0q      23  1.27082e+11  1.21204e+11   1.32365e+11
## 2 2023-03-31    +1q      23  9.73035e+10  8.96080e+10   1.02950e+11
## 3 2023-09-30     0y      32  4.07153e+11  3.86805e+11   4.19743e+11
## 4 2024-09-30    +1y      26  4.28651e+11  4.05730e+11   4.48983e+11
## 5       &lt;NA&gt;    +5y      NA           NA           NA            NA
## 6       &lt;NA&gt;    -5y      NA           NA           NA            NA
##   year_ago_revenue
## 1               NA
## 2               NA
## 3      3.94328e+11
## 4      4.07153e+11
## 5               NA
## 6               NA</code></pre>
</section>
<section id="earnings-history" class="level5">
<h5 class="anchored" data-anchor-id="earnings-history">
Earnings History
</h5>
<pre class="r"><code>aapl$earnings_history</code></pre>
<pre><code>##      quarter period eps_estimate eps_actual eps_difference surprise_percent
## 1 2021-12-31    -4q         1.89       2.10           0.21            0.111
## 2 2022-03-31    -3q         1.43       1.52           0.09            0.063
## 3 2022-06-30    -2q         1.16       1.20           0.04            0.034
## 4 2022-09-30    -1q         1.27       1.29           0.02            0.016</code></pre>
</section>
<section id="eps-trend" class="level5">
<h5 class="anchored" data-anchor-id="eps-trend">
EPS Trend
</h5>
<pre class="r"><code>aapl$earnings_trend$eps_trend</code></pre>
<pre><code>##         date period current seven_days_ago thirty_days_ago sixty_days_ago
## 1 2022-12-31     0q    2.08           2.13            2.13           2.12
## 2 2023-03-31    +1q    1.47           1.52            1.53           1.52
## 3 2023-09-30     0y    6.26           6.43            6.46           6.45
## 4 2024-09-30    +1y    6.82           6.90            6.88           6.88
## 5       &lt;NA&gt;    +5y      NA             NA              NA             NA
## 6       &lt;NA&gt;    -5y      NA             NA              NA             NA
##   ninety_days_ago
## 1            2.11
## 2            1.52
## 3            6.43
## 4            6.79
## 5              NA
## 6              NA</code></pre>
</section>
<section id="eps-revision" class="level5">
<h5 class="anchored" data-anchor-id="eps-revision">
EPS Revision
</h5>
<pre class="r"><code>aapl$earnings_trend$eps_revision</code></pre>
<pre><code>##         date period up_last_7_days up_last_30_days down_last_30_days
## 1 2022-12-31     0q              4               6                15
## 2 2023-03-31    +1q              4               5                13
## 3 2023-09-30     0y              3               5                24
## 4 2024-09-30    +1y              2               4                 9
## 5       &lt;NA&gt;    +5y             NA              NA                NA
## 6       &lt;NA&gt;    -5y             NA              NA                NA
##   down_last_90_days
## 1                NA
## 2                NA
## 3                NA
## 4                NA
## 5                NA
## 6                NA</code></pre>
</section>
<section id="growth-estimates" class="level5">
<h5 class="anchored" data-anchor-id="growth-estimates">
Growth Estimates
</h5>
<pre class="r"><code>aapl$earnings_trend$growth</code></pre>
<pre><code>##         date period      growth
## 1 2022-12-31     0q -0.01000000
## 2 2023-03-31    +1q -0.03300000
## 3 2023-09-30     0y  0.02500000
## 4 2024-09-30    +1y  0.08899999
## 5       &lt;NA&gt;    +5y  0.08890000
## 6       &lt;NA&gt;    -5y  0.22590000</code></pre>
</section>
</section>
<section id="options" class="level4">
<h4 class="anchored" data-anchor-id="options">
Options
</h4>
<section id="option-chain" class="level5">
<h5 class="anchored" data-anchor-id="option-chain">
Option Chain
</h5>
<pre class="r"><code>head(aapl$option_chain)</code></pre>
<pre><code>## # A tibble: 6 x 16
##   expiration          option_type contra~1 strike curre~2 last_~3 change perce~4
##   &lt;dttm&gt;              &lt;chr&gt;       &lt;chr&gt;     &lt;dbl&gt; &lt;chr&gt;     &lt;dbl&gt;  &lt;dbl&gt;   &lt;dbl&gt;
## 1 2022-11-04 00:00:00 call        AAPL221~     70 USD        69.8      0       0
## 2 2022-11-04 00:00:00 call        AAPL221~     75 USD        64.6      0       0
## 3 2022-11-04 00:00:00 call        AAPL221~     80 USD        59.7      0       0
## 4 2022-11-04 00:00:00 call        AAPL221~     85 USD        59.6      0       0
## 5 2022-11-04 00:00:00 call        AAPL221~     90 USD        50.0      0       0
## 6 2022-11-04 00:00:00 call        AAPL221~     95 USD        44.9      0       0
## # ... with 8 more variables: open_interest &lt;int&gt;, bid &lt;dbl&gt;, ask &lt;dbl&gt;,
## #   contract_size &lt;chr&gt;, last_trade_date &lt;dttm&gt;, implied_volatility &lt;dbl&gt;,
## #   in_the_money &lt;lgl&gt;, volume &lt;int&gt;, and abbreviated variable names
## #   1: contract_symbol, 2: currency, 3: last_price, 4: percent_change</code></pre>
</section>
<section id="option-expiration-dates" class="level5">
<h5 class="anchored" data-anchor-id="option-expiration-dates">
Option Expiration Dates
</h5>
<pre class="r"><code>aapl$option_expiration_dates</code></pre>
<pre><code>##  [1] "2022-11-04" "2022-11-11" "2022-11-18" "2022-11-25" "2022-12-02"
##  [6] "2022-12-09" "2022-12-16" "2023-01-20" "2023-02-17" "2023-03-17"
## [11] "2023-04-21" "2023-05-19" "2023-06-16" "2023-07-21" "2023-09-15"
## [16] "2024-01-19" "2024-03-15" "2024-06-21" "2025-01-17"</code></pre>
</section>
<section id="option-strikes" class="level5">
<h5 class="anchored" data-anchor-id="option-strikes">
Option Strikes
</h5>
<pre class="r"><code>aapl$option_strikes</code></pre>
<pre><code>##  [1]  30.0  35.0  40.0  45.0  50.0  55.0  60.0  65.0  70.0  75.0  80.0  85.0
## [13]  90.0  95.0 100.0 105.0 110.0 115.0 120.0 125.0 126.0 127.0 128.0 129.0
## [25] 130.0 131.0 132.0 133.0 134.0 135.0 136.0 137.0 138.0 139.0 140.0 141.0
## [37] 142.0 143.0 144.0 145.0 146.0 147.0 148.0 149.0 150.0 152.5 155.0 157.5
## [49] 160.0 162.5 165.0 167.5 170.0 172.5 175.0 177.5 180.0 185.0 190.0 195.0
## [61] 200.0 205.0 210.0 215.0 220.0 225.0 230.0 235.0 240.0 245.0 250.0 255.0
## [73] 260.0 265.0 270.0 275.0 280.0 285.0 290.0 295.0 300.0 310.0 320.0</code></pre>
</section>
</section>
<section id="holding-pattern" class="level4">
<h4 class="anchored" data-anchor-id="holding-pattern">
Holding Pattern
</h4>
<p>
Data showing breakdown of owners of given symbol(s), insiders, institutions, etc.
</p>
<pre class="r"><code>aapl$major_holders</code></pre>
<pre><code>## $maxAge
## [1] 1
## 
## $insidersPercentHeld
## [1] 0.00072
## 
## $institutionsPercentHeld
## [1] 0.60186
## 
## $institutionsFloatPercentHeld
## [1] 0.60229
## 
## $institutionsCount
## [1] 5463</code></pre>
<p>
Top 10 owners of a given symbol.
</p>
<pre class="r"><code>aapl$institution_ownership</code></pre>
<pre><code>##          date                  organization percent_held   position
## 1  2022-06-30    Vanguard Group, Inc. (The)           NA 1277319054
## 2  2022-06-30                Blackrock Inc.           NA 1028688317
## 3  2022-06-30       Berkshire Hathaway, Inc           NA  894802319
## 4  2022-06-30      State Street Corporation           NA  598178524
## 5  2022-06-30                      FMR, LLC           NA  344317974
## 6  2022-06-30 Geode Capital Management, LLC           NA  278256192
## 7  2022-06-30 Price (T.Rowe) Associates Inc           NA  237910783
## 8  2022-06-30                Morgan Stanley           NA  182450565
## 9  2022-06-30    Northern Trust Corporation           NA  179828922
## 10 2022-06-30   Bank of America Corporation           NA  149133915
##           value percent_change
## 1  177394076456         0.0058
## 2  142864238487         0.0010
## 3  124270150431         0.0044
## 4   83075036333        -0.0255
## 5   47818881910        -0.0207
## 6   38644221303         0.0227
## 7   33041050704         0.0207
## 8   25338735358         0.4585
## 9   24974641565        -0.0333
## 10  20711718843         0.0328</code></pre>
</section>
<section id="others" class="level4">
<h4 class="anchored" data-anchor-id="others">
Others
</h4>
<p>
Data related to historical recommendations (buy, hold, sell) for a given symbol.
</p>
<pre class="r"><code>aapl$recommendation_trend</code></pre>
<pre><code>##   period strong_buy buy hold sell strong_sell
## 1     0m         11  21    6    0           0
## 2    -1m         13  25    6    1           0
## 3    -2m         14  25    7    1           0
## 4    -3m         13  20    8    0           0</code></pre>
<p>
Technical indicators for given symbol.
</p>
<pre class="r"><code>aapl$technical_insights$secReports[[1]]</code></pre>
<pre><code>## $id
## [1] "0000320193-22-000108_320193"
## 
## $type
## [1] "Periodic Financial Reports"
## 
## $title
## [1] "10-K : Periodic Financial Reports"
## 
## $description
## [1] "Annual report pursuant to Section 13 and 15(d)"
## 
## $filingDate
## [1] 1.666915e+12
## 
## $snapshotUrl
## [1] "https://cdn.yahoofinance.com/prod/sec-reports-thumbnails/0000320193/000032019322000108/thumbnail.png"
## 
## $formType
## [1] "10-K"</code></pre>
<p>
Symbols displayed in the <strong>People Also Watch</strong> sidebar.
</p>
<pre class="r"><code>aapl$recommendations</code></pre>
<pre><code>##   symbol    score
## 1   AMZN 0.323217
## 2   TSLA 0.303238
## 3   META 0.288718
## 4   GOOG 0.286876
## 5   NFLX 0.218178</code></pre>
</section>
</section>
<section id="indices" class="level2">
<h2 class="anchored" data-anchor-id="indices">
Indices
</h2>
<p>
Use the <code>Index</code> class for getting all data related to indices from Yahoo Finance API.
</p>
<section id="instantiate-index-class" class="level4">
<h4 class="anchored" data-anchor-id="instantiate-index-class">
Instantiate Index Class
</h4>
<p>
We can instantiate the <code>Index</code> class by passing the index symbol. For instance, to get data for <a href="https://finance.yahoo.com/quote/%5ENSEI?p=%5ENSEI">Nifty 50</a>, pass <code>^NSEI</code> as the first argument to the <code>Index</code> class.
</p>
<pre class="r"><code>nifty &lt;- Index$new('^NSEI')</code></pre>
</section>
<section id="summary-1" class="level4">
<h4 class="anchored" data-anchor-id="summary-1">
Summary
</h4>
<p>
Use the <code>summary_detail</code> property to retrieve data from the <a href="https://finance.yahoo.com/quote/%5ENSEI?p=%5ENSEI">Summary</a> tab.
</p>
<pre class="r"><code>nifty$summary_detail</code></pre>
<pre><code>## $language
## [1] "en-US"
## 
## $region
## [1] "US"
## 
## $quoteType
## [1] "INDEX"
## 
## $typeDisp
## [1] "Index"
## 
## $quoteSourceName
## [1] "Free Realtime Quote"
## 
## $triggerable
## [1] TRUE
## 
## $customPriceAlertConfidence
## [1] "HIGH"
## 
## $currency
## [1] "INR"
## 
## $tradeable
## [1] FALSE
## 
## $cryptoTradeable
## [1] FALSE
## 
## $messageBoardId
## [1] "finmb_INDEXNSEI"
## 
## $exchangeTimezoneName
## [1] "Asia/Kolkata"
## 
## $exchangeTimezoneShortName
## [1] "IST"
## 
## $gmtOffSetMilliseconds
## [1] 19800000
## 
## $market
## [1] "in_market"
## 
## $esgPopulated
## [1] FALSE
## 
## $regularMarketChangePercent
## [1] 0.1238129
## 
## $regularMarketPrice
## [1] 18075.05
## 
## $marketState
## [1] "REGULAR"
## 
## $exchange
## [1] "NSI"
## 
## $shortName
## [1] "NIFTY 50"
## 
## $firstTradeDateMilliseconds
## [1] 1.190001e+12
## 
## $priceHint
## [1] 2
## 
## $regularMarketChange
## [1] 22.35156
## 
## $regularMarketTime
## [1] 1667552435
## 
## $regularMarketDayHigh
## [1] 18108
## 
## $regularMarketDayRange
## [1] "18017.15 - 18108.0"
## 
## $regularMarketDayLow
## [1] 18017.15
## 
## $regularMarketVolume
## [1] 0
## 
## $regularMarketPreviousClose
## [1] 18052.7
## 
## $bid
## [1] 0
## 
## $ask
## [1] 0
## 
## $bidSize
## [1] 0
## 
## $askSize
## [1] 0
## 
## $fullExchangeName
## [1] "NSE"
## 
## $regularMarketOpen
## [1] 18053.4
## 
## $averageDailyVolume3Month
## [1] 275566
## 
## $averageDailyVolume10Day
## [1] 245930
## 
## $fiftyTwoWeekLowChange
## [1] 2891.65
## 
## $fiftyTwoWeekLowChangePercent
## [1] 0.1904481
## 
## $fiftyTwoWeekRange
## [1] "15183.4 - 18350.95"
## 
## $fiftyTwoWeekHighChange
## [1] -275.8984
## 
## $fiftyTwoWeekHighChangePercent
## [1] -0.01503456
## 
## $fiftyTwoWeekLow
## [1] 15183.4
## 
## $fiftyTwoWeekHigh
## [1] 18350.95
## 
## $fiftyDayAverage
## [1] 17526.7
## 
## $fiftyDayAverageChange
## [1] 548.3535
## 
## $fiftyDayAverageChangePercent
## [1] 0.03128676
## 
## $twoHundredDayAverage
## [1] 16994.76
## 
## $twoHundredDayAverageChange
## [1] 1080.293
## 
## $twoHundredDayAverageChangePercent
## [1] 0.06356624
## 
## $sourceInterval
## [1] 15
## 
## $exchangeDataDelayedBy
## [1] 15
## 
## $symbol
## [1] "^NSEI"</code></pre>
</section>
<section id="historical-data-1" class="level4">
<h4 class="anchored" data-anchor-id="historical-data-1">
Historical Data
</h4>
<p>
Use the <code>get_history()</code> method to retrieve historical data for a given index. By default, it returns YTD data.
</p>
<pre class="r"><code>nifty$get_history(start = '2022-10-20', interval = '1d')</code></pre>
<pre><code>##                   date volume     high      low     open    close adj_close
## 1  2022-10-20 03:45:00 249600 17584.15 17421.00 17423.10 17563.95  17563.95
## 2  2022-10-21 03:45:00 277700 17670.15 17520.75 17622.85 17576.30  17576.30
## 3  2022-10-24 03:45:00  45000 17777.55 17707.40 17736.35 17730.75  17730.75
## 4  2022-10-25 03:45:00 251400 17811.50 17637.00 17808.30 17656.35  17656.35
## 5  2022-10-27 03:45:00 324600 17783.90 17654.50 17771.40 17736.95  17736.95
## 6  2022-10-28 03:45:00 250000 17838.90 17723.70 17756.40 17786.80  17786.80
## 7  2022-10-31 03:45:00 227200 18022.80 17899.90 17910.20 18012.20  18012.20
## 8  2022-11-01 03:45:00 349900 18175.80 18060.15 18130.70 18145.40  18145.40
## 9  2022-11-02 03:45:00 270900 18178.75 18048.65 18177.90 18082.85  18082.85
## 10 2022-11-03 03:45:00 213000 18106.30 17959.20 17968.35 18052.70  18052.70
## 11 2022-11-04 09:00:35      0 18108.00 18017.15 18053.40 18075.05  18075.05</code></pre>
<p>
To get data from a specific date upto the current date, use the <code>start</code> argument to supply the initial date. The time between data points can be specified using the <code>interval</code> argument.
</p>
<p>
To retrieve data between two specific dates, use both the <code>start</code> and <code>end</code> argument. The dates should be either <code>String</code> or <code>Date</code> object in <code>yyyy-mm-dd</code> format.
</p>
<pre class="r"><code>nifty$get_history(start = '2022-10-01', end = '2022-10-14', interval = '1d')</code></pre>
<pre><code>##                  date volume     high      low     open    close adj_close
## 1 2022-10-03 03:45:00 278400 17114.65 16855.55 17102.10 16887.35  16887.35
## 2 2022-10-04 03:45:00 226000 17287.30 17117.30 17147.45 17274.30  17274.30
## 3 2022-10-06 03:45:00 265500 17428.80 17315.65 17379.25 17331.80  17331.80
## 4 2022-10-07 03:45:00 216300 17337.35 17216.95 17287.20 17314.65  17314.65
## 5 2022-10-10 03:45:00 234000 17280.15 17064.70 17094.35 17241.00  17241.00
## 6 2022-10-11 03:45:00 282600 17261.80 16950.30 17256.05 16983.55  16983.55
## 7 2022-10-12 03:45:00 256000 17142.35 16960.05 17025.55 17123.60  17123.60
## 8 2022-10-13 03:45:00 266400 17112.35 16956.95 17087.35 17014.35  17014.35</code></pre>
<p>
The <code>period</code> arguments allows the user to retrieve data for specific length of time.
</p>
<pre class="r"><code>nifty$get_history(period = '1mo', interval = '1d')</code></pre>
<pre><code>##                   date volume     high      low     open    close adj_close
## 1  2022-10-04 03:45:00 226000 17287.30 17117.30 17147.45 17274.30  17274.30
## 2  2022-10-06 03:45:00 265500 17428.80 17315.65 17379.25 17331.80  17331.80
## 3  2022-10-07 03:45:00 216300 17337.35 17216.95 17287.20 17314.65  17314.65
## 4  2022-10-10 03:45:00 234000 17280.15 17064.70 17094.35 17241.00  17241.00
## 5  2022-10-11 03:45:00 282600 17261.80 16950.30 17256.05 16983.55  16983.55
## 6  2022-10-12 03:45:00 256000 17142.35 16960.05 17025.55 17123.60  17123.60
## 7  2022-10-13 03:45:00 266400 17112.35 16956.95 17087.35 17014.35  17014.35
## 8  2022-10-14 03:45:00 227000 17348.55 17169.75 17322.30 17185.70  17185.70
## 9  2022-10-17 03:45:00 212200 17328.55 17098.55 17144.80 17311.80  17311.80
## 10 2022-10-18 03:45:00 239500 17527.80 17434.05 17438.75 17486.95  17486.95
## 11 2022-10-19 03:45:00 210500 17607.60 17472.85 17568.15 17512.25  17512.25
## 12 2022-10-20 03:45:00 249600 17584.15 17421.00 17423.10 17563.95  17563.95
## 13 2022-10-21 03:45:00 277700 17670.15 17520.75 17622.85 17576.30  17576.30
## 14 2022-10-24 03:45:00  45000 17777.55 17707.40 17736.35 17730.75  17730.75
## 15 2022-10-25 03:45:00 251400 17811.50 17637.00 17808.30 17656.35  17656.35
## 16 2022-10-27 03:45:00 324600 17783.90 17654.50 17771.40 17736.95  17736.95
## 17 2022-10-28 03:45:00 250000 17838.90 17723.70 17756.40 17786.80  17786.80
## 18 2022-10-31 03:45:00 227200 18022.80 17899.90 17910.20 18012.20  18012.20
## 19 2022-11-01 03:45:00 349900 18175.80 18060.15 18130.70 18145.40  18145.40
## 20 2022-11-02 03:45:00 270900 18178.75 18048.65 18177.90 18082.85  18082.85
## 21 2022-11-03 03:45:00 213000 18106.30 17959.20 17968.35 18052.70  18052.70
## 22 2022-11-04 09:00:36      0 18108.00 18017.15 18053.40 18074.30  18074.30</code></pre>
<p>
Please view the <a href="https://yahoofinancer.rsquaredacademy.com/reference/index-class#arguments-2">documentation</a> to see the valid values for <code>interval</code> and <code>period</code>.
</p>
</section>
</section>
<section id="currency" class="level2">
<h2 class="anchored" data-anchor-id="currency">
Currency
</h2>
<section id="summary-2" class="level4">
<h4 class="anchored" data-anchor-id="summary-2">
Summary
</h4>
<p>
<code>currency_summary()</code> retrieve information available via the Summary tab in Yahoo Finance.. View the <a href="https://yahoofinancer.rsquaredacademy.com/reference/currency_summary.html">documentation</a> to learn more.
</p>
<pre class="r"><code>currency_summary(from = "USD", to = "INR")</code></pre>
<pre><code>## $language
## [1] "en-US"
## 
## $region
## [1] "US"
## 
## $quoteType
## [1] "CURRENCY"
## 
## $typeDisp
## [1] "Currency"
## 
## $quoteSourceName
## [1] "Delayed Quote"
## 
## $triggerable
## [1] TRUE
## 
## $customPriceAlertConfidence
## [1] "HIGH"
## 
## $currency
## [1] "INR"
## 
## $shortName
## [1] "USD/INR"
## 
## $regularMarketChangePercent
## [1] -0.2913199
## 
## $regularMarketPrice
## [1] 82.485
## 
## $regularMarketChange
## [1] -0.2409973
## 
## $regularMarketDayHigh
## [1] 82.731
## 
## $regularMarketDayLow
## [1] 82.45
## 
## $regularMarketPreviousClose
## [1] 82.726
## 
## $bid
## [1] 82.48
## 
## $ask
## [1] 82.53
## 
## $regularMarketOpen
## [1] 82.731
## 
## $fiftyTwoWeekLow
## [1] 73.6975
## 
## $fiftyTwoWeekHigh
## [1] 83.386
## 
## $fiftyDayAverage
## [1] 81.12029
## 
## $twoHundredDayAverage
## [1] 78.23625
## 
## $exchange
## [1] "CCY"
## 
## $messageBoardId
## [1] "finmb_INR_X"
## 
## $exchangeTimezoneName
## [1] "Europe/London"
## 
## $exchangeTimezoneShortName
## [1] "GMT"
## 
## $gmtOffSetMilliseconds
## [1] 0
## 
## $market
## [1] "ccy_market"
## 
## $esgPopulated
## [1] FALSE
## 
## $marketState
## [1] "REGULAR"
## 
## $regularMarketTime
## [1] 1667552433
## 
## $regularMarketDayRange
## [1] "82.45 - 82.731"
## 
## $regularMarketVolume
## [1] 0
## 
## $bidSize
## [1] 0
## 
## $askSize
## [1] 0
## 
## $fullExchangeName
## [1] "CCY"
## 
## $averageDailyVolume3Month
## [1] 0
## 
## $averageDailyVolume10Day
## [1] 0
## 
## $fiftyTwoWeekLowChange
## [1] 8.787498
## 
## $fiftyTwoWeekLowChangePercent
## [1] 0.1192374
## 
## $fiftyTwoWeekRange
## [1] "73.6975 - 83.386"
## 
## $fiftyTwoWeekHighChange
## [1] -0.901001
## 
## $tradeable
## [1] FALSE
## 
## $cryptoTradeable
## [1] FALSE
## 
## $fiftyTwoWeekHighChangePercent
## [1] -0.01080518
## 
## $firstTradeDateMilliseconds
## [1] 1.070237e+12
## 
## $priceHint
## [1] 4
## 
## $fiftyDayAverageChange
## [1] 1.364708
## 
## $fiftyDayAverageChangePercent
## [1] 0.01682326
## 
## $twoHundredDayAverageChange
## [1] 4.248749
## 
## $twoHundredDayAverageChangePercent
## [1] 0.05430665
## 
## $sourceInterval
## [1] 15
## 
## $exchangeDataDelayedBy
## [1] 0
## 
## $symbol
## [1] "USDINR=X"</code></pre>
</section>
<section id="converter" class="level4">
<h4 class="anchored" data-anchor-id="converter">
Converter
</h4>
<p>
<code>currency_converter()</code> retrieve current conversion rate between two currencies as well as historical rates.
</p>
<pre class="r"><code>currency_converter('GBP', 'USD', '2022-07-01', '2022-07-10')</code></pre>
<pre><code>##                  date     high      low     open    close volume adj_close
## 1 2022-06-30 23:00:00 1.216205 1.198064 1.215998 1.216086      0  1.216086
## 2 2022-07-03 23:00:00 1.216515 1.208634 1.210580 1.210273      0  1.210273
## 3 2022-07-04 23:00:00 1.212606 1.190051 1.211402 1.211446      0  1.211446
## 4 2022-07-05 23:00:00 1.198638 1.187761 1.194957 1.194914      0  1.194914
## 5 2022-07-06 23:00:00 1.202183 1.191001 1.191895 1.192321      0  1.192321
## 6 2022-07-07 23:00:00 1.205531 1.192194 1.203196 1.202805      0  1.202805</code></pre>
<p>
The time between data points can be specified using the <code>interval</code> argument. The <code>period</code> arguments allows the user to retrieve data for specific length of time.
</p>
<pre class="r"><code>currency_converter('GBP', 'USD', period = '1mo', interval = '1d')</code></pre>
<pre><code>##                   date     high      low     open    close volume adj_close
## 1  2022-10-03 23:00:00 1.144597 1.128222 1.133967 1.133414      0  1.133414
## 2  2022-10-04 23:00:00 1.149491 1.122776 1.145056 1.145134      0  1.145134
## 3  2022-10-05 23:00:00 1.138343 1.112038 1.135589 1.135538      0  1.135538
## 4  2022-10-06 23:00:00 1.122372 1.109201 1.116520 1.116208      0  1.116208
## 5  2022-10-09 23:00:00 1.111086 1.102354 1.107444 1.107273      0  1.107273
## 6  2022-10-10 23:00:00 1.117131 1.100183 1.107849 1.107641      0  1.107641
## 7  2022-10-11 23:00:00 1.109866 1.092645 1.096780 1.096780      0  1.096780
## 8  2022-10-12 23:00:00 1.136196 1.105828 1.109927 1.110556      0  1.110556
## 9  2022-10-13 23:00:00 1.135873 1.116196 1.130148 1.130391      0  1.130391
## 10 2022-10-16 23:00:00 1.143864 1.121705 1.123873 1.124417      0  1.124417
## 11 2022-10-17 23:00:00 1.141201 1.125809 1.135525 1.135396      0  1.135396
## 12 2022-10-18 23:00:00 1.134984 1.122246 1.133954 1.134430      0  1.134430
## 13 2022-10-19 23:00:00 1.133324 1.117368 1.120951 1.121101      0  1.121101
## 14 2022-10-20 23:00:00 1.129318 1.106378 1.122536 1.122146      0  1.122146
## 15 2022-10-23 23:00:00 1.137980 1.127485 1.133645 1.134469      0  1.134469
## 16 2022-10-24 23:00:00 1.149954 1.127370 1.130544 1.130774      0  1.130774
## 17 2022-10-25 23:00:00 1.162358 1.143419 1.146066 1.146224      0  1.146224
## 18 2022-10-26 23:00:00 1.164415 1.155095 1.163345 1.163819      0  1.163819
## 19 2022-10-27 23:00:00 1.160012 1.150616 1.156738 1.156604      0  1.156604
## 20 2022-10-31 00:00:00 1.161332 1.147987 1.160079 1.159595      0  1.159595
## 21 2022-11-01 00:00:00 1.156524 1.144675 1.147184 1.146815      0  1.146815
## 22 2022-11-02 00:00:00 1.152738 1.145856 1.149214 1.148950      0  1.148950
## 23 2022-11-03 00:00:00 1.142178 1.116258 1.138226 1.138433      0  1.138433
## 24 2022-11-04 08:59:42 1.124088 1.115101 1.115922 1.122322      0  1.122322</code></pre>
<p>
View the <a href="https://yahoofinancer.rsquaredacademy.com/reference/currency_converter.html">documentation</a> to learn more.
</p>
</section>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<ul>
<li>
<a href="https://yahoofinancer.rsquaredacademy.com">Documentation</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy/yahoofinancer">GitHub</a>
</li>
</ul>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
All feedback is welcome. Issues (bugs and feature requests) can be posted to <a href="https://github.com/rsquaredacademy/yahoofinancer/issues">github tracker</a>. For help with code or other related questions, feel free to reach out to us at <a href="mailto:pkgs@rsquaredacademy.com" class="email">pkgs@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>packages</category>
  <guid>https://blog.rsquaredacademy.com/posts/introducing-yahoofinancer-fetch-data-from-yahoo-finance-api/</guid>
  <pubDate>Fri, 04 Nov 2022 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/github-yfr.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Handling Categorical Data in R - Part 4</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2022-01-14-handling-categorical-data-in-r-part-4.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<p>
<img src="https://blog.rsquaredacademy.com/img/forcats-part-4.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
This is part 4 of a series on “Handling Categorical Data in R” where we are learning to <strong>read</strong>, <strong>store</strong>, <strong>summarize</strong>, <strong>reshape</strong> &amp; <strong>visualize</strong> categorical data.
</p>
<p>
Below are the links to the other articles of this series:
</p>
<ul>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-1/">Part 1 - Introduction to Factor</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-2/">Part 2 - Summarize Categorical Data</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-3/">Part 3 - Reshape Categorical Data</a>
</li>
</ul>
<p>
In this article, we will explore the different ways of visualizing categorical data using <a href="https://ggplot2.tidyverse.org/">ggplot2</a>.
</p>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
You can download all the data sets, R scripts, practice questions and their solutions from our <a href="https://github.com/rsquaredacademy-education/online-courses/">GitHub</a> repository.
</p>
</section>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
In this section, we will learn to visualize categorical data. We will look at the following type of plots:
</p>
<ul>
<li>
univariate bar plot
</li>
<li>
bivariate bar plot
<ul>
<li>
grouped
</li>
<li>
stacked
</li>
<li>
proportional
</li>
</ul>
</li>
<li>
mosaic plot
</li>
<li>
pie chart
</li>
<li>
donut chart
</li>
</ul>
<p>
We will be using <a href="https://ggplot2.tidyverse.org">ggplot2</a> package throughout this article. So you should know the basics of data visualization with ggplot2. If you are new to or have never used ggplot2, do not worry. We have several <a href="https://blog.rsquaredacademy.com/data-visualization/index.html">tutorials</a> and an <a href="https://viz-ggplot2.rsquaredacademy.com/">ebook</a> on ggplot2, you can go through them first and then come back to this article. Let us read the case study data before we start our visualization journey.
</p>
<pre class="r"><code># read data
data &lt;- readRDS('analytics.rds')</code></pre>
</section>
<section id="bar-plot" class="level2">
<h2 class="anchored" data-anchor-id="bar-plot">
Bar Plot
</h2>
<p>
Bar charts provide a visual representation of categorical data. The bars can be plotted either vertically or horizontally. The categories/groups appear along the horizontal X axis and the height of the bar represents a measured value.
</p>
<pre class="r"><code>ggplot(data) +
  geom_bar(aes(x = device), fill = "blue") +
  xlab("Device") + ylab("Count")</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/bar_plot-1.png" width="672">
</p>
<p>
In the above example, the bars represent the count/frequency of the categories. If the bars represent continuous data, the value could be <em>mean</em> or <em>sum</em> of the variable being represented.
</p>
<section id="grouped-bar-plot" class="level3">
<h3 class="anchored" data-anchor-id="grouped-bar-plot">
Grouped Bar Plot
</h3>
<p>
A grouped bar chart plots values for two levels of a categorical variable instead of one. You should use grouped bar chart when making comparisons across different categories of data. Use it when you want to look at how the second category variable changes within each level of the first and vice versa.
</p>
<pre class="r"><code>ggplot(data) +
  geom_bar(aes(x = device, fill = gender), position = "dodge") +
  xlab("Device") + ylab("Count")</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/grouped_bar_plot-1.png" width="672">
</p>
</section>
<section id="stacked-bar-plot" class="level3">
<h3 class="anchored" data-anchor-id="stacked-bar-plot">
Stacked Bar Plot
</h3>
<p>
In stacked bar plots, the bars are stacked on top of each other instead of placing them next to each other. Use stacked bar plots while looking at cumulative value.
</p>
<pre class="r"><code>ggplot(data) +
  geom_bar(aes(x = device, fill = gender)) +
  xlab("Device") + ylab("Count")</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/stacked_bar_plot-1.png" width="672">
</p>
</section>
<section id="proportional-bar-plot" class="level3">
<h3 class="anchored" data-anchor-id="proportional-bar-plot">
Proportional Bar Plot
</h3>
<p>
Also known as percent stacked plot, the height of all bars in this plot are the same. The distribution of the second categorical variable is scaled to 1 or 100. The length of each bar is determined by its share in the category. Use this when you want to concurrently observe each of several variables as they fluctuate and as their percentage ratio’s change.
</p>
<pre class="r"><code>data %&gt;% 
  select(device, gender) %&gt;% 
  table() %&gt;% 
  tibble::as_tibble() %&gt;% 
  ggplot(aes(x = device, y = n, fill = gender)) +
  geom_bar(stat = "identity", position = "fill") +
  xlab("Device") + ylab("Gender")</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/prop_bar_plot-1.png" width="672">
</p>
</section>
</section>
<section id="mosaic-plot" class="level2">
<h2 class="anchored" data-anchor-id="mosaic-plot">
Mosaic Plot
</h2>
<p>
A mosaic plot is a graphical representation of a two way table or contingency table. It was introduced by Hartigan &amp; Kleiner and is divided into rectangles. Proportions on horizontal axis represents the number of observations for each level of the X variable. The vertical length of each rectangle is proportional to the proportion of Y variable in each level of X variable.
</p>
<pre class="r"><code>ggplot(data = data) +
  geom_mosaic(aes(x = product(channel, device), fill = channel)) +
  xlab("Device") + ylab("Channel")</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/mosaic_plot-1.png" width="672">
</p>
</section>
<section id="pie-chart" class="level2">
<h2 class="anchored" data-anchor-id="pie-chart">
Pie Chart
</h2>
<p>
Pie chart is a circular chart, divided into slices to show relevant sizes of data. It shows the distribution of the different levels of a categorical variable as a circle is divided into radial slices. Each level corresponds with a single slice of the circle and size indicates the proportion of the level. Use it when comparing each group’s contribution to the whole as opposed to comparing groups to each other.
</p>
<section id="base-r" class="level3">
<h3 class="anchored" data-anchor-id="base-r">
Base R
</h3>
<pre class="r"><code>data %&gt;% 
  pull(device) %&gt;%
  table() %&gt;%
  pie()</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/pie_chart-1.png" width="672">
</p>
</section>
<section id="d-pie-chart" class="level3">
<h3 class="anchored" data-anchor-id="d-pie-chart">
3D Pie Chart
</h3>
<pre class="r"><code>data %&gt;% 
  pull(device) %&gt;% 
  table() %&gt;% 
  pie3D(explode = 0.1)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/pie_chart_3d-1.png" width="672">
</p>
</section>
<section id="ggplot2" class="level3">
<h3 class="anchored" data-anchor-id="ggplot2">
ggplot2
</h3>
<pre class="r"><code>data %&gt;% 
  pull(device) %&gt;% 
  fct_count() %&gt;% 
  rename(device = f, count = n) %&gt;% 
  ggplot() +
  geom_bar(aes(x = "", y = count, fill = device), width = 1, stat = "identity") +
  coord_polar("y", start = 0)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/gg_pie_chart-1.png" width="672">
</p>
</section>
</section>
<section id="donut-chart" class="level2">
<h2 class="anchored" data-anchor-id="donut-chart">
Donut Chart
</h2>
<p>
Donut chart is a variation of the pie chart. It has a round hole in the middle which makes it look like a donut. The focus is on the length of the arcs and not the proportions of the slices. Blank spaces inside donut chart can be used to display information inside it.
</p>
<pre class="r"><code>data %&gt;% 
  pull(device) %&gt;% 
  fct_count() %&gt;% 
  rename(device = f, count = n) %&gt;% 
  ggdonutchart("count", label = "device", fill = "device", color = "white",
               palette = c("#00AFBB", "#E7B800", "#FC4E07"))</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/gg_donut_chart-1.png" width="672">
</p>
</section>
<section id="summary" class="level2">
<h2 class="anchored" data-anchor-id="summary">
Summary
</h2>
<ul>
<li>
Bar charts provide a visual representation of categorical data.
</li>
<li>
Use grouped bar chart to make comparison against different categories of data.
</li>
<li>
Use stacked bar chart while looking at cumulative data.
</li>
<li>
Use proportional bar chart when you want to concurrently observe each of the several variables as they fluctuate.
</li>
<li>
Use mosaic plot to discover associations between two variables.
</li>
<li>
Use pie chart and donut chart when comparing each group’s contribution to the whole.
</li>
</ul>
</section>
<section id="your-turn" class="level2">
<h2 class="anchored" data-anchor-id="your-turn">
Your Turn…
</h2>
<p>
Generate all the below plots:
</p>
<ol style="list-style-type: decimal">
<li>
Bar plot of <code>channel</code>
</li>
</ol>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_1-1.png" width="672" style="display: block; margin: auto;">
</p>
<ol start="2" style="list-style-type: decimal">
<li>
Display grouped bar plot of <code>user_type</code> by <code>channel</code>
</li>
</ol>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_2-1.png" width="672" style="display: block; margin: auto;">
</p>
<ol start="3" style="list-style-type: decimal">
<li>
Display stacked bar plot of <code>channel</code> by <code>gender</code>
</li>
</ol>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_3-1.png" width="672" style="display: block; margin: auto;">
</p>
<ol start="4" style="list-style-type: decimal">
<li>
Display proportional bar plot of <code>channel</code> by <code>device</code>
</li>
</ol>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_4-1.png" width="672" style="display: block; margin: auto;">
</p>
<ol start="5" style="list-style-type: decimal">
<li>
Display mosaic plot of <code>device</code> by <code>channel</code>
</li>
</ol>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_5-1.png" width="672" style="display: block; margin: auto;">
</p>
<ol start="6" style="list-style-type: decimal">
<li>
Display pie or donut chart of <code>channel</code>
</li>
</ol>
<section id="pie-chart-1" class="level5">
<h5 class="anchored" data-anchor-id="pie-chart-1">
6.1 Pie Chart
</h5>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_6-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
<section id="d-pie-chart-1" class="level5">
<h5 class="anchored" data-anchor-id="d-pie-chart-1">
6.2 3D Pie Chart
</h5>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_7-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
<section id="pie-chart-ggplot2" class="level5">
<h5 class="anchored" data-anchor-id="pie-chart-ggplot2">
6.3 Pie Chart (ggplot2)
</h5>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_8-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
<section id="donut-chart-1" class="level5">
<h5 class="anchored" data-anchor-id="donut-chart-1">
6.4 Donut Chart
</h5>
<p>
<img src="https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/2022-01-14-handling-categorical-data-in-r-part-4_files/figure-html/plot_9-1.png" width="960" style="display: block; margin: auto;">
</p>
<p>
*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.
</p>
<p>
We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*
</p>
<p>
<strong>Email: <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong>
</p>
</section>
</section>



 ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-4/</guid>
  <pubDate>Fri, 14 Jan 2022 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/forcats-part-4.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Handling Categorical Data in R - Part 3</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-3/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2022-01-13-handling-categorical-data-in-r-part-3.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<script src="../../rmarkdown-libs/kePrint/kePrint.js"></script>
<p><link href="../../rmarkdown-libs/lightable/lightable.css" rel="stylesheet"></p>
<p>
<img src="https://blog.rsquaredacademy.com/img/forcats-part-3.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
This is part 3 of a series on “Handling Categorical Data in R” where we are learning to <strong>read</strong>, <strong>store</strong>, <strong>summarize</strong>, <strong>reshape</strong> &amp; <strong>visualize</strong> categorical data.
</p>
<p>
Below are the links to the other articles of this series:
</p>
<ul>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-1/">Part 1 - Introduction to Factor</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-2/">Part 2 - Summarize Categorical Data</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-4/">Part 4 - Visualize Categorical Data</a>
</li>
</ul>
<p>
In this article, we will learn to manipulate/reshape categorical data by changing the value and order of levels/categories.
</p>
<section id="table-of-contents" class="level2">
<h2 class="anchored" data-anchor-id="table-of-contents">
Table of Contents
</h2>
<ul>
<li>
Resources
</li>
<li>
Introduction
</li>
<li>
How to change value of levels?
</li>
<li>
How to add or remove levels?
</li>
<li>
How to change order of levels?
</li>
<li>
Practice Questions
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
You can download all the data sets, R scripts, practice questions and their solutions from our <a href="https://github.com/rsquaredacademy-education/online-courses/">GitHub</a> repository.
</p>
</section>
<section id="intro" class="level2">
<h2 class="anchored" data-anchor-id="intro">
Introduction
</h2>
<p>
In this section, our focus will be on handling the levels of a categorical variable, and exploring the <a href="https://forcats.tidyverse.org/">forcats</a> package for the same. We will basically look at 3 key operations or transformations we would like to do when it comes to factors which are:
</p>
<ul>
<li>
change value of levels
</li>
<li>
add or remove levels
</li>
<li>
change order of levels
</li>
</ul>
<p>
Before we start working with the value of the levels, let us read the case study data and take a quick look at some of the functions we used in the previous articles.
</p>
<pre class="r"><code># read data
data &lt;- readRDS('analytics.rds')</code></pre>
<p>
We will store the source of traffic as <code>channel</code> instead of referring to the column in the <code>data.frame</code> every time.
</p>
<pre class="r"><code>channel &lt;- data$channel</code></pre>
<p>
Let us go back to the function we used for tabulating data, <code>fct_count()</code>. If you observe the result, it is in the same order as displayed by <code>levels()</code>.
</p>
<pre class="r"><code>fct_count(channel)</code></pre>
<pre><code>## # A tibble: 8 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Display          3375
## 5 Organic Search 139668
## 6 Paid Search      4395
## 7 Referral        35615
## 8 Social           8031</code></pre>
<p>
If you want to sort the results by the count i.e.&nbsp;most common level comes at the top, use the <code>sort</code> argument.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_sort.png" width="90%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>fct_count(channel, sort = TRUE)</code></pre>
<pre><code>## # A tibble: 8 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 Organic Search 139668
## 2 Direct          39853
## 3 Referral        35615
## 4 Social           8031
## 5 Affiliates       7388
## 6 (Other)          6073
## 7 Paid Search      4395
## 8 Display          3375</code></pre>
<p>
If you want to view the proportion along with the count, set the <code>prop</code> argument to <code>TRUE</code>.
</p>
<pre class="r"><code>fct_count(channel, prop = TRUE)</code></pre>
<pre><code>## # A tibble: 8 x 3
##   f                   n      p
##   &lt;fct&gt;           &lt;int&gt;  &lt;dbl&gt;
## 1 (Other)          6073 0.0248
## 2 Affiliates       7388 0.0302
## 3 Direct          39853 0.163 
## 4 Display          3375 0.0138
## 5 Organic Search 139668 0.571 
## 6 Paid Search      4395 0.0180
## 7 Referral        35615 0.146 
## 8 Social           8031 0.0329</code></pre>
<p>
One of the important steps in data preparation/sanitization is to check if the levels are valid i.e.&nbsp;only levels which should be present in the data are actually present. <code>fct_match()</code> can be used to check validity of levels. It returns a logical vector if the level is present and an error if not.
</p>
<pre class="r"><code>table(fct_match(channel, "Social"))</code></pre>
<pre><code>## 
##  FALSE   TRUE 
## 236367   8031</code></pre>
</section>
<section id="changevalue" class="level2">
<h2 class="anchored" data-anchor-id="changevalue">
Change Value of Levels
</h2>
<p>
In this section, we will learn how to change the value of the levels. In order to keep it interesting, we will state an objective from our case study and then map it into a function from the <a href="https://forcats.tidyverse.org/">forcats</a> package.
</p>
<section id="combine-both-paid-organic-search-into-a-single-level-search" class="level3">
<h3 class="anchored" data-anchor-id="combine-both-paid-organic-search-into-a-single-level-search">
Combine both Paid &amp; Organic Search into a single level, Search
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_collapse.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In this case, we want to change the value of two levels, <em>Paid Search</em> &amp; <em>Organic Search</em> and give them the common value <em>Search</em>. You can also look at it as collapsing two levels into one. There are two functions we can use here:
</p>
<ul>
<li>
<code>fct_collapse()</code>
</li>
<li>
<code>fct_recode()</code>
</li>
</ul>
<p>
Let us look at <code>fct_collapse()</code> first. After specifying the categorical variable, we specify the new value followed by a character vector of the existing values. Remember, the new value is not enclosed in quotes (single or double) but the existing values must be a <code>character</code> vector.
</p>
<pre class="r"><code>fct_count(
  fct_collapse(
    channel,
    Search = c("Paid Search", "Organic Search")
  )
)</code></pre>
<pre><code>## # A tibble: 7 x 2
##   f               n
##   &lt;fct&gt;       &lt;int&gt;
## 1 (Other)      6073
## 2 Affiliates   7388
## 3 Direct      39853
## 4 Display      3375
## 5 Search     144063
## 6 Referral    35615
## 7 Social       8031</code></pre>
<p>
In the case of <code>fct_recode()</code>, each value being changed must be specified in a new line. Similar to <code>fct_collapse()</code>, the new value is not enclosed in quotes but the existing values must be.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_recode.png" width="90%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>fct_count(
  fct_recode(
    channel,
    Search = "Paid Search",
    Search = "Organic Search"
  )
)</code></pre>
<pre><code>## # A tibble: 7 x 2
##   f               n
##   &lt;fct&gt;       &lt;int&gt;
## 1 (Other)      6073
## 2 Affiliates   7388
## 3 Direct      39853
## 4 Display      3375
## 5 Search     144063
## 6 Referral    35615
## 7 Social       8031</code></pre>
<p>
The <a href="https://dplyr.tidyverse.org/">dplyr</a> and <a href="https://cran.r-project.org/package=car">car</a> packages also have recode functions.
</p>
</section>
<section id="retain-only-those-channels-which-have-driven-a-minimum-traffic-of-5000-to-the-website" class="level3">
<h3 class="anchored" data-anchor-id="retain-only-those-channels-which-have-driven-a-minimum-traffic-of-5000-to-the-website">
Retain only those channels which have driven a minimum traffic of 5000 to the website
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_lump_1.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
Instead of having all the channels, we desire to retain only those channels which have driven at least <em>5000</em> visits to the website. What about the rest of the channels which have driven less than <em>5000</em>? We will recategorize them as <strong>Other</strong>. Keep in mind that we already have a <strong>(Other)</strong> level in our data. <code>fct_lump_min()</code> will lump together all levels which do not have a minimum count specified. In our case study, only <strong>Display</strong> drives less than <em>5000</em> visits and it will be categorized into <strong>Other</strong>.
</p>
<pre class="r"><code>fct_count(fct_lump_min(channel, 5000))</code></pre>
<pre><code>## # A tibble: 7 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770</code></pre>
</section>
<section id="retain-only-top-3-referring-channels-and-categorize-rest-into-other" class="level3">
<h3 class="anchored" data-anchor-id="retain-only-top-3-referring-channels-and-categorize-rest-into-other">
Retain only top 3 referring channels and categorize rest into Other
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_lump_2.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
Suppose you decide to retain only the top 3 channels in terms of the traffic driven to the website. In our case study, these are <strong>Direct</strong>, <strong>Organic Search</strong> and <strong>Referral</strong>. We want to retain these 3 levels and categorize the rest as <strong>Other</strong>. <code>fct_lump_n()</code> will retain top <code>n</code> levels by count/frequency and lump the rest into <strong>Other</strong>.
</p>
<pre class="r"><code>fct_count(fct_lump_n(channel, 3))</code></pre>
<pre><code>## # A tibble: 4 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 Direct          39853
## 2 Organic Search 139668
## 3 Referral        35615
## 4 Other           29262</code></pre>
<p>
In our case, <code>n</code> is 3 and hence the top 3 channels in terms of traffic driven are retained while the rest are lumped into <strong>Other</strong>.
</p>
</section>
<section id="retain-only-those-channels-which-have-driven-at-least-2-of-the-overall-traffic" class="level3">
<h3 class="anchored" data-anchor-id="retain-only-those-channels-which-have-driven-at-least-2-of-the-overall-traffic">
Retain only those channels which have driven at least 2% of the overall traffic
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_lump_3.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In the second scenario above, we retained channels based on minimum traffic driven by them to the website. The criteria was count of visits. If you want to specify the criteria as a percentage or proportion instead of count, use <code>fct_lump_prop()</code>. The criteria is a value between <strong>0</strong> and <strong>1</strong>. In our case study, we want to retain channels that have driven at least <strong>2%</strong> of the overall traffic. Hence, we have specified the criteria as <code>0.02</code>.
</p>
<pre class="r"><code>fct_count(fct_lump_prop(channel, 0.02))</code></pre>
<pre><code>## # A tibble: 7 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770</code></pre>
<p>
As you can see, only <strong>Display</strong> drives less than <strong>2%</strong> of overall traffic and has been lumped into <strong>Other</strong>.
</p>
</section>
<section id="retain-the-following-channels-and-merge-the-rest-into-other" class="level3">
<h3 class="anchored" data-anchor-id="retain-the-following-channels-and-merge-the-rest-into-other">
Retain the following channels and merge the rest into Other
</h3>
<ul>
<li>
Organic Search
</li>
<li>
Direct
</li>
<li>
Referral
</li>
</ul>
<p>
In the previous scenarios, we have been retaining or lumping channels based on some criteria like count or percentage of traffic driven to the website. In this scenario, we want to retain certain levels by specifying their labels and combine the rest into <strong>Other</strong>. While we can use <code>fct_collapse()</code> or <code>fct_recode()</code>, a more appropriate function would be <code>fct_other()</code>. We will do a comparison of the three functions in a short while.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_others.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
<code>fct_other()</code> has two arguments, <code>keep</code> and <code>drop</code>. <code>keep</code> is used when we know the levels we want to retain and <code>drop</code> is used when we know the levels we want to drop. In this scenario, we know the levels we want to retain and hence we will use the <code>keep</code> argument and specify them. <strong>Organic Search</strong>, <strong>Direct</strong> and <strong>Referral</strong> will be retained while the rest of the channels will be lumped into <strong>Other</strong>.
</p>
<pre class="r"><code>fct_count(
  fct_other(
    channel, 
    keep = c("Organic Search", "Direct", "Referral"))
)</code></pre>
<pre><code>## # A tibble: 4 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 Direct          39853
## 2 Organic Search 139668
## 3 Referral        35615
## 4 Other           29262</code></pre>
</section>
<section id="merge-the-following-channels-into-other-and-retain-rest-of-them" class="level3">
<h3 class="anchored" data-anchor-id="merge-the-following-channels-into-other-and-retain-rest-of-them">
Merge the following channels into Other and retain rest of them:
</h3>
<ul>
<li>
Display
</li>
<li>
Paid Search
</li>
</ul>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_others_drop.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In this scenario, we know the levels we want to drop and hence we will use the <code>drop</code> argument and specify them. <strong>Display</strong> and <strong>Paid Search</strong> will be lumped into <strong>Other</strong> while the rest of the channels will be retained.
</p>
<pre class="r"><code>fct_count(
  fct_other(
    channel, 
    drop = c("Display", "Paid Search")
  )
)</code></pre>
<pre><code>## # A tibble: 7 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 (Other)          6073
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615
## 6 Social           8031
## 7 Other            7770</code></pre>
<p>
In the previous scenario, we said we will compare <code>fct_other()</code> with <code>fct_collapse()</code> and <code>fct_recode()</code>. Let us use the other two functions as well and see the difference.
</p>
<pre class="r"><code># collapse
fct_count(
  fct_collapse(
  channel,
  Other = c("(Other)", "Affiliate", "Display", "Paid Search", "Social")
  )
)</code></pre>
<pre><code>## Warning: Unknown levels in `f`: Affiliate</code></pre>
<pre><code>## # A tibble: 5 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 Other           21874
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615</code></pre>
<pre class="r"><code># recode
fct_count(
  fct_recode(
  channel,
  Other = "(Other)", 
  Other = "Affiliate", 
  Other = "Display", 
  Other = "Paid Search", 
  Other = "Social"
  )
)</code></pre>
<pre><code>## Warning: Unknown levels in `f`: Affiliate</code></pre>
<pre><code>## # A tibble: 5 x 2
##   f                   n
##   &lt;fct&gt;           &lt;int&gt;
## 1 Other           21874
## 2 Affiliates       7388
## 3 Direct          39853
## 4 Organic Search 139668
## 5 Referral        35615</code></pre>
<p>
As you can observe, <code>fct_other()</code> requires less typing and is easier to specify.
</p>
</section>
<section id="anonymize-the-data-set-before-sharing-it-with-your-colleagues" class="level3">
<h3 class="anchored" data-anchor-id="anonymize-the-data-set-before-sharing-it-with-your-colleagues">
Anonymize the data set before sharing it with your colleagues
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_anonymize.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
Anonymizing data is extremely important when you are sharing sensitive data with others. Here, we want to anonymize the channels which drive traffic to the website so that we can share it with others without divulging the names of the channels. <code>fct_anon()</code> allows us to anonymize the levels in the data. Using the <code>prefix</code> argument, we can specify the prefix to be used while anonymizing the data.
</p>
<pre class="r"><code>fct_count(fct_anon(channel, prefix = "ch_"))</code></pre>
<pre><code>## # A tibble: 8 x 2
##   f          n
##   &lt;fct&gt;  &lt;int&gt;
## 1 ch_1    6073
## 2 ch_2    8031
## 3 ch_3    4395
## 4 ch_4   39853
## 5 ch_5  139668
## 6 ch_6    3375
## 7 ch_7   35615
## 8 ch_8    7388</code></pre>
</section>
<section id="key-functions" class="level3">
<h3 class="anchored" data-anchor-id="key-functions">
Key Functions
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Function
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>fct_collapse()</code>
</td>
<td style="text-align:left;">
Collapse factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_recode()</code>
</td>
<td style="text-align:left;">
Recode factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_lump_min()</code>
</td>
<td style="text-align:left;">
Lump factor levels with count lesser than specified value
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_lump_n()</code>
</td>
<td style="text-align:left;">
Lump all levels except the top n levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_lump_prop()</code>
</td>
<td style="text-align:left;">
Lump factor levels with count lesser than specified proportion
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_lump_lowfreq()</code>
</td>
<td style="text-align:left;">
Lump together least frequent levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_other()</code>
</td>
<td style="text-align:left;">
Replace levels with Other level
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_anon()</code>
</td>
<td style="text-align:left;">
Anonymize factor levels
</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="addremove" class="level2">
<h2 class="anchored" data-anchor-id="addremove">
Add / Remove Levels
</h2>
<p>
In this small section, we will learn to:
</p>
<ul>
<li>
add new levels
</li>
<li>
drop levels
</li>
<li>
make missing values explicit
</li>
</ul>
<section id="add-a-new-level-blog" class="level3">
<h3 class="anchored" data-anchor-id="add-a-new-level-blog">
Add a new level, Blog
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_expand.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
<code>fct_expand()</code> allows us to add new levels to the data. The label of the new level must be specified after the variable name and must be enclosed in quotes. If the level already exists, it will be ignored. Let us add a new level, <code>Blog</code>.
</p>
<pre class="r"><code>levels(fct_expand(channel, "Blog"))</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"        
## [9] "Blog"</code></pre>
</section>
<section id="drop-existing-level" class="level3">
<h3 class="anchored" data-anchor-id="drop-existing-level">
Drop existing level
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_drop.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
On the other hand, <code>fct_drop()</code> will drop levels which have no values i.e.&nbsp;unused levels. If you want to drop only specific levels, use the <code>only</code> argument and specify the name of the level in quotes. Let us drop the new level we added in the previous example.
</p>
<pre class="r"><code>levels(fct_drop(fct_expand(channel, "Blog")))</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
</section>
<section id="make-missing-values-explicit" class="level3">
<h3 class="anchored" data-anchor-id="make-missing-values-explicit">
Make missing values explicit
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_explicit_na.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In our data set, the gender column has many missing values, and in R, missing values are represented by <code>NA</code>. Suppose you are sharing the data or analysis with someone who is not an R user, and does not know what <code>NA</code> represents. In such a scenario, we can use the <code>fct_explicit_na()</code> function to make the missing values in the gender column explicit i.e.&nbsp;it will appear as <code>(Missing)</code> instead of <code>NA</code>. This will help non R users to understand that there are missing values in the data.
</p>
<pre class="r"><code>fct_count(fct_explicit_na(data$gender))</code></pre>
<pre><code>## # A tibble: 3 x 2
##   f              n
##   &lt;fct&gt;      &lt;int&gt;
## 1 female     40565
## 2 male       61617
## 3 (Missing) 142216</code></pre>
</section>
<section id="key-functions-1" class="level3">
<h3 class="anchored" data-anchor-id="key-functions-1">
Key Functions
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Function
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>fct_expand()</code>
</td>
<td style="text-align:left;">
Add additional levels to a factor
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_drop()</code>
</td>
<td style="text-align:left;">
Drop unused factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_explicit_na()</code>
</td>
<td style="text-align:left;">
Make missing values explicit
</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="changeorder" class="level2">
<h2 class="anchored" data-anchor-id="changeorder">
Change Order of Levels
</h2>
<p>
In this last section, we will learn how to change the order of the levels. We will look at the following scenarios from our case study:
</p>
<p>
We want to make
</p>
<ul>
<li>
Organic Search the first level
</li>
<li>
Referral the third level
</li>
<li>
Display the last level
</li>
</ul>
<section id="make-organic-search-the-first-level" class="level3">
<h3 class="anchored" data-anchor-id="make-organic-search-the-first-level">
Make Organic Search the first level
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_relevel_1.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In this scenario, we want the levels to appear in a certain order. In the first case, we want <strong>Organic Search</strong> to be the first level. <code>fct_relevel()</code> allows us to manually reorder the levels. To move a level to the beginning, specify the label (it must be enclosed in quotes).
</p>
<pre class="r"><code>levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_relevel(channel, "Organic Search"))</code></pre>
<pre><code>## [1] "Organic Search" "(Other)"        "Affiliates"     "Direct"        
## [5] "Display"        "Paid Search"    "Referral"       "Social"</code></pre>
</section>
<section id="make-referral-the-third-level" class="level3">
<h3 class="anchored" data-anchor-id="make-referral-the-third-level">
Make Referral the third level
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_relevel_2.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
The <code>after</code> argument is useful when we want to move the level to the end or anywhere between the beginning and end. In the second case, we want <strong>Referral</strong> to be the third level. After specifying the label, use the <code>after</code> argument and specify the level after which <strong>Referral</strong> should appear. Since we want to move it to the third position, we will set the value of <code>after</code> to <code>2</code> i.e.&nbsp;<strong>Referral</strong> should come after the second position.
</p>
<pre class="r"><code>levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_relevel(channel, "Referral", after = 2))</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Referral"       "Direct"        
## [5] "Display"        "Organic Search" "Paid Search"    "Social"</code></pre>
</section>
<section id="make-display-the-last-level" class="level3">
<h3 class="anchored" data-anchor-id="make-display-the-last-level">
Make Display the last level
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_relevel_3.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In this last case, we want to move <strong>Display</strong> to the end. If you know the number of levels, you can specify a value here. In our data, there are eight channels i.e.&nbsp;eight levels, so we can set the value of <code>after</code> to <code>7</code>. What happens when we do not know the number of levels or if they tend to vary? In such cases, to move a level to the end, set the value of <code>after</code> to <code>Inf</code>.
</p>
<pre class="r"><code>levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_relevel(channel, "Display", after = Inf))</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Organic Search"
## [5] "Paid Search"    "Referral"       "Social"         "Display"</code></pre>
<p>
Let us now look at a scenario where we want to order the levels by
</p>
<ul>
<li>
frequency (largest to smallest)
</li>
<li>
order of appearance (in data)
</li>
</ul>
</section>
<section id="order-levels-by-frequency" class="level3">
<h3 class="anchored" data-anchor-id="order-levels-by-frequency">
Order levels by frequency
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_infreq.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In the first case, the levels with the most frequency should appear at the top. <code>fct_infreq()</code> will order the levels by their frequency.
</p>
<pre class="r"><code># reorder levels
levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_infreq(channel))</code></pre>
<pre><code>## [1] "Organic Search" "Direct"         "Referral"       "Social"        
## [5] "Affiliates"     "(Other)"        "Paid Search"    "Display"</code></pre>
</section>
<section id="order-levels-by-appearance" class="level3">
<h3 class="anchored" data-anchor-id="order-levels-by-appearance">
Order levels by appearance
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_inorder.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
In the second case, the order of the levels should be the same as the order of their appearance in the data. <code>fct_inorder()</code> will order the levels according to the order in which they appear in the data.
</p>
<pre class="r"><code># reorder levels
levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_inorder(channel))</code></pre>
<pre><code>## [1] "Organic Search" "Direct"         "Referral"       "Affiliates"    
## [5] "(Other)"        "Social"         "Display"        "Paid Search"</code></pre>
</section>
<section id="reverse-the-order-of-the-levels" class="level3">
<h3 class="anchored" data-anchor-id="reverse-the-order-of-the-levels">
Reverse the order of the levels
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_rev.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
The order of the levels can be reversed using <code>fct_rev()</code>.
</p>
<pre class="r"><code># reorder levels
levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_rev(channel))</code></pre>
<pre><code>## [1] "Social"         "Referral"       "Paid Search"    "Organic Search"
## [5] "Display"        "Direct"         "Affiliates"     "(Other)"</code></pre>
</section>
<section id="randomly-shuffle-the-order-of-the-levels" class="level3">
<h3 class="anchored" data-anchor-id="randomly-shuffle-the-order-of-the-levels">
Randomly shuffle the order of the levels
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_shuffle.png" width="90%" style="display: block; margin: auto;">
</p>
<p>
The order of the levels can be randomly shuffled using <code>fct_shuffle()</code>.
</p>
<pre class="r"><code># reorder levels
levels(channel)</code></pre>
<pre><code>## [1] "(Other)"        "Affiliates"     "Direct"         "Display"       
## [5] "Organic Search" "Paid Search"    "Referral"       "Social"</code></pre>
<pre class="r"><code>levels(fct_shuffle(channel))</code></pre>
<pre><code>## [1] "Organic Search" "Referral"       "Display"        "(Other)"       
## [5] "Direct"         "Paid Search"    "Affiliates"     "Social"</code></pre>
</section>
<section id="key-functions-2" class="level3">
<h3 class="anchored" data-anchor-id="key-functions-2">
Key Functions
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Function
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>fct_relevel()</code>
</td>
<td style="text-align:left;">
Reorder factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_shift()</code>
</td>
<td style="text-align:left;">
Shift factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_infreq()</code>
</td>
<td style="text-align:left;">
Reorder factor levels by frequency
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_rev()</code>
</td>
<td style="text-align:left;">
Reverse order of factor levels
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_inorder()</code>
</td>
<td style="text-align:left;">
Reorder factor levels by first appearance
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>fct_shuffle()</code>
</td>
<td style="text-align:left;">
Randomly shuffle factor levels
</td>
</tr>
</tbody>
</table>
</section>
</section>
<section id="practice" class="level2">
<h2 class="anchored" data-anchor-id="practice">
Your Turn…
</h2>
<ol style="list-style-type: decimal">
<li>
<p>
Display the count/frequency of the following variables in the descending order
</p>
<ul>
<li>
<code>device</code>
</li>
<li>
<code>landing_page</code>
</li>
<li>
<code>exit_page</code>
</li>
</ul>
</li>
<li>
<p>
Check if <code>laptop</code> is a level in the <code>device</code> column.
</p>
</li>
<li>
<p>
Combine the following levels in <code>landing_page</code> into <code>Account</code>
</p>
<ul>
<li>
<code>My Account</code>
</li>
<li>
<code>Register</code>
</li>
<li>
<code>Sign In</code>
</li>
<li>
<code>Your Info</code>
</li>
</ul>
</li>
<li>
<p>
Combine levels in <code>landing_page</code> that drive less than 1000 visits.
</p>
</li>
<li>
<p>
Get top 10 landing and exit pages.
</p>
</li>
<li>
<p>
Get landing pages that drive at least 5% of the total traffic to the website.
</p>
</li>
<li>
<p>
Retain only the following levels in the <code>browser</code> column:
</p>
<ul>
<li>
<code>Chrome</code>
</li>
<li>
<code>Firefox</code>
</li>
<li>
<code>Safari</code>
</li>
<li>
<code>Edge</code>
</li>
</ul>
</li>
<li>
<p>
Anonymize landing and exit page levels.
</p>
</li>
<li>
<p>
Make <code>Home</code> first level in the <code>landing_page</code> column.
</p>
</li>
<li>
<p>
Make <code>Apparel</code> second level in the <code>landing_page</code> column.
</p>
</li>
<li>
<p>
Make <code>Specials</code> last level in the <code>landing_page</code> column.
</p>
</li>
<li>
<p>
Order the levels in the browser by frequency:
</p>
</li>
<li>
<p>
Order the levels in landing page by appearance:
</p>
</li>
<li>
<p>
Shuffle the levels in os
</p>
</li>
<li>
<p>
Reverse the levels in browser
</p>
</li>
</ol>
<p>
*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.
</p>
<p>
We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*
</p>
<p>
<strong>Email: <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong>
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-3/</guid>
  <pubDate>Thu, 13 Jan 2022 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/forcats-part-3.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Handling Categorical Data in R - Part 2</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-2/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2022-01-12-handling-categorical-data-in-r-part-2.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<p>
<img src="https://blog.rsquaredacademy.com/img/forcats-part-2.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
This is part 2 of a series on “Handling Categorical Data in R” where we are learning to <strong>read</strong>, <strong>store</strong>, <strong>summarize</strong>, <strong>reshape</strong> &amp; <strong>visualize</strong> categorical data.
</p>
<p>
Below are the links to the other articles of this series:
</p>
<ul>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-1/">Part 1 - Introduction to Factor</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-3/">Part 3 - Reshape Categorical Data</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-4/">Part 4 - Visualize Categorical Data</a>
</li>
</ul>
<p>
In this article, we will learn to summarize categorical data. In the process, we will do a deep dive on working with tables in R and explore a diverse set of packages.
</p>
<section id="table-of-contents" class="level2">
<h2 class="anchored" data-anchor-id="table-of-contents">
Table of Contents
</h2>
<ul>
<li>
Resources
</li>
<li>
Introduction
</li>
<li>
Tables in R
</li>
<li>
Practice Questions
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
You can download all the data sets, R scripts, practice questions and their solutions from our <a href="https://github.com/rsquaredacademy-education/online-courses/">GitHub</a> repository.
</p>
</section>
<section id="intro" class="level2">
<h2 class="anchored" data-anchor-id="intro">
Introduction
</h2>
<p>
Categorical data cannot be summarized in the same way as numeric data. It does not make sense to look at range, standard deviation etc. since data consists of a few distinct values only. So how do we summarize such data? We can look at
</p>
<ul>
<li>
count/frequency
</li>
<li>
proportion
</li>
<li>
cumulative frequency
</li>
<li>
cross table
</li>
<li>
contingency table etc.
</li>
</ul>
<p>
In this section, we will explore the above ways of summarizing categorical data. We will also spend some time learning about tables as you will be using them extensively while working with categorical data. R has many packages for tabulating data and we list and explore all of them in the R scripts shared in the GitHub repository.
</p>
<section id="number-of-categories" class="level3">
<h3 class="anchored" data-anchor-id="number-of-categories">
Number of Categories
</h3>
<p>
From our case study, we want to know the number of devices used to browse the website, the name of the devices and the proportion of traffic they drive to our website. Let us read the case study data set before we analyze the website traffic.
</p>
<pre class="r"><code># read data
data &lt;- readRDS('analytics.rds')</code></pre>
<p>
Let us begin with the number of devices. To view the number of groups/categories in a categorical variable, use <code>nlevels()</code>.
</p>
<pre class="r"><code>nlevels(data$device)</code></pre>
<pre><code>## [1] 3</code></pre>
<p>
There are 3 categories of devices used by the visitors to browse the website. This can also be used for data sanitization i.e.&nbsp;as an analyst you know that there are only 3 valid categories of device into which any visitor can be classified into. If you see more than 3 categories, you might want to check if there are any issues in data collection or processing. Now that we know there are 3 categories of devices, let us check if they are valid. The <code>levels()</code> function will return the labels of the groups.
</p>
</section>
<section id="category-names" class="level3">
<h3 class="anchored" data-anchor-id="category-names">
Category Names
</h3>
<p>
Knowing the number of levels is useful but not sufficient. <code>levels()</code> is one of the most useful functions when it comes to dealing with categorical data.
</p>
<pre class="r"><code>levels(data$device)</code></pre>
<pre><code>## [1] "Desktop" "Mobile"  "Tablet"</code></pre>
<p>
Other functions that you can use include <code>unique()</code> and <code>fct_unique()</code>. Both these functions will return the unique names/labels along with the levels while <code>levels()</code> returns the labels of the levels.
</p>
<pre class="r"><code>unique(data$device)</code></pre>
<pre><code>## [1] Desktop Mobile  Tablet 
## Levels: Desktop Mobile Tablet</code></pre>
<pre class="r"><code>fct_unique(data$device)</code></pre>
<pre><code>## [1] Desktop Mobile  Tablet 
## Levels: Desktop Mobile Tablet</code></pre>
</section>
<section id="names-counts" class="level3">
<h3 class="anchored" data-anchor-id="names-counts">
Names &amp; Counts
</h3>
<p>
So we have checked the number of devices and their names. Let us now examine their distribution i.e.&nbsp;count/frequency. <code>table()</code> and <code>summary()</code> will display the levels and their counts while <code>fct_count()</code> will return a tibble with 2 columns (level &amp; count). It is extremely useful for further data processing or visualization (using ggplot2).
</p>
<pre class="r"><code>table(data$device)</code></pre>
<pre><code>## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634</code></pre>
<pre class="r"><code>fct_count(data$device)</code></pre>
<pre><code>## # A tibble: 3 x 2
##   f            n
##   &lt;fct&gt;    &lt;int&gt;
## 1 Desktop 177282
## 2 Mobile   63482
## 3 Tablet    3634</code></pre>
<pre class="r"><code>summary(data$device)</code></pre>
<pre><code>## Desktop  Mobile  Tablet 
##  177282   63482    3634</code></pre>
</section>
</section>
<section id="tables" class="level2">
<h2 class="anchored" data-anchor-id="tables">
Tables
</h2>
<p>
In the previous section, we used the <code>table()</code> function to tabulate categorical data. We will recreate the tabulation for device and store it in a new variable tab.
</p>
<pre class="r"><code>tab &lt;- table(data$device)
tab</code></pre>
<pre><code>## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634</code></pre>
<p>
What does this function return? It is not a <code>vector</code>, <code>list</code>, <code>data.frame</code> or <code>matrix</code>. Let us use the <code>class()</code> function to check the class of the object returned by <code>table()</code>. It returns an object of the class <strong>table</strong>. This is a new type of object. Let us spend some time understanding tables as they are useful for organizing and summarizing categorical data. <strong>table</strong> is also the most used object when it comes to dealing with categorical data.
</p>
<p>
The <code>table()</code> function returns the counts of the categories but let us say we want to view the proportion or percentage instead of counts i.e.&nbsp;the proportion or percentage of traffic driven to our website by the different devices. The <code>proportions()</code> or <code>prop.table()</code> function comes in handy in such cases. It takes a table object as input (tab in our case).
</p>
<pre class="r"><code>prop.table(tab)</code></pre>
<pre><code>## 
##    Desktop     Mobile     Tablet 
## 0.72538237 0.25974844 0.01486919</code></pre>
<pre class="r"><code>proportions(tab)</code></pre>
<pre><code>## 
##    Desktop     Mobile     Tablet 
## 0.72538237 0.25974844 0.01486919</code></pre>
<p>
To get the percentages, multiply the output by 100. Use the <code>round()</code> function to round the decimal places according to your requirements.
</p>
<pre class="r"><code>proportions(tab) * 100</code></pre>
<pre><code>## 
##   Desktop    Mobile    Tablet 
## 72.538237 25.974844  1.486919</code></pre>
<pre class="r"><code>round(proportions(tab) * 100, 2)</code></pre>
<pre><code>## 
## Desktop  Mobile  Tablet 
##   72.54   25.97    1.49</code></pre>
<p>
So far, we have used <code>table()</code> to tabulate a single categorical variable. It can be used for a lot more than just tabulating data. We can examine the relationship between two categorical variables as well as create multidimensional tables. Let us look at the relationship between <strong>gender</strong> and <strong>device</strong> in our case study. Does gender affect the type of device used? To answer this, we will create a two way or cross table. In the <code>table()</code> function, we can specify multiple variables by separating them with a comma.
</p>
<pre class="r"><code>tab2 &lt;- table(data$gender, data$device)
tab2</code></pre>
<pre><code>##         
##          Desktop Mobile Tablet
##   female   32803   7268    494
##   male     46418  14503    696
##   &lt;NA&gt;     98061  41711   2444</code></pre>
<p>
Keep in mind that the order of the variables matter. Rows represent the first variable while column represents the second.
</p>
<pre class="r"><code>table(data$device, data$gender)</code></pre>
<pre><code>##          
##           female  male  &lt;NA&gt;
##   Desktop  32803 46418 98061
##   Mobile    7268 14503 41711
##   Tablet     494   696  2444</code></pre>
<p>
The <code>proportions()</code> function works with two way tables as well.
</p>
<pre class="r"><code>proportions(tab2)</code></pre>
<pre><code>##         
##              Desktop      Mobile      Tablet
##   female 0.134219593 0.029738378 0.002021293
##   male   0.189927904 0.059341729 0.002847814
##   &lt;NA&gt;   0.401234871 0.170668336 0.010000082</code></pre>
<pre class="r"><code>proportions(tab2) * 100</code></pre>
<pre><code>##         
##             Desktop     Mobile     Tablet
##   female 13.4219593  2.9738378  0.2021293
##   male   18.9927904  5.9341729  0.2847814
##   &lt;NA&gt;   40.1234871 17.0668336  1.0000082</code></pre>
<p>
We would like to introduce another function at this point of time, <code>margin.table()</code>. What does this function do? It computes the marginal frequencies i.e.&nbsp;the sum of the rows or columns. It takes a <strong>table</strong> object as input. The <code>margin</code> argument allows us to specify whether we want the sum of rows or columns. <code>1</code> indicates rows and <code>2</code> indicates columns.
</p>
<pre class="r"><code>margin.table(tab2, 1) # sum of rows</code></pre>
<pre><code>## 
## female   male   &lt;NA&gt; 
##  40565  61617 142216</code></pre>
<pre class="r"><code>margin.table(tab2, 2) # sum of columns</code></pre>
<pre><code>## 
## Desktop  Mobile  Tablet 
##  177282   63482    3634</code></pre>
<p>
If the margin argument is NULL (which it is by default), the function returns the sum of all cells of the table.
</p>
<pre class="r"><code>margin.table(tab2)</code></pre>
<pre><code>## [1] 244398</code></pre>
<p>
<code>table()</code> does not display row or column labels. It does display the group labels though. Let us revisit the output from <code>tab2</code>. You can observe that while it includes the group labels, the row and column labels are missing. The output from the <code>dimnames()</code> function shows the group labels of the variables but the row &amp; column labels are absent.
</p>
<pre class="r"><code>dimnames(tab2)</code></pre>
<pre><code>## [[1]]
## [1] "female" "male"   NA      
## 
## [[2]]
## [1] "Desktop" "Mobile"  "Tablet"</code></pre>
<pre class="r"><code>names(tab2)</code></pre>
<pre><code>## NULL</code></pre>
<pre class="r"><code>names(dimnames(tab2)) </code></pre>
<pre><code>## [1] "" ""</code></pre>
<p>
The output from <code>names(dimnames(tab2))</code> is also empty. Let us add the variable names as the row &amp; column labels to <code>tab2</code>.
</p>
<pre class="r"><code>names(dimnames(tab2)) &lt;- c("Gender", "Device")
tab2</code></pre>
<pre><code>##         Device
## Gender   Desktop Mobile Tablet
##   female   32803   7268    494
##   male     46418  14503    696
##   &lt;NA&gt;     98061  41711   2444</code></pre>
<p>
Now look at the output from <code>tab2</code> and you can observe the difference. The same is also visible when we run <code>dimnames(tab2)</code>.
</p>
<pre class="r"><code>dimnames(tab2)</code></pre>
<pre><code>## $Gender
## [1] "female" "male"   NA      
## 
## $Device
## [1] "Desktop" "Mobile"  "Tablet"</code></pre>
<p>
To add margin totals to the table, use <code>addmargins()</code>. Like <code>proportions()</code> and <code>margin.table()</code>, it also takes a <code>table</code> object as the input.
</p>
<pre class="r"><code>addmargins(tab2)</code></pre>
<pre><code>##         Device
## Gender   Desktop Mobile Tablet    Sum
##   female   32803   7268    494  40565
##   male     46418  14503    696  61617
##   &lt;NA&gt;     98061  41711   2444 142216
##   Sum     177282  63482   3634 244398</code></pre>
<p>
<code>rowSums()</code> returns the row total while <code>colSums()</code> returns the column total. They are similar to <code>margin.table()</code>.
</p>
<pre class="r"><code>rowSums(tab2)</code></pre>
<pre><code>## female   male   &lt;NA&gt; 
##  40565  61617 142216</code></pre>
<pre class="r"><code>colSums(tab2)</code></pre>
<pre><code>## Desktop  Mobile  Tablet 
##  177282   63482    3634</code></pre>
<p>
<code>xtabs()</code> is another way of creating multidimensional tables in R. In comparison to <code>table()</code>, it
</p>
<ul>
<li>
uses <code>formula</code> notation for input
</li>
<li>
the data argument ensures variable names are referenced instead of using $ i.e.&nbsp;<code>data<img src="https://latex.codecogs.com/png.latex?variable%3C/code%3E%3C/li%3E%0A%3Cli%3Edisplays%20row%20&amp;amp;%20column%20labels%20by%20default%3C/li%3E%0A%3C/ul%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Etabx%20&amp;lt;-%20xtabs(~gender+device,%20data%20=%20data)%0Atabx%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%2032803%20%20%207268%20%20%20%20494%0A##%20%20%20male%20%20%20%20%2046418%20%2014503%20%20%20%20696%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%2098061%20%2041711%20%20%202444%3C/code%3E%3C/pre%3E%0A%3Cp%3EThe%20following%20functions%20work%20with%20%3Ccode%3Extabs()%3C/code%3E%20as%20well%3C/p%3E%0A%3Cul%3E%0A%3Cli%3E%3Ccode%3Eproportions()%3C/code%3E%3C/li%3E%0A%3Cli%3E%3Ccode%3Emargin.table()%3C/code%3E%3C/li%3E%0A%3Cli%3E%3Ccode%3Eaddmargins()%3C/code%3E%3C/li%3E%0A%3C/ul%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Eproportions(tabx)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20%20%20%20%20Desktop%20%20%20%20%20%20Mobile%20%20%20%20%20%20Tablet%0A##%20%20%20female%200.134219593%200.029738378%200.002021293%0A##%20%20%20male%20%20%200.189927904%200.059341729%200.002847814%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%200.401234871%200.170668336%200.010000082%3C/code%3E%3C/pre%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Emargin.table(tabx,%201)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20gender%0A##%20female%20%20%20male%20%20%20&amp;lt;NA&amp;gt;%0A##%20%2040565%20%2061617%20142216%3C/code%3E%3C/pre%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Emargin.table(tabx,%202)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20device%0A##%20Desktop%20%20Mobile%20%20Tablet%0A##%20%20177282%20%20%2063482%20%20%20%203634%3C/code%3E%3C/pre%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Eaddmargins(tabx)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%20%20%20%20Sum%0A##%20%20%20female%20%20%2032803%20%20%207268%20%20%20%20494%20%2040565%0A##%20%20%20male%20%20%20%20%2046418%20%2014503%20%20%20%20696%20%2061617%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%2098061%20%2041711%20%20%202444%20142216%0A##%20%20%20Sum%20%20%20%20%20177282%20%2063482%20%20%203634%20244398%3C/code%3E%3C/pre%3E%0A%3Cp%3ESo%20far,%20we%20have%20been%20working%20with%20one%20or%20two%20dimensional%20tables.%20Both%20the%20%3Ccode%3Etable()%3C/code%3E%20and%20%3Ccode%3Extabs()%3C/code%3E%20functions%20are%20capable%20of%20creating%20multidimensional%20tables.%20Keep%20in%20mind%20that%20multidimensional%20tables%20are%20complex%20and%20it%20becomes%20increasingly%20difficult%20to%20understand%20or%20interpret%20them.%3C/p%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Etab3%20&amp;lt;-%20xtabs(~gender+device+channel,%20data%20=%20data)%0Atab3%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20,%20,%20channel%20=%20(Other)%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%20%20786%20%20%20%20258%20%20%20%20%20%200%0A##%20%20%20male%20%20%20%20%20%201063%20%20%20%20507%20%20%20%20%2019%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%20%202173%20%20%201186%20%20%20%20%2081%0A##%0A##%20,%20,%20channel%20=%20Affiliates%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%201314%20%20%20%20%2060%20%20%20%20%20%200%0A##%20%20%20male%20%20%20%20%20%201714%20%20%20%20169%20%20%20%20%20%200%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%20%203518%20%20%20%20548%20%20%20%20%2065%0A##%0A##%20,%20,%20channel%20=%20Direct%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%204785%20%20%20%20977%20%20%20%20%2059%0A##%20%20%20male%20%20%20%20%20%207010%20%20%202381%20%20%20%20%2095%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%2015824%20%20%208292%20%20%20%20430%0A##%0A##%20,%20,%20channel%20=%20Display%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%20%20123%20%20%20%20753%20%20%20%20104%0A##%20%20%20male%20%20%20%20%20%20%20210%20%20%20%20491%20%20%20%20%2073%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%20%20%20554%20%20%20%20911%20%20%20%20156%0A##%0A##%20,%20,%20channel%20=%20Organic%20Search%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%2017109%20%20%204480%20%20%20%20282%0A##%20%20%20male%20%20%20%20%2025016%20%20%209563%20%20%20%20448%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%2054071%20%2027223%20%20%201476%0A##%0A##%20,%20,%20channel%20=%20Paid%20Search%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%20%20645%20%20%20%20230%20%20%20%20%2022%0A##%20%20%20male%20%20%20%20%20%20%20887%20%20%20%20478%20%20%20%20%2026%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%20%201274%20%20%20%20782%20%20%20%20%2051%0A##%0A##%20,%20,%20channel%20=%20Referral%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%207387%20%20%20%20%2074%20%20%20%20%20%200%0A##%20%20%20male%20%20%20%20%20%209251%20%20%20%20185%20%20%20%20%20%200%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%2018052%20%20%20%20615%20%20%20%20%2051%0A##%0A##%20,%20,%20channel%20=%20Social%0A##%0A##%20%20%20%20%20%20%20%20%20device%0A##%20gender%20%20%20Desktop%20Mobile%20Tablet%0A##%20%20%20female%20%20%20%20%20654%20%20%20%20436%20%20%20%20%2027%0A##%20%20%20male%20%20%20%20%20%201267%20%20%20%20729%20%20%20%20%2035%0A##%20%20%20&amp;lt;NA&amp;gt;%20%20%20%20%20%202595%20%20%202154%20%20%20%20134%3C/code%3E%3C/pre%3E%0A%3Cp%3E%3Cstrong%3Eftable%3C/strong%3E%20stands%20for%20flat%20tables%20and%20is%20useful%20for%20printing%20attractive%20tables.%20It%20makes%20it%20easy%20to%20read%20and%20interpret%20multidimensional%20tables.%20In%20the%20next%20example,%20we%20will%20use%20%3Ccode%3Eftable()%3C/code%3E%20to%20print%20the%20tables%20we%20have%20created%20in%20the%20previous%20examples%20and%20compare%20the%20outputs.%3C/p%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Eftable(tabx)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20device%20Desktop%20Mobile%20Tablet%0A##%20gender%0A##%20female%20%20%20%20%20%20%20%20%20%2032803%20%20%207268%20%20%20%20494%0A##%20male%20%20%20%20%20%20%20%20%20%20%20%2046418%20%2014503%20%20%20%20696%0A##%20NA%20%20%20%20%20%20%20%20%20%20%20%20%20%2098061%20%2041711%20%20%202444%3C/code%3E%3C/pre%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Eftable(tab2)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20Device%20Desktop%20Mobile%20Tablet%0A##%20Gender%0A##%20female%20%20%20%20%20%20%20%20%20%2032803%20%20%207268%20%20%20%20494%0A##%20male%20%20%20%20%20%20%20%20%20%20%20%2046418%20%2014503%20%20%20%20696%0A##%20NA%20%20%20%20%20%20%20%20%20%20%20%20%20%2098061%20%2041711%20%20%202444%3C/code%3E%3C/pre%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Eftable(tab3)%3C/code%3E%3C/pre%3E%0A%3Cpre%3E%3Ccode%3E##%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20channel%20(Other)%20Affiliates%20Direct%20Display%20Organic%20Search%20Paid%20Search%20Referral%20Social%0A##%20gender%20device%0A##%20female%20Desktop%20%20%20%20%20%20%20%20%20%20%20%20%20786%20%20%20%20%20%20%201314%20%20%204785%20%20%20%20%20123%20%20%20%20%20%20%20%20%20%2017109%20%20%20%20%20%20%20%20%20645%20%20%20%20%207387%20%20%20%20654%0A##%20%20%20%20%20%20%20%20Mobile%20%20%20%20%20%20%20%20%20%20%20%20%20%20258%20%20%20%20%20%20%20%20%2060%20%20%20%20977%20%20%20%20%20753%20%20%20%20%20%20%20%20%20%20%204480%20%20%20%20%20%20%20%20%20230%20%20%20%20%20%20%2074%20%20%20%20436%0A##%20%20%20%20%20%20%20%20Tablet%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%200%20%20%20%20%20%20%20%20%20%200%20%20%20%20%2059%20%20%20%20%20104%20%20%20%20%20%20%20%20%20%20%20%20282%20%20%20%20%20%20%20%20%20%2022%20%20%20%20%20%20%20%200%20%20%20%20%2027%0A##%20male%20%20%20Desktop%20%20%20%20%20%20%20%20%20%20%20%201063%20%20%20%20%20%20%201714%20%20%207010%20%20%20%20%20210%20%20%20%20%20%20%20%20%20%2025016%20%20%20%20%20%20%20%20%20887%20%20%20%20%209251%20%20%201267%0A##%20%20%20%20%20%20%20%20Mobile%20%20%20%20%20%20%20%20%20%20%20%20%20%20507%20%20%20%20%20%20%20%20169%20%20%202381%20%20%20%20%20491%20%20%20%20%20%20%20%20%20%20%209563%20%20%20%20%20%20%20%20%20478%20%20%20%20%20%20185%20%20%20%20729%0A##%20%20%20%20%20%20%20%20Tablet%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2019%20%20%20%20%20%20%20%20%20%200%20%20%20%20%2095%20%20%20%20%20%2073%20%20%20%20%20%20%20%20%20%20%20%20448%20%20%20%20%20%20%20%20%20%2026%20%20%20%20%20%20%20%200%20%20%20%20%2035%0A##%20NA%20%20%20%20%20Desktop%20%20%20%20%20%20%20%20%20%20%20%202173%20%20%20%20%20%20%203518%20%2015824%20%20%20%20%20554%20%20%20%20%20%20%20%20%20%2054071%20%20%20%20%20%20%20%201274%20%20%20%2018052%20%20%202595%0A##%20%20%20%20%20%20%20%20Mobile%20%20%20%20%20%20%20%20%20%20%20%20%201186%20%20%20%20%20%20%20%20548%20%20%208292%20%20%20%20%20911%20%20%20%20%20%20%20%20%20%2027223%20%20%20%20%20%20%20%20%20782%20%20%20%20%20%20615%20%20%202154%0A##%20%20%20%20%20%20%20%20Tablet%20%20%20%20%20%20%20%20%20%20%20%20%20%20%2081%20%20%20%20%20%20%20%20%2065%20%20%20%20430%20%20%20%20%20156%20%20%20%20%20%20%20%20%20%20%201476%20%20%20%20%20%20%20%20%20%2051%20%20%20%20%20%20%2051%20%20%20%20134%3C/code%3E%3C/pre%3E%0A%3Cp%3EBy%20default,%20missing%20values%20(NAs)%20are%20excluded%20from%20tables.%20Let%20us%20modify%20the%20gender%20data%20from%20our%20case%20study%20a%20bit%20and%20see%20how%20the%20%3Ccode%3Etable()%3C/code%3E%20function%20deals%20with%20missing%20values.%20We%20won%E2%80%99t%20explicitly%20specify%20%3Ccode%3ENA%3C/code%3E%20as%20a%20level%20while%20recreating%20the%20gender%20data.%3C/p%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Egen%20&amp;lt;-%20as.factor(as.character(data">gender)) table(gen)</code>

<pre><code>## gen
## female   male 
##  40565  61617</code></pre>
<p>
As you can see, <code>table()</code> excludes missing values while tabulating the data. In order to ensure that missing values are also counted, we can use the <code>useNA</code> argument. It can take two values:
</p>
<ul>
<li>
ifany
</li>
<li>
always
</li>
</ul>
<p>
In the first case, it will show <code>NA</code> as a level and the count only if there are missing values in the data. In the second case, it will always show <code>NA</code> as a level irrespective of whether there are missing values in the data or not.
</p>
<pre class="r"><code>table(gen, useNA = "ifany")</code></pre>
<pre><code>## gen
## female   male   &lt;NA&gt; 
##  40565  61617 142216</code></pre>
<pre class="r"><code>table(data$device, useNA = "always")</code></pre>
<pre><code>## 
## Desktop  Mobile  Tablet    &lt;NA&gt; 
##  177282   63482    3634       0</code></pre>
<p>
In this final section on tables, we will learn how to select/access the different parts of a <code>table</code>. We will use <code>[</code> operator to select rows and columns of a table (it is similar to selecting data from a <code>data.frame</code>). Below are a few examples:
</p>
<ul>
<li>
select first row
</li>
</ul>
<pre class="r"><code>tab2[1, ]           </code></pre>
<pre><code>## Desktop  Mobile  Tablet 
##   32803    7268     494</code></pre>
<ul>
<li>
select first column
</li>
</ul>
<pre class="r"><code>tab2[, 1]           </code></pre>
<pre><code>## female   male   &lt;NA&gt; 
##  32803  46418  98061</code></pre>
<ul>
<li>
select first two rows
</li>
</ul>
<pre class="r"><code>tab2[1:2, ]         </code></pre>
<pre><code>##         Device
## Gender   Desktop Mobile Tablet
##   female   32803   7268    494
##   male     46418  14503    696</code></pre>
<ul>
<li>
select first two columns
</li>
</ul>
<pre class="r"><code>tab2[, 1:2]        </code></pre>
<pre><code>##         Device
## Gender   Desktop Mobile
##   female   32803   7268
##   male     46418  14503
##   &lt;NA&gt;     98061  41711</code></pre>
<ul>
<li>
select nth row
</li>
</ul>
<pre class="r"><code>tab2[2, ]          </code></pre>
<pre><code>## Desktop  Mobile  Tablet 
##   46418   14503     696</code></pre>
<ul>
<li>
select nth column
</li>
</ul>
<pre class="r"><code>tab2[, 2]          </code></pre>
<pre><code>## female   male   &lt;NA&gt; 
##   7268  14503  41711</code></pre>
<ul>
<li>
select row by group label
</li>
</ul>
<pre class="r"><code>tab2["female", ]   </code></pre>
<pre><code>## Desktop  Mobile  Tablet 
##   32803    7268     494</code></pre>
<ul>
<li>
select column by group label
</li>
</ul>
<pre class="r"><code>tab2[, "Mobile"]   </code></pre>
<pre><code>## female   male   &lt;NA&gt; 
##   7268  14503  41711</code></pre>
<p>
Before we end this section, let us learn how to test if an object is of class table using <code>is.table()</code>.
</p>
<pre class="r"><code>is.table(tab2)</code></pre>
<pre><code>## [1] TRUE</code></pre>
<p>
Next, we will look at different R packages for two way/contingency tables.
</p>
<section id="contingency-table" class="level3">
<h3 class="anchored" data-anchor-id="contingency-table">
Contingency Table
</h3>
<p>
For cross tables with output similar to SAS or SPSS, use any of the below:
</p>
<ul>
<li>
<code>CrossTable()</code> from the <a href="cran.r-project.org/package=gmodels">gmodels</a> package
</li>
<li>
<code>ds_cross_table()</code> from the <a href="https://descriptr.rsquaredacademy.com">descriptr</a> package
</li>
</ul>
<pre class="r"><code>gmodels::CrossTable(data$device, data$gender)</code></pre>
<pre><code>## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## | Chi-square contribution |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  102182 
## 
##  
##              | data$gender 
##  data$device |    female |      male | Row Total | 
## -------------|-----------|-----------|-----------|
##      Desktop |     32803 |     46418 |     79221 | 
##              |    58.228 |    38.334 |           | 
##              |     0.414 |     0.586 |     0.775 | 
##              |     0.809 |     0.753 |           | 
##              |     0.321 |     0.454 |           | 
## -------------|-----------|-----------|-----------|
##       Mobile |      7268 |     14503 |     21771 | 
##              |   218.694 |   143.975 |           | 
##              |     0.334 |     0.666 |     0.213 | 
##              |     0.179 |     0.235 |           | 
##              |     0.071 |     0.142 |           | 
## -------------|-----------|-----------|-----------|
##       Tablet |       494 |       696 |      1190 | 
##              |     0.986 |     0.649 |           | 
##              |     0.415 |     0.585 |     0.012 | 
##              |     0.012 |     0.011 |           | 
##              |     0.005 |     0.007 |           | 
## -------------|-----------|-----------|-----------|
## Column Total |     40565 |     61617 |    102182 | 
##              |     0.397 |     0.603 |           | 
## -------------|-----------|-----------|-----------|
## 
## </code></pre>
<pre class="r"><code>descriptr::ds_cross_table(data, device, gender)</code></pre>
<pre><code>##     Cell Contents
##  |---------------|
##  |     Frequency |
##  |       Percent |
##  |       Row Pct |
##  |       Col Pct |
##  |---------------|
## 
##  Total Observations:  244398 
## 
## ----------------------------------------------------------------------------
## |              |                          gender                           |
## ----------------------------------------------------------------------------
## |       device |       female |         male |           NA |    Row Total |
## ----------------------------------------------------------------------------
## |      Desktop |        32803 |        46418 |        98061 |       177282 |
## |              |        0.134 |         0.19 |        0.401 |              |
## |              |         0.19 |         0.26 |         0.55 |         0.73 |
## |              |         0.81 |         0.75 |         0.69 |              |
## ----------------------------------------------------------------------------
## |       Mobile |         7268 |        14503 |        41711 |        63482 |
## |              |         0.03 |        0.059 |        0.171 |              |
## |              |         0.11 |         0.23 |         0.66 |         0.26 |
## |              |         0.18 |         0.24 |         0.29 |              |
## ----------------------------------------------------------------------------
## |       Tablet |          494 |          696 |         2444 |         3634 |
## |              |        0.002 |        0.003 |         0.01 |              |
## |              |         0.14 |         0.19 |         0.67 |         0.01 |
## |              |         0.01 |         0.01 |         0.02 |              |
## ----------------------------------------------------------------------------
## | Column Total |        40565 |        61617 |       142216 |       244398 |
## |              |        0.166 |        0.252 |        0.582 |              |
## ----------------------------------------------------------------------------</code></pre>
<p>
We list and explore different R packages for summarizing categorical data in our GitHub repository.
</p>
</section>
<section id="key-functions" class="level3">
<h3 class="anchored" data-anchor-id="key-functions">
Key Functions
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_module_4_functions.png" width="70%" style="display: block; margin: auto;">
</p>
</section>
</li></ul></section> ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-2/</guid>
  <pubDate>Wed, 12 Jan 2022 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/forcats-part-2.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Handling Categorical Data in R - Part 1</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-1/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2022-01-07-handling-categorical-data-in-r-part-1.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<script src="../../rmarkdown-libs/kePrint/kePrint.js"></script>
<p><link href="../../rmarkdown-libs/lightable/lightable.css" rel="stylesheet"></p>
<p>
<img src="https://blog.rsquaredacademy.com/img/forcats-part-1.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
This is part 1 of a series on “Handling Categorical Data in R.” Almost every data science project involves working with categorical data, and we should know how to <strong>read</strong>, <strong>store</strong>, <strong>summarize</strong>, <strong>reshape</strong> &amp; <strong>visualize</strong> such data. Working with categorical data is different from working with other data types such as numbers or text. In this article, we will understand what categorical data is, how R stores it using <strong>factor</strong>, and explore the rich set of functions (built-in &amp; through packages) provided by R for working with such data. Throughout the series, we will also <strong>work through a case study</strong> to better understand the concepts we learn. Happy learning!
</p>
<p>
Below are the links to the other articles of this series:
</p>
<ul>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-2/">Part 2 - Summarize Categorical Data</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-3/">Part 3 - Reshape Categorical Data</a>
</li>
<li>
<a href="https://blog.rsquaredacademy.com/handling-categorical-data-in-r-part-4/">Part 4 - Visualize Categorical Data</a>
</li>
</ul>
<section id="table-of-contents" class="level2">
<h2 class="anchored" data-anchor-id="table-of-contents">
Table of Contents
</h2>
<ul>
<li>
Resources
</li>
<li>
Introduction
</li>
<li>
Case Study
</li>
<li>
Factors
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
You can download all the data sets, R scripts, practice questions and their solutions from our <a href="https://github.com/rsquaredacademy-education/online-courses/">GitHub</a> repository.
</p>
</section>
<section id="intro" class="level2">
<h2 class="anchored" data-anchor-id="intro">
Introduction
</h2>
<p>
Before we begin our deep dive on categorical data, let us get a quick overview of different data types. Feel free to skip this section if you know the difference between nominal and ordinal data.
</p>
<section id="data-types" class="level3">
<h3 class="anchored" data-anchor-id="data-types">
Data Types
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_data_types.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
In the chart above, we can see that data can be primarily classified into <strong>qualitative</strong> or <strong>quantitative</strong>. (<strong>The word categorical is used interchangeably with qualitative throughout the series</strong>). Qualitative data consists of labels or names. Quantitative data, on the other hand, consists of numbers and indicate how much or how many. This brings us to the next level of classification:
</p>
<ul>
<li>
discrete
</li>
<li>
continuous
</li>
</ul>
<p>
In the chart, we can observe that qualitative data is always <strong>discrete</strong> where as quantitative data may be <strong>discrete</strong> or <strong>continuous</strong>. Qualitative data is further classified into
</p>
<ul>
<li>
nominal
</li>
<li>
ordinal
</li>
</ul>
<p>
First, we will understand discrete and continuous data, and then proceed to explore nominal and ordinal data.
</p>
<p>
<strong>Discrete Data</strong>
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_discrete.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Discrete data arises in situations where counting is involved. It can take on only a finite number of values and cannot be divided into smaller parts. For example, let us consider the number of students in a class. We can have 5 0r 10 students but not 5.5 students (we can’t have half a student).
</p>
<p>
<strong>Continuous Data</strong>
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_measure.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Continuous data arises in situations where measuring is involved. It can take any numeric value in a specified range and can be divided into smaller parts and still have meaning. Examples include money, temperature, length, volume etc.
</p>
</section>
<section id="categorical-data" class="level3">
<h3 class="anchored" data-anchor-id="categorical-data">
Categorical Data
</h3>
<p>
Since our interest is in categorical data, we will spend more time understanding the different types of categorical data through various examples. Let us begin by formally defining categorical data:
</p>
<ul>
<li>
it is always discrete
</li>
<li>
it may be divided into groups
</li>
<li>
consists of names or labels
</li>
<li>
takes on limited &amp; fixed number of possible values
</li>
<li>
arises in situation when counting is involved
</li>
<li>
analysis generally involves the use of data tables
</li>
</ul>
</section>
<section id="dichotomous" class="level3">
<h3 class="anchored" data-anchor-id="dichotomous">
Dichotomous
</h3>
<p>
A categorical variable that can take on exactly two values is termed as <strong>binary</strong> or <strong>dichotomous</strong> variable.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_yes_no.png" width="80%" style="display: block; margin: auto;">
</p>
</section>
<section id="polychotomous" class="level3">
<h3 class="anchored" data-anchor-id="polychotomous">
Polychotomous
</h3>
<p>
Categorical variables with more than two possible values are called <strong>polychotomous</strong> variables.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_yes_no_maybe.png" width="80%" style="display: block; margin: auto;">
</p>
</section>
<section id="ordinal" class="level3">
<h3 class="anchored" data-anchor-id="ordinal">
Ordinal
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_satisfaction_rating.jpg" width="80%" style="display: block; margin: auto;">
</p>
<p>
In ordinal data, the categories can be ordered or ranked. Examples include
</p>
<ul>
<li>
socio-economic status
</li>
<li>
education level
</li>
<li>
income level
</li>
<li>
satisfaction rating
</li>
</ul>
<p>
While we can rank the categories, we cannot assign a value to them. For example, in satisfaction ranking, we cannot say that like is twice as positive as dislike i.e.&nbsp;we are unable to say how much they differ from each other. While the order or rank of data is meaningful, the difference between two pieces of data cannot be measured/determined or are meaningless. Ordinal data provide information about relative comparisons, but not the magnitude of the differences.
</p>
</section>
<section id="nominal" class="level3">
<h3 class="anchored" data-anchor-id="nominal">
Nominal
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_transportation_mode.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Nominal data do not have an intrinsic order and cannot be ordered or measured. Examples include
</p>
<ul>
<li>
blood group
</li>
<li>
gender
</li>
<li>
religion
</li>
<li>
color
</li>
</ul>
<p>
Categorical data are sometimes coded with numbers, with those numbers replacing names. Although such numbers might appear to be quantitative, they are actually categorical data. When they do take numerical values, those numbers do not have any mathematical meaning. Examples include months expressed in numbers.
</p>
</section>
</section>
<section id="casestudy" class="level2">
<h2 class="anchored" data-anchor-id="casestudy">
Case Study
</h2>
<p>
As is the practice, throughout this series, we will work on a case study related to an e-commerce firm. As most of you would already be aware, a lot of data is captured when you go on the internet by the websites you browse as well as by third party cookies. Data collected is then used to display ads as well as to feed to recommendation algorithms.
</p>
<p>
The data used in the case study represents the basic information that is captured when users visit any website. It closely resembles real world data for an e-commerce store. We will try to generate insights about the visitors to be used by an imaginary marketing team for better targeting and promotion. The case study data set can be imported using the RStudio IDE or R code.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_web_analytics.png" width="80%" style="display: block; margin: auto;">
</p>
<section id="data" class="level3">
<h3 class="anchored" data-anchor-id="data">
Data
</h3>
<p>
The data set is available in both <strong>CSV</strong> &amp; <strong>RDS</strong> formats.
</p>
<p>
<strong>CSV</strong>
</p>
<p>
If you want to specify the data types while reading the data, use the <a href="https://readr.tidyverse.org/">readr</a> package. We have explored how to import data into R in a previous <a href="https://blog.rsquaredacademy.com/import-data-into-r-part-1/">article</a>. We will read a subset of columns from the data set (it has 20 columns) which will cover both nominal and ordinal data types. To import the data, we will use the <code>read_csv()</code> function. The first input is the name of the data set, <code>analytics.csv</code>. Ensure that the name is enclosed in single/double quotes.
</p>
<pre class="r"><code>read_csv("analytics_raw.csv", 
         col_types = cols_only(device = col_factor(levels = c("Desktop", "Tablet", "Mobile")), 
                               gender = col_factor(levels = c("female", "male", "NA")), 
                               user_rating = col_factor(levels = c("1", "2", "3", "4", "5"),
                                                        ordered = TRUE)))</code></pre>
<pre><code>## New names:
## * `` -&gt; ...1</code></pre>
<pre><code>## # A tibble: 244,398 x 3
##    device  gender user_rating
##    &lt;fct&gt;   &lt;fct&gt;  &lt;ord&gt;      
##  1 Desktop female 4          
##  2 Mobile  NA     5          
##  3 Desktop NA     4          
##  4 Desktop NA     5          
##  5 Desktop NA     4          
##  6 Mobile  NA     4          
##  7 Desktop NA     4          
##  8 Desktop NA     4          
##  9 Desktop female 5          
## 10 Desktop NA     4          
## # ... with 244,388 more rows</code></pre>
<p>
Since we are specifying the column data types while importing the data, we will use the <code>col_types</code> argument to list out the data types. As we are reading in a subset of the columns and not all of them, we will use the <code>cols_only()</code> function indicating that only the columns specified must be read in and not all of them.
</p>
<p>
Categorical data and the levels/groups are specified using the <code>col_factor()</code> function. Use the <code>levels</code> argument to specify the levels/groups and the <code>ordered</code> argument to indicate if the data is ordinal. By default, it is set to <code>FALSE</code>, change this to <code>TRUE</code> if the column is ordinal.
</p>
<p>
<strong>RDS</strong>
</p>
<p>
The <code>.rds</code> file can be read using <code>readRDS()</code>.
</p>
<pre class="r"><code>data &lt;- readRDS('analytics.rds')
head(data)</code></pre>
<pre><code>## # A tibble: 6 x 19
##   device  os        browser user_type channel gender frequency recency page_depth
##   &lt;fct&gt;   &lt;fct&gt;     &lt;fct&gt;   &lt;fct&gt;     &lt;fct&gt;   &lt;fct&gt;      &lt;dbl&gt;   &lt;dbl&gt;      &lt;dbl&gt;
## 1 Desktop Windows   Chrome  New Visi~ Organi~ female         1       0          1
## 2 Mobile  iOS       Safari  Returnin~ Organi~ &lt;NA&gt;           3       1          1
## 3 Desktop Chrome OS Chrome  New Visi~ Direct  &lt;NA&gt;           1       0          5
## 4 Desktop Macintosh Chrome  Returnin~ Organi~ &lt;NA&gt;           2       0          1
## 5 Desktop Macintosh Chrome  Returnin~ Referr~ &lt;NA&gt;           5       8          1
## 6 Mobile  Android   Chrome  New Visi~ Organi~ &lt;NA&gt;           1       0          5
## # ... with 10 more variables: hour_of_day &lt;chr&gt;, age &lt;dbl&gt;, duration &lt;dbl&gt;,
## #   landing_page &lt;fct&gt;, exit_page &lt;fct&gt;, country &lt;fct&gt;, quantity &lt;dbl&gt;,
## #   revenue &lt;dbl&gt;, purchase_flag &lt;lgl&gt;, user_rating &lt;dbl&gt;</code></pre>
</section>
<section id="data-dictionary" class="level3">
<h3 class="anchored" data-anchor-id="data-dictionary">
Data Dictionary
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Column
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
device
</td>
<td style="text-align:left;">
Device used to browse the website
</td>
</tr>
<tr>
<td style="text-align:left;">
os
</td>
<td style="text-align:left;">
Operating system of the device
</td>
</tr>
<tr>
<td style="text-align:left;">
browser
</td>
<td style="text-align:left;">
Browser used to visit the website
</td>
</tr>
<tr>
<td style="text-align:left;">
user_type
</td>
<td style="text-align:left;">
New or returning visitor
</td>
</tr>
<tr>
<td style="text-align:left;">
channel
</td>
<td style="text-align:left;">
Source of traffic
</td>
</tr>
<tr>
<td style="text-align:left;">
gender
</td>
<td style="text-align:left;">
Gender of the visitor
</td>
</tr>
<tr>
<td style="text-align:left;">
frequency
</td>
<td style="text-align:left;">
Count of visits to the website
</td>
</tr>
<tr>
<td style="text-align:left;">
recency
</td>
<td style="text-align:left;">
Number of days since last visit
</td>
</tr>
<tr>
<td style="text-align:left;">
page_depth
</td>
<td style="text-align:left;">
Number of website pages browsed
</td>
</tr>
<tr>
<td style="text-align:left;">
hour_of_day
</td>
<td style="text-align:left;">
Hour of day
</td>
</tr>
<tr>
<td style="text-align:left;">
age
</td>
<td style="text-align:left;">
Age of the visitor
</td>
</tr>
<tr>
<td style="text-align:left;">
duration
</td>
<td style="text-align:left;">
Time spent on the website (in seconds)
</td>
</tr>
<tr>
<td style="text-align:left;">
landing_page
</td>
<td style="text-align:left;">
Page on which visitor landed
</td>
</tr>
<tr>
<td style="text-align:left;">
exit_page
</td>
<td style="text-align:left;">
Page on which visitor exited
</td>
</tr>
<tr>
<td style="text-align:left;">
country
</td>
<td style="text-align:left;">
Country of origin
</td>
</tr>
<tr>
<td style="text-align:left;">
city
</td>
<td style="text-align:left;">
City of the visitor
</td>
</tr>
<tr>
<td style="text-align:left;">
quantity
</td>
<td style="text-align:left;">
Number of units purchased
</td>
</tr>
<tr>
<td style="text-align:left;">
revenue
</td>
<td style="text-align:left;">
Total revenue
</td>
</tr>
<tr>
<td style="text-align:left;">
purchase_flag
</td>
<td style="text-align:left;">
Whether the visitor checked out?
</td>
</tr>
<tr>
<td style="text-align:left;">
user_rating
</td>
<td style="text-align:left;">
Website UI rating given by visitor
</td>
</tr>
</tbody>
</table>
<p>
Now that we have an overview of the case study, let us move on to the next section where we explore how R stores categorical data using factors.
</p>
</section>
</section>
<section id="factors" class="level2">
<h2 class="anchored" data-anchor-id="factors">
Factors
</h2>
<p>
In this very important section, we will learn how R
</p>
<ul>
<li>
stores categorical data
</li>
<li>
checks if given data is categorical
</li>
<li>
converts other data types to factor
</li>
<li>
handles missing values in categorical data
</li>
<li>
specifies the orders of the categories/levels
</li>
<li>
stores ordinal data
</li>
</ul>
<p>
In R, categorical data is stored as <code>factor</code>. Before we explore the <code>factor</code> family of functions, let us generate the sample data we will use in this module. We will generate the <code>device</code> column from the case study data set using the <code>sample()</code> function. We provide the following inputs to generate the data:
</p>
<ul>
<li>
values from which the data must be generated
</li>
<li>
the size of the sample
</li>
<li>
indicate if the values must be repeated (TRUE/FALSE)
</li>
</ul>
<pre class="r"><code>device &lt;- sample(c("Desktop", "Mobile", "Tablet"), size = 25, replace = TRUE)
device</code></pre>
<pre><code>##  [1] "Mobile"  "Mobile"  "Tablet"  "Mobile"  "Mobile"  "Mobile"  "Tablet" 
##  [8] "Tablet"  "Desktop" "Mobile"  "Mobile"  "Mobile"  "Tablet"  "Mobile" 
## [15] "Desktop" "Tablet"  "Desktop" "Mobile"  "Mobile"  "Tablet"  "Desktop"
## [22] "Mobile"  "Tablet"  "Mobile"  "Mobile"</code></pre>
<section id="membership-testing" class="level3">
<h3 class="anchored" data-anchor-id="membership-testing">
Membership Testing
</h3>
<p>
Great! We have successfully generated the sample data and along the way learnt a new R function for sampling. First, let us check if the sample is a <code>factor</code> using the membership function <code>is.factor()</code>.
</p>
<pre class="r"><code>is.factor(device)</code></pre>
<pre><code>## [1] FALSE</code></pre>
<p>
Membership testing functions always have the prefix <code>is_</code> and return only logical values. If the object is a member of the specified class, they return <code>TRUE</code> else <code>FALSE</code>. Since our sample data is not stored as a factor, R has returned <code>FALSE</code>.
</p>
</section>
<section id="coercion" class="level3">
<h3 class="anchored" data-anchor-id="coercion">
Coercion
</h3>
<p>
Let us try to coerce it into <code>factor</code> using the coercion function <code>as.factor()</code>.
</p>
<pre class="r"><code>as.factor(device)</code></pre>
<pre><code>##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet</code></pre>
<p>
Do you spot any difference in the output? In the last line, it displays the levels or categories of the variable. Don’t worry if you didn’t spot it. We are just getting started and you will pick it up by the end of this section. Another function that can be used to coerce data into factor is <code>as_factor()</code> from the <a href="https://forcats.tidyverse.org/">forcats</a> package.
</p>
<pre class="r"><code>as_factor(device)</code></pre>
<pre><code>##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Mobile Tablet Desktop</code></pre>
<p>
Did you notice any difference between these two functions? Focus on the last line of the output where the levels are displayed. Now observe the order of the levels. <code>as.factor()</code> displays levels in the alphabetical order whereas <code>as_factor()</code> displays them in order of appearance in the data. <strong>Mobile</strong>, followed by <strong>Tablet</strong>, and then <strong>Desktop</strong>. If you look at the data, they appear in the same order.
</p>
</section>
<section id="factor-function" class="level3">
<h3 class="anchored" data-anchor-id="factor-function">
Factor Function
</h3>
<p>
If you want finer control while creating factors, use the <code>factor()</code> function. <code>as.factor()</code> should suffice in most cases but use <code>factor()</code> when you want to:
</p>
<ul>
<li>
specify levels
</li>
<li>
modify labels
</li>
<li>
include <code>NA</code> as a level/category
</li>
<li>
create ordered factors
</li>
<li>
specify order of levels
</li>
</ul>
<p>
The first input is a vector, usually a numeric or character vector with a small number of unique values. In our example, it is a character vector of length 25 (i.e.&nbsp;25 values) but 3 unique values.
</p>
<pre class="r"><code>factor(device)</code></pre>
<pre><code>##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet</code></pre>
<p>
If you want to specify the levels or categories, use the <code>levels</code> argument.
</p>
<pre class="r"><code>factor(device, levels = c("Desktop", "Mobile", "Tablet"))</code></pre>
<pre><code>##  [1] Mobile  Mobile  Tablet  Mobile  Mobile  Mobile  Tablet  Tablet  Desktop
## [10] Mobile  Mobile  Mobile  Tablet  Mobile  Desktop Tablet  Desktop Mobile 
## [19] Mobile  Tablet  Desktop Mobile  Tablet  Mobile  Mobile 
## Levels: Desktop Mobile Tablet</code></pre>
<p>
Levels not specified will be replaced by <code>NA</code>. Let us specify only <strong>Desktop</strong> and <strong>Mobile</strong> as the levels in the device column and see what happens.
</p>
<pre class="r"><code>factor(device, levels = c("Desktop", "Mobile"))</code></pre>
<pre><code>##  [1] Mobile  Mobile  &lt;NA&gt;    Mobile  Mobile  Mobile  &lt;NA&gt;    &lt;NA&gt;    Desktop
## [10] Mobile  Mobile  Mobile  &lt;NA&gt;    Mobile  Desktop &lt;NA&gt;    Desktop Mobile 
## [19] Mobile  &lt;NA&gt;    Desktop Mobile  &lt;NA&gt;    Mobile  Mobile 
## Levels: Desktop Mobile</code></pre>
<p>
As you can see, <strong>Tablet</strong> has been replaced by <code>NA</code>.
</p>
</section>
<section id="modify-labels" class="level3">
<h3 class="anchored" data-anchor-id="modify-labels">
Modify Labels
</h3>
<p>
You can change the labels of the levels using the <code>labels</code> argument. The labels must be in the same order as the levels. We will modify the labels to <strong>Desk</strong>, <strong>Mob</strong> &amp; <strong>Tab</strong> for <strong>Desktop</strong>, <strong>Mobile</strong> &amp; <strong>Tablet</strong> respectively.
</p>
<pre class="r"><code>factor(device, 
       levels = c("Desktop", "Mobile", "Tablet"),
       labels = c("Desk", "Mob", "Tab"))</code></pre>
<pre><code>##  [1] Mob  Mob  Tab  Mob  Mob  Mob  Tab  Tab  Desk Mob  Mob  Mob  Tab  Mob  Desk
## [16] Tab  Desk Mob  Mob  Tab  Desk Mob  Tab  Mob  Mob 
## Levels: Desk Mob Tab</code></pre>
<p>
You can see that not only the values but the levels are also modified.
</p>
</section>
<section id="missing-values" class="level3">
<h3 class="anchored" data-anchor-id="missing-values">
Missing Values
</h3>
<p>
Let us regenerate the device column but include some missing values (<code>NA</code>) deliberately to see how <code>factor()</code> handles them.
</p>
<pre class="r"><code># sample with missing values
device &lt;- sample(c("Desktop", "Mobile", "Tablet", NA), size = 25, replace = TRUE)
device</code></pre>
<pre><code>##  [1] "Desktop" "Tablet"  "Tablet"  NA        "Mobile"  NA        "Desktop"
##  [8] "Desktop" "Tablet"  "Mobile"  "Tablet"  "Desktop" "Desktop" "Mobile" 
## [15] NA        "Desktop" "Mobile"  "Tablet"  "Mobile"  "Mobile"  "Tablet" 
## [22] NA        "Desktop" "Tablet"  "Mobile"</code></pre>
<pre class="r"><code># store as categorical data
factor(device)</code></pre>
<pre><code>##  [1] Desktop Tablet  Tablet  &lt;NA&gt;    Mobile  &lt;NA&gt;    Desktop Desktop Tablet 
## [10] Mobile  Tablet  Desktop Desktop Mobile  &lt;NA&gt;    Desktop Mobile  Tablet 
## [19] Mobile  Mobile  Tablet  &lt;NA&gt;    Desktop Tablet  Mobile 
## Levels: Desktop Mobile Tablet</code></pre>
<p>
<code>NA</code> is not shown as one of the levels. Why does this happen? By default, it will ignore them. If you look at the arguments of the <code>factor()</code> function, the <code>exclude</code> argument is set to <code>NA</code> by default i.e.&nbsp;<code>NA</code> is excluded automatically. What should we do to ensure that <code>NA</code> is also treated as a level? In order to treat <code>NA</code> as a level, set the <code>exclude</code> argument to <code>NULL</code>.
</p>
<pre class="r"><code>factor(device, exclude = NULL)</code></pre>
<pre><code>##  [1] Desktop Tablet  Tablet  &lt;NA&gt;    Mobile  &lt;NA&gt;    Desktop Desktop Tablet 
## [10] Mobile  Tablet  Desktop Desktop Mobile  &lt;NA&gt;    Desktop Mobile  Tablet 
## [19] Mobile  Mobile  Tablet  &lt;NA&gt;    Desktop Tablet  Mobile 
## Levels: Desktop Mobile Tablet &lt;NA&gt;</code></pre>
<p>
As you can see, <code>NA</code> is displayed as one of the levels in the data.
</p>
</section>
<section id="ordered-factors" class="level3">
<h3 class="anchored" data-anchor-id="ordered-factors">
Ordered Factors
</h3>
<p>
So far, we have been looking at nominal data. Let us now explore how R handles ordered data. We will generate a new data set of satisfaction ratings to use in this section. Satisfaction ratings are widely used to measure a customer’s satisfaction with an organization, service or a product.
</p>
<pre class="r"><code>rating &lt;- sample(c("Dislike", "Neutral", "Like"), size = 25, replace = TRUE)
rating</code></pre>
<pre><code>##  [1] "Like"    "Dislike" "Neutral" "Dislike" "Neutral" "Like"    "Neutral"
##  [8] "Dislike" "Dislike" "Dislike" "Dislike" "Neutral" "Dislike" "Neutral"
## [15] "Dislike" "Dislike" "Like"    "Neutral" "Dislike" "Neutral" "Like"   
## [22] "Neutral" "Dislike" "Like"    "Dislike"</code></pre>
<p>
It consists of three values Dislike, Neutral &amp; Like in that order. You can see that there is an intrinsic order here. Like is better than neutral which in turn is better than dislike. While we can order them, we can’t quantify the difference between them. We can’t say neutral is so many times better than dislike.
</p>
<p>
<strong>Membership Testing</strong>
</p>
<p>
As we did earlier, let us check if the data is ordered using the membership function <code>is.ordered()</code>.
</p>
<pre class="r"><code>is.ordered(rating)</code></pre>
<pre><code>## [1] FALSE</code></pre>
<p>
R returns <code>FALSE</code> as the variable <code>rating</code> is not ordered. Let us use <code>as.ordered()</code> to coerce it into an ordered factor.
</p>
<pre class="r"><code>as.ordered(rating)</code></pre>
<pre><code>##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike &lt; Like &lt; Neutral</code></pre>
<p>
Look at the last line where the levels are displayed. In case of ordered factors, you will see a <code>&lt;</code> between the labels. This is used to indicate the order of the levels. Now <code>rating</code> is both an ordered but the order of the levels is not correct. It should be <code>Dislike &lt; Neutral &lt; Like</code> but is displayed in order of appearance in the data. Let us use the <code>factor()</code> function since we need more control over how the levels are ranked and set the <code>ordered</code> argument to <code>TRUE</code>.
</p>
<pre class="r"><code>factor(rating, ordered = TRUE)</code></pre>
<pre><code>##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike &lt; Like &lt; Neutral</code></pre>
<p>
The ranking of the levels has not changed and is still the same. Why is this happening? If you observe carefully, the ranking follows the alphabetical order (<strong>D</strong>esktop, <strong>M</strong>obile, <strong>T</strong>able). The <code>factor()</code> function uses the same order for the levels.
</p>
</section>
<section id="modify-order-of-levels" class="level3">
<h3 class="anchored" data-anchor-id="modify-order-of-levels">
Modify Order of Levels
</h3>
<p>
To change the order/ranking of the levels, we need to specify it using the <code>levels</code> argument. Let us do that in the next example.
</p>
<pre class="r"><code>factor(rating, levels = c("Dislike", "Neutral", "Like"), ordered = TRUE)</code></pre>
<pre><code>##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike &lt; Neutral &lt; Like</code></pre>
<p>
Now, you can see that the levels are ranked correctly. The <code>ordered()</code> function can also be used to create ordered factors. Let us recreate the previous example using the <code>ordered()</code> function.
</p>
<pre class="r"><code>ordered(rating, levels = c("Dislike", "Neutral", "Like"))</code></pre>
<pre><code>##  [1] Like    Dislike Neutral Dislike Neutral Like    Neutral Dislike Dislike
## [10] Dislike Dislike Neutral Dislike Neutral Dislike Dislike Like    Neutral
## [19] Dislike Neutral Like    Neutral Dislike Like    Dislike
## Levels: Dislike &lt; Neutral &lt; Like</code></pre>
<p>
You can specify levels, modify labels and handle missing values using the <code>ordered()</code> function as well.
</p>
</section>
<section id="key-functions" class="level3">
<h3 class="anchored" data-anchor-id="key-functions">
Key Functions
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/fct_module_3_functions.png" width="70%" style="display: block; margin: auto;">
</p>
</section>
<section id="summary" class="level3">
<h3 class="anchored" data-anchor-id="summary">
Summary
</h3>
<ul>
<li>
R uses <strong>factor</strong> to handle categorical data.
</li>
<li>
Use <code>as.factor()</code> or <code>as_factor()</code> to coerce other data types to <strong>factor</strong>.
</li>
<li>
Use <code>is.factor()</code> or <code>is.ordered()</code> to identify factor &amp; ordered factor respectively.
</li>
<li>
Use <code>factor()</code> to
<ul>
<li>
specify labels
</li>
<li>
modify labels
</li>
<li>
handle missing data
</li>
<li>
create ordered factors
</li>
<li>
specify order of levels
</li>
</ul>
</li>
<li>
Use <code>ordered()</code> to create ordered factors.
</li>
</ul>
</section>
<section id="your-turn" class="level3">
<h3 class="anchored" data-anchor-id="your-turn">
Your Turn…
</h3>
<p>
Use <code>analytics_raw.rds</code> data set to answer the below questions.
</p>
<ol style="list-style-type: decimal">
<li>
<p>
Check whether the below variables are factor
</p>
<ul>
<li>
<code>device</code>
</li>
<li>
<code>page_depth</code>
</li>
<li>
<code>landing_page</code>
</li>
</ul>
</li>
<li>
<p>
Coerce the following variables to type factor
</p>
<ul>
<li>
<code>device</code>
</li>
<li>
<code>os</code>
</li>
<li>
<code>browser</code>
</li>
<li>
<code>user_type</code>
</li>
<li>
<code>channel</code>
</li>
<li>
<code>gender</code>
</li>
<li>
<code>landing_page</code>
</li>
<li>
<code>exit_page</code>
</li>
<li>
<code>city</code>
</li>
<li>
<code>country</code>
</li>
<li>
<code>user_type</code>
</li>
</ul>
</li>
<li>
<p>
Use only the following levels in the <code>gender</code> column:
</p>
<ul>
<li>
<code>male</code>
</li>
<li>
<code>female</code>
</li>
</ul>
</li>
<li>
<p>
Include <code>NA</code> as a level in the gender column.
</p>
</li>
<li>
<p>
Change label of <code>NA</code> to <code>missing</code> in the <code>gender</code> column.
</p>
</li>
<li>
<p>
Change the labels of the levels in <code>user_type</code> column to
</p>
<ul>
<li>
<code>New</code>
</li>
<li>
<code>Returning</code>
</li>
</ul>
</li>
<li>
<p>
Check if the <code>user_rating</code> column is ordered. If not, coerce it to type ordered factor.
</p>
</li>
</ol>
<p>
*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.
</p>
<p>
We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*
</p>
<p>
<strong>Email: <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong>
</p>
</section>
</section>



 ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-categorical-data-in-r-part-1/</guid>
  <pubDate>Fri, 07 Jan 2022 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/forcats-part-1.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>rfm 0.2.2</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/rfm-0-2-2/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2020-07-31-rfm-0-2-2.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/github_rfm.png" width="70%" style="display: block; margin: auto;">
</p>
<p>
We’re excited to announce the release of <code>rfm</code> 0.2.2 on CRAN! <code>rfm</code> provides tools for customer segmentation using <strong>R</strong>ecency <strong>F</strong>requency <strong>M</strong>onetary value analysis. It includes a Shiny app for interactive segmentation. You can install rfm with:
</p>
<pre class="r"><code>install.packages("rfm")</code></pre>
<p>
In this blog post, we will summarize the changes implemented in the current (<code>0.2.2</code>) and previous release (<code>0.2.1</code>).
</p>
<section id="segmentation" class="level2">
<h2 class="anchored" data-anchor-id="segmentation">
Segmentation
</h2>
<p>
In previous versions, <code>rfm_segment()</code> would overwrite a segment if the intervals used to define the segment was a subset of another segment. It was expected that the end user would be careful to ensure that the intervals for each segment would be unique and not a subset of any other segment. You can see the example <a href="https://github.com/rsquaredacademy/rfm/issues/67">here</a>.
</p>
<p>
We are grateful to <a href="https://github.com/leungi" target="_blank"><span class="citation"><span class="citation" data-cites="leungi">@leungi</span></span></a> for bringing this to our attention and also for fixing it. Now, <code>rfm_segment()</code> does not overwrite the segments even if the intervals for one segment is a subset of another.
</p>
<pre class="r"><code># analysis date
analysis_date &lt;- lubridate::as_date("2006-12-31")

# rfm score
rfm_result &lt;- rfm_table_order(rfm_data_orders, customer_id, order_date,
                              revenue, analysis_date)

rfm_result</code></pre>
<pre><code>## # A tibble: 995 x 9
##    customer_id date_most_recent recency_days transaction_cou~ amount
##    &lt;chr&gt;       &lt;date&gt;                  &lt;dbl&gt;            &lt;dbl&gt;  &lt;dbl&gt;
##  1 Abbey O'Re~ 2006-06-09                205                6    472
##  2 Add Senger  2006-08-13                140                3    340
##  3 Aden Lesch~ 2006-06-20                194                4    405
##  4 Admiral Se~ 2006-08-21                132                5    448
##  5 Agness O'K~ 2006-10-02                 90                9    843
##  6 Aileen Bar~ 2006-10-08                 84                9    763
##  7 Ailene Her~ 2006-03-25                281                8    699
##  8 Aiyanna Br~ 2006-04-29                246                4    157
##  9 Ala Schmid~ 2006-01-16                349                3    363
## 10 Alannah Bo~ 2005-04-21                619                4    196
## # ... with 985 more rows, and 4 more variables: recency_score &lt;int&gt;,
## #   frequency_score &lt;int&gt;, monetary_score &lt;int&gt;, rfm_score &lt;dbl&gt;</code></pre>
<pre class="r"><code># segmentation
segment_names &lt;- c(
  "Champions", "Loyal Customers", "Potential Loyalist",
  "New Customers", "Promising", "Need Attention", "About To Sleep",
  "At Risk", "Can't Lose Them", "Lost"
)

recency_lower &lt;- c(4, 2, 3, 4, 3, 2, 2, 1, 1, 1)
recency_upper &lt;- c(5, 5, 5, 5, 4, 3, 3, 2, 1, 2)
frequency_lower &lt;- c(4, 3, 1, 1, 1, 2, 1, 2, 4, 1)
frequency_upper &lt;- c(5, 5, 3, 1, 1, 3, 2, 5, 5, 2)
monetary_lower &lt;- c(4, 3, 1, 1, 1, 2, 1, 2, 4, 1)
monetary_upper &lt;- c(5, 5, 3, 1, 1, 3, 2, 5, 5, 2)

segments &lt;-
  rfm_segment(
    rfm_result,
    segment_names,
    recency_lower,
    recency_upper,
    frequency_lower,
    frequency_upper,
    monetary_lower,
    monetary_upper
  )

# segment size
segments %&gt;%
  count(segment) %&gt;%
  arrange(desc(n)) %&gt;%
  rename(Segment = segment, Count = n)</code></pre>
<pre><code>## # A tibble: 8 x 2
##   Segment            Count
##   &lt;chr&gt;              &lt;int&gt;
## 1 Loyal Customers      278
## 2 Potential Loyalist   229
## 3 Champions            158
## 4 Lost                 111
## 5 At Risk               86
## 6 About To Sleep        50
## 7 Others                48
## 8 Need Attention        35</code></pre>
<p>
In the above example, the interval used to define the <strong>Champions</strong> segment is a subset of <strong>Loyal Customers</strong>. In the previous versions, those customers who should have been assigned <strong>Champions</strong> were reassigned as <strong>Loyal Customers</strong> if the criteria for <strong>Champions</strong> was evaluated before <strong>Loyal Customers</strong>. From version 0.2.0, <code>rfm_segment()</code> will avoid such overwriting.
</p>
<p align="center">
<a href="https://rsquared-academy.thinkific.com/courses/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/rfm-course-ad.png" width="100%" alt="new courses ad" style="text-decoration: none;"></a>
</p>
</section>
<section id="visualization" class="level2">
<h2 class="anchored" data-anchor-id="visualization">
Visualization
</h2>
<p>
<code>rfm</code> used print all the plots by default instead of returning a plot object. This resulted in difficulties for some end users who wanted to:
</p>
<ul>
<li>
further modify the plot
</li>
<li>
include the plot in a panel of other plots
</li>
</ul>
<p>
From version 0.2.1, all plotting functions use an additional argument <code>print_plot</code>. It is set to <code>TRUE</code> by default to avoid any disruption to current work flows. Those users who want a plot object to be returned can set the above argument to <code>FALSE</code>.
</p>
<pre class="r"><code># analysis date
analysis_date &lt;- lubridate::as_date('2007-01-01')

# transactions data
rfm_order &lt;- rfm_table_order(rfm_data_orders, customer_id, order_date,
                             revenue, analysis_date)

# customer data
rfm_customer &lt;- rfm_table_customer(rfm_data_customer, customer_id,
                                   number_of_orders, recency_days, revenue,
                                   analysis_date)

# plots
p1 &lt;- 
  rfm_heatmap(rfm_order, 
              plot_title = "Transaction Data", 
              print_plot = FALSE)

p2 &lt;- 
  rfm_heatmap(rfm_customer, 
              plot_title = "Customer Data", 
              print_plot = FALSE)

# using patchwork
p1 + p2</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/rfm-0-2-2/2020-07-31-rfm-0-2-2_files/figure-html/viz-1.png" width="960">
</p>
</section>
<section id="custom-threshold-for-rfm-scores" class="level2">
<h2 class="anchored" data-anchor-id="custom-threshold-for-rfm-scores">
Custom Threshold for RFM Scores
</h2>
<p>
Lots of users wanted to know the threshold used for generating the RFM scores. From version 0.2.1, <code>rfm_table_*</code> family of functions return the threshold.
</p>
<pre class="r"><code>analysis_date &lt;- lubridate::as_date('2006-12-31')
result &lt;- rfm_table_order(rfm_data_orders, customer_id, order_date,
                          revenue, analysis_date)

# threshold
result$threshold</code></pre>
<pre><code>## # A tibble: 5 x 6
##   recency_lower recency_upper frequency_lower frequency_upper monetary_lower
##           &lt;dbl&gt;         &lt;dbl&gt;           &lt;dbl&gt;           &lt;dbl&gt;          &lt;dbl&gt;
## 1            1           115                1               4            12 
## 2          115           181                4               5           256.
## 3          181           297.               5               6           382 
## 4          297.          482                6               8           506.
## 5          482           977                8              15           666 
## # ... with 1 more variable: monetary_upper &lt;dbl&gt;</code></pre>
<p>
Another request (see <a href="https://github.com/rsquaredacademy/rfm/issues/58">here</a>) was to be able to use custom or user specific threshold for generating RFM score. <code>rfm</code> uses quantiles to generate the lower and upper thresholds used for generating the scores. Unfortunately, if the data is skewed, using quantiles is not effective. From version 0.2.1, users can specify custom threshold for generating the RFM score and we will learn how to do this using an example.
</p>
<pre class="r"><code>analysis_date &lt;- lubridate::as_date('2006-12-31')
result &lt;- rfm_table_order(rfm_data_orders, customer_id, order_date, revenue,
                          analysis_date)
result$threshold</code></pre>
<pre><code>## # A tibble: 5 x 6
##   recency_lower recency_upper frequency_lower frequency_upper monetary_lower
##           &lt;dbl&gt;         &lt;dbl&gt;           &lt;dbl&gt;           &lt;dbl&gt;          &lt;dbl&gt;
## 1            1           115                1               4            12 
## 2          115           181                4               5           256.
## 3          181           297.               5               6           382 
## 4          297.          482                6               8           506.
## 5          482           977                8              15           666 
## # ... with 1 more variable: monetary_upper &lt;dbl&gt;</code></pre>
<p>
If you look at the above output, we have 5 bins/scores and there are six different values. Let us focus on the <code>monetary_*</code> columns in the threshold table. The lower threshold of the first bin and the upper threshold of the last bin are the <code>min</code> and <code>max</code> values form the <code>revenue</code> column of <code>rfm_data_orders</code> and the rest of the values are returned by the <code>quantile()</code> function.
</p>
<pre class="r"><code>revenue &lt;- 
  rfm_data_orders %&gt;% 
  group_by(customer_id) %&gt;% 
  summarize(total = sum(revenue))</code></pre>
<pre><code>## `summarise()` ungrouping (override with `.groups` argument)</code></pre>
<pre class="r"><code># revenue at customer level
revenue</code></pre>
<pre><code>## # A tibble: 995 x 2
##    customer_id        total
##  * &lt;chr&gt;              &lt;dbl&gt;
##  1 Abbey O'Reilly DVM   472
##  2 Add Senger           340
##  3 Aden Lesch Sr.       405
##  4 Admiral Senger       448
##  5 Agness O'Keefe       843
##  6 Aileen Barton        763
##  7 Ailene Hermann       699
##  8 Aiyanna Bruen PhD    157
##  9 Ala Schmidt DDS      363
## 10 Alannah Borer        196
## # ... with 985 more rows</code></pre>
<pre class="r"><code># min and max
min(revenue$total)</code></pre>
<pre><code>## [1] 12</code></pre>
<pre class="r"><code>max(revenue$total)</code></pre>
<pre><code>## [1] 1488</code></pre>
<p>
Let us look at the quantiles used for generating the scores.
</p>
<pre class="r"><code>quantile(revenue$total, probs = seq(0, 1, length.out = 6))</code></pre>
<pre><code>##     0%    20%    40%    60%    80%   100% 
##   12.0  254.8  381.0  505.4  665.0 1488.0</code></pre>
<p>
The intervals are created in the below style:
</p>
<p>
<code>Left-closed, right-open: [ a , b ) = { x ∣ a ≤ x &lt; b }</code>
</p>
<p>
Since rfm uses left closed intervals to generate the scores, we add <code>1</code> to all values except the minimum value. Now, let us recreate the RFM scores using custom threshold instead of quantiles.
</p>
<pre class="r"><code>rfm_table_order(rfm_data_orders, 
                customer_id, 
                order_date, 
                revenue,
                analysis_date, 
                recency_bins = c(115, 181, 297, 482),
                frequency_bins = c(4, 5, 6, 8), 
                monetary_bins = c(256, 382, 506, 666))</code></pre>
<pre><code>## # A tibble: 995 x 9
##    customer_id date_most_recent recency_days transaction_cou~ amount
##    &lt;chr&gt;       &lt;date&gt;                  &lt;dbl&gt;            &lt;dbl&gt;  &lt;dbl&gt;
##  1 Abbey O'Re~ 2006-06-09                205                6    472
##  2 Add Senger  2006-08-13                140                3    340
##  3 Aden Lesch~ 2006-06-20                194                4    405
##  4 Admiral Se~ 2006-08-21                132                5    448
##  5 Agness O'K~ 2006-10-02                 90                9    843
##  6 Aileen Bar~ 2006-10-08                 84                9    763
##  7 Ailene Her~ 2006-03-25                281                8    699
##  8 Aiyanna Br~ 2006-04-29                246                4    157
##  9 Ala Schmid~ 2006-01-16                349                3    363
## 10 Alannah Bo~ 2005-04-21                619                4    196
## # ... with 985 more rows, and 4 more variables: recency_score &lt;int&gt;,
## #   frequency_score &lt;int&gt;, monetary_score &lt;int&gt;, rfm_score &lt;dbl&gt;</code></pre>
<p>
We have used the values from the threshold table to reproduce the earlier result. If you observe carefully, we have specified <code>4</code> values while generating <code>5</code> bins/scores. Whenever using custom threshold, values supplied should be one less than the number of bins/scores generated as <code>rfm</code> internally computes the <code>min</code> and <code>max</code> values. In general, if you have <code>n</code> bins/scores, you only specify the upper threshold for <code>n - 1</code> bins/scores.
</p>
<p>
We have tried our best to explain how to use custom threshold but completely understand that it can be confusing to implement at beginning. If you have any questions about this method, feel free to write to us at <strong><a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong> and our team will be happy to help you.
</p>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">
Acknowledgements
</h2>
<p>
We are grateful to <a href="https://github.com/leungi" target="_blank"><span class="citation"><span class="citation" data-cites="leungi">@leungi</span></span></a>, <a href="https://github.com/gfagherazzi" target="_blank"><span class="citation"><span class="citation" data-cites="gfagherazzi">@gfagherazzi</span></span></a> and <a href="https://github.com/DavidGarciaEstaun" target="_blank"><span class="citation"><span class="citation" data-cites="DavidGarciaEstaun">@DavidGarciaEstaun</span></span></a> for their inputs.
</p>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<ul>
<li>
<a href="https://rfm.rsquaredacademy.com">Documentation</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy/rfm">GitHub</a>
</li>
</ul>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.
</p>
<p>
We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*
</p>
<p>
<strong>Email: <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong>
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>packages</category>
  <guid>https://blog.rsquaredacademy.com/posts/rfm-0-2-2/</guid>
  <pubDate>Fri, 31 Jul 2020 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/github_rfm.png" medium="image" type="image/png" height="75" width="144"/>
</item>
<item>
  <title>A Comprehensive Introduction to Handling Date &amp; Time in R</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-date-and-time-in-r/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2020-04-17-handling-date-and-time-in-r.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<div class="callout callout-style-default callout-note callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
<span class="screen-reader-only">Note</span>Updated September 2026
</div>
</div>
<div class="callout-body-container callout-body">
<p>All code re-verified on R 4.5.2 + lubridate 1.9.4 — no API changes needed. <code>clock</code> deep-dive ships as a standalone follow-up.</p>
</div>
</div>
<script src="../../rmarkdown-libs/kePrint/kePrint.js"></script>
<p>
<img src="https://blog.rsquaredacademy.com/img/handling-date-and-time-in-r.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
In this tutorial, we will learn to handle date &amp; time in R. We will start off by learning how to <strong>get current date &amp; time</strong> before moving on to understand <strong>how R handles date/time internally</strong> and the different classes such as <code>Date</code> &amp; <code>POSIXct/lt</code>. We will spend some time exploring <strong>time zones, daylight savings and ISO 8001 standard</strong> for representing date/time. We will look at all the <strong>weird formats in which date/time come in real world</strong> and learn to <strong>parse them using conversion specifications</strong>. After this, we will also <strong>learn how to handle date/time columns while reading external data into R</strong>. We will learn to <strong>extract and update different date/time components</strong> such as year, month, day, hour, minute etc., <strong>create sequence of dates</strong> in different ways and explore intervals, durations and period. We will end the tutorial by learning how to <strong>round/rollback dates</strong>. Throughout the tutorial, we will also <strong>work through a case study</strong> to better understand the concepts we learn. Happy learning!
</p>
<section id="table-of-contents" class="level2">
<h2 class="anchored" data-anchor-id="table-of-contents">
Table of Contents
</h2>
<ul>
<li>
Resources
</li>
<li>
Introduction
</li>
<li>
Case Study
</li>
<li>
Date &amp; Time Classes
</li>
<li>
Date Arithmetic
</li>
<li>
Timezones &amp; Daylight Savings
</li>
<li>
Date &amp; Time Formats
</li>
<li>
Parse/Read Date &amp; Time
</li>
<li>
Date &amp; Time Components
</li>
<li>
Create, Update &amp; Verify
</li>
<li>
Intervals, Durations &amp; Period
</li>
<li>
Round &amp; Rollback
</li>
<li>
References
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all the resources related to this tutorial:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/handling-date-and-time-in-r.pdf" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/1072419" target="_blank">RStudio Cloud Project</a>
</li>
<li>
<a href="https://wrangle-r.rsquaredacademy.com/lubridate.html" target="_blank">ebook</a>
</li>
</ul>
<p align="center">
<a href="https://rsquared-academy.thinkific.com/courses/handling-date-and-time-in-R" target="_blank"><img src="https://blog.rsquaredacademy.com/img/lubirdate-blog-course-ad.png" width="100%" alt="new courses ad" style="text-decoration: none;"></a>
</p>
</section>
<section id="intro" class="level2">
<h2 class="anchored" data-anchor-id="intro">
Introduction
</h2>
{{% youtube “322IcnZiYx4” %}}
<section id="date" class="level3">
<h3 class="anchored" data-anchor-id="date">
Date
</h3>
<p>
Let us begin by looking at the current date and time. <code>Sys.Date()</code> and <code>today()</code> will return the current date.
</p>
<pre class="r"><code>Sys.Date()</code></pre>
<pre><code>## [1] "2020-06-10"</code></pre>
<pre class="r"><code>lubridate::today()</code></pre>
<pre><code>## [1] "2020-06-10"</code></pre>
</section>
<section id="time" class="level3">
<h3 class="anchored" data-anchor-id="time">
Time
</h3>
<p>
<code>Sys.time()</code> and <code>now()</code> return the date, time and the timezone. In <code>now()</code>, we can specify the timezone using the <code>tzone</code> argument.
</p>
<pre class="r"><code>Sys.time()</code></pre>
<pre><code>## [1] "2020-06-10 22:47:50 IST"</code></pre>
<pre class="r"><code>lubridate::now()</code></pre>
<pre><code>## [1] "2020-06-10 22:47:50 IST"</code></pre>
<pre class="r"><code>lubridate::now(tzone = "UTC")</code></pre>
<pre><code>## [1] "2020-06-10 17:17:50 UTC"</code></pre>
</section>
<section id="am-or-pm" class="level3">
<h3 class="anchored" data-anchor-id="am-or-pm">
AM or PM?
</h3>
<p>
<code>am()</code> and <code>pm()</code> allow us to check whether date/time occur in the <code>AM</code> or <code>PM</code>? They return a logical value i.e.&nbsp;<code>TRUE</code> or <code>FALSE</code>
</p>
<pre class="r"><code>lubridate::am(now())</code></pre>
<pre><code>## [1] FALSE</code></pre>
<pre class="r"><code>lubridate::pm(now())</code></pre>
<pre><code>## [1] TRUE</code></pre>
</section>
<section id="leap-year" class="level3">
<h3 class="anchored" data-anchor-id="leap-year">
Leap Year
</h3>
<p>
We can also check if the current year is a leap year using <code>leap_year()</code>.
</p>
<pre class="r"><code>Sys.Date()</code></pre>
<pre><code>## [1] "2020-06-10"</code></pre>
<pre class="r"><code>lubridate::leap_year(Sys.Date())</code></pre>
<pre><code>## [1] TRUE</code></pre>
</section>
<section id="summary" class="level3">
<h3 class="anchored" data-anchor-id="summary">
Summary
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Function
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>Sys.Date()</code>
</td>
<td style="text-align:left;">
Current Date
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::today()</code>
</td>
<td style="text-align:left;">
Current Date
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>Sys.time()</code>
</td>
<td style="text-align:left;">
Current Time
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::now()</code>
</td>
<td style="text-align:left;">
Current Time
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::am()</code>
</td>
<td style="text-align:left;">
Whether time occurs in am?
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::pm()</code>
</td>
<td style="text-align:left;">
Whether time occurs in pm?
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::leap_year()</code>
</td>
<td style="text-align:left;">
Check if the year is a leap year?
</td>
</tr>
</tbody>
</table>
</section>
<section id="your-turn" class="level3">
<h3 class="anchored" data-anchor-id="your-turn">
Your Turn
</h3>
<ul>
<li>
get current date
</li>
<li>
get current time
</li>
<li>
check whether the time occurs in am or pm?
</li>
<li>
check whether the following years were leap years
<ul>
<li>
2018
</li>
<li>
2016
</li>
</ul>
</li>
</ul>
</section>
</section>
<section id="casestudy" class="level2">
<h2 class="anchored" data-anchor-id="casestudy">
Case Study
</h2>
<p>
Throughout the tutorial, we will work on a case study related to transactions of an imaginary trading company. The data set includes information about invoice and payment dates.
</p>
<section id="data" class="level3">
<h3 class="anchored" data-anchor-id="data">
Data
</h3>
<pre class="r"><code>transact &lt;- readr::read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/transact.csv')</code></pre>
<pre><code>## # A tibble: 2,466 x 3
##    Invoice    Due        Payment   
##    &lt;date&gt;     &lt;date&gt;     &lt;date&gt;    
##  1 2013-01-02 2013-02-01 2013-01-15
##  2 2013-01-26 2013-02-25 2013-03-03
##  3 2013-07-03 2013-08-02 2013-07-08
##  4 2013-02-10 2013-03-12 2013-03-17
##  5 2012-10-25 2012-11-24 2012-11-28
##  6 2012-01-27 2012-02-26 2012-02-22
##  7 2013-08-13 2013-09-12 2013-09-09
##  8 2012-12-16 2013-01-15 2013-01-12
##  9 2012-05-14 2012-06-13 2012-07-01
## 10 2013-07-01 2013-07-31 2013-07-26
## # ... with 2,456 more rows</code></pre>
<p>
We will explore more about reading data sets with date/time columns after learning how to parse date/time. We have shared the code for reading the data sets used in the practice questions both in the Learning Management System as well as in our GitHub repo.
</p>
</section>
<section id="data-dictionary" class="level3">
<h3 class="anchored" data-anchor-id="data-dictionary">
Data Dictionary
</h3>
<p>
The data set has 3 columns. All the dates are in the format (yyyy-mm-dd).
</p>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Column
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
Invoice
</td>
<td style="text-align:left;">
Invoice Date
</td>
</tr>
<tr>
<td style="text-align:left;">
Due
</td>
<td style="text-align:left;">
Due Date
</td>
</tr>
<tr>
<td style="text-align:left;">
Payment
</td>
<td style="text-align:left;">
Payment Date
</td>
</tr>
</tbody>
</table>
<p>
In the case study, we will try to answer a few questions we have about the <code>transact</code> data.
</p>
<ul>
<li>
extract date, month and year from Due
</li>
<li>
compute the number of days to settle invoice
</li>
<li>
compute days over due
</li>
<li>
check if due year is a leap year
</li>
<li>
check when due day in february is 29, whether it is a leap year
</li>
<li>
how many invoices were settled within due date
</li>
<li>
how many invoices are due in each quarter
</li>
</ul>
</section>
</section>
<section id="classes" class="level2">
<h2 class="anchored" data-anchor-id="classes">
Date &amp; Time Classes
</h2>
{{% youtube “IEX49t8sSgw” %}}
<p>
In this section, we will look at two things. First, how to create date/time data in R, and second, how to convert other data types to date/time. Let us begin by creating the release date of R 3.6.2.
</p>
<pre class="r"><code>release_date &lt;- 2019-12-12
release_date</code></pre>
<pre><code>## [1] 1995</code></pre>
<p>
Okay! Why do we see <code>1995</code> when we call the date? What is happening here? Let us quickly check the data type of <code>release_date</code>.
</p>
<pre class="r"><code>class(release_date)</code></pre>
<pre><code>## [1] "numeric"</code></pre>
<p>
The data type is <code>numeric</code> i.e.&nbsp;R has subtracted <code>12</code> twice from <code>2019</code> to return <code>1995</code>. Clearly, the above method is not the right way to store date/time. Let us see if we can get some hints from the built-in R functions we used in the previous section. If you observe the output, all of them returned date/time wrapped in quotes. Hmmm… let us wrap our date in quotes and see what happens.
</p>
<pre class="r"><code>release_date &lt;- "2019-12-12"
release_date</code></pre>
<pre><code>## [1] "2019-12-12"</code></pre>
<p>
Alright, now R does not do any arithmetic and returns the date as we specified. Great! Is this the right format to store date/time then? No.&nbsp;Why? What is the problem if date/time is saved as character/string? The problem is the nature or type of operations done on date/time are different when compared to string/character, number or logical values.
</p>
<ul>
<li>
how do we add/subtract dates?
</li>
<li>
how do we extract components such as year, month, day etc.
</li>
</ul>
<p>
To answer the above questions, we will first check the data type of <code>Sys.Date()</code> and <code>now()</code>.
</p>
<pre class="r"><code>class(Sys.Date())</code></pre>
<pre><code>## [1] "Date"</code></pre>
<pre class="r"><code>class(lubridate::now())</code></pre>
<pre><code>## [1] "POSIXct" "POSIXt"</code></pre>
<pre class="r"><code>class(release_date)</code></pre>
<pre><code>## [1] "character"</code></pre>
<p>
As you can see from the above output, there are 3 different classes for storing date/time in R
</p>
<ul>
<li>
<code>Date</code>
</li>
<li>
<code>POSIXct</code>
</li>
<li>
<code>POSIXlt</code>
</li>
</ul>
<p>
Let us explore each of the above classes one by one.
</p>
<section id="date-1" class="level3">
<h3 class="anchored" data-anchor-id="date-1">
Date
</h3>
<section id="introduction" class="level4">
<h4 class="anchored" data-anchor-id="introduction">
Introduction
</h4>
<p>
The <code>Date</code> class represents calendar dates. Let us go back to <code>Sys.Date()</code>. If you check the class of <code>Sys.Date()</code>, it is <code>Date</code>. Internally, this date is a number i.e.&nbsp;an integer. The <code>unclass()</code> function will show how dates are stored internally.
</p>
<pre class="r"><code>Sys.Date()</code></pre>
<pre><code>## [1] "2020-06-10"</code></pre>
<pre class="r"><code>unclass(Sys.Date())</code></pre>
<pre><code>## [1] 18423</code></pre>
<p>
What does this integer represent? Why has R stored the date as an integer? In R, dates are represented as the number of days since <code>1970-01-01</code>. All the dates in R are internally stored in this way. Before we explore this concept further, let us learn to create <code>Date</code> objects in R. We will continue to use the release date of R 3.6.2, <code>2019-12-12</code>.
</p>
<p>
Until now, we have stored the above date as character/string but now we will use <code>as.Date()</code> to save it as a <code>Date</code> object. <code>as.Date()</code> is the easiest and simplest way to create dates in R.
</p>
<pre class="r"><code>release_date &lt;- as.Date("2019-12-12")
release_date</code></pre>
<pre><code>## [1] "2019-12-12"</code></pre>
<p>
The <code>as_date()</code> function from the lubridate package is similar to <code>as.Date()</code>.
</p>
<pre class="r"><code>release_date &lt;- lubridate::as_date("2019-12-12")
release_date</code></pre>
<pre><code>## [1] "2019-12-12"</code></pre>
<p>
If you look at the difference between <code>release_date</code> and <code>1970-01-01</code>, it will be the same as <code>unclass(release_date)</code>.
</p>
<pre class="r"><code>release_date - as.Date("1970-01-01")</code></pre>
<pre><code>## Time difference of 18242 days</code></pre>
<pre class="r"><code>unclass(release_date)</code></pre>
<pre><code>## [1] 18242</code></pre>
<p>
Let us come back to <code>1970-01-01</code> i.e.&nbsp;the origin for dates in R.
</p>
<pre class="r"><code>lubridate::origin</code></pre>
<pre><code>## [1] "1970-01-01 UTC"</code></pre>
<p>
From the previous examples, we know that dates are internally stored as number of days since <code>1970-01-01</code>. How about dates older than the origin? How are they stored? Let us look at that briefly.
</p>
<pre class="r"><code>unclass(as.Date("1963-08-28"))</code></pre>
<pre><code>## [1] -2318</code></pre>
<p>
Dates older than the origin are stored as negative integers. For those who are not aware, Martin Luther King, Jr.&nbsp;delivered his famous <strong>I Have a Dream</strong> speech on <code>1963-08-28</code>. Let us move on and learn how to convert numbers into dates.
</p>
</section>
<section id="convert-numeric" class="level4">
<h4 class="anchored" data-anchor-id="convert-numeric">
Convert Numeric
</h4>
<p>
The <code>as.Date()</code> function can be used to convert any of the following to a <code>Date</code> object
</p>
<ul>
<li>
character/string
</li>
<li>
number
</li>
<li>
factor (categorical/qualitative)
</li>
</ul>
<p>
We have explored how to convert strings to date. How about converting numbers to date? Sure, we can create date from numbers by specifying the origin and number of days since it.
</p>
<pre class="r"><code>as.Date(18242, origin = "1970-01-01")</code></pre>
<pre><code>## [1] "2019-12-12"</code></pre>
<p>
The origin can be changed to another date (while changing the number as well.)
</p>
<pre class="r"><code>as.Date(7285, origin = "2000-01-01")</code></pre>
<pre><code>## [1] "2019-12-12"</code></pre>
</section>
</section>
<section id="iso-8601" class="level3">
<h3 class="anchored" data-anchor-id="iso-8601">
ISO 8601
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/iso.png" width="70%" style="display: block; margin: auto;">
</p>
<p>
If you have carefully observed, the format in which we have been specifying the dates as well as of those returned by functions such as <code>Sys.Date()</code> or <code>Sys.time()</code> is the same i.e.&nbsp;<code>YYYY-MM-DD</code>. It includes
</p>
<ul>
<li>
the year including the century
</li>
<li>
the month
</li>
<li>
the date
</li>
</ul>
<p>
The month and date separated by <code>-</code>. This default format used in R is the ISO 8601 standard for date/time. ISO 8601 is the internationally accepted way to represent dates and times and uses the 24 hour clock system. Let us create the release date using another function <code>ISOdate()</code>.
</p>
<pre class="r"><code>ISOdate(year  = 2019,
        month = 12,
        day   = 12,
        hour  = 8,
        min   = 5, 
        sec   = 3,
        tz    = "UTC")</code></pre>
<pre><code>## [1] "2019-12-12 08:05:03 UTC"</code></pre>
<p>
We will look at all the different weird ways in which date/time are specified in the real world in the Date &amp; Time Formats section. For the time being, let us continue exploring date/time classes in R. The next class we are going to look at is <code>POSIXct/POSIXlt</code>.
</p>
</section>
<section id="posix" class="level3">
<h3 class="anchored" data-anchor-id="posix">
POSIX
</h3>
<p>
You might be wondering what is this POSIX thing? POSIX stands for <strong>P</strong>ortable <strong>O</strong>perating <strong>S</strong>ystem <strong>I</strong>nterface. It is a family of standards specified for maintaining compatibility between different operating systems. Before we learn to create POSIX objects, let us look at <code>now()</code> from lubridate.
</p>
<pre class="r"><code>class(lubridate::now())</code></pre>
<pre><code>## [1] "POSIXct" "POSIXt"</code></pre>
<p>
<code>now()</code> returns current date/time as a POSIXct object. Let us look at its internal representation using <code>unclass()</code>
</p>
<pre class="r"><code>unclass(lubridate::now())</code></pre>
<pre><code>## [1] 1591809475
## attr(,"tzone")
## [1] ""</code></pre>
<p>
The output you see is the number of seconds since January 1, 1970.
</p>
<section id="posixct" class="level4">
<h4 class="anchored" data-anchor-id="posixct">
POSIXct
</h4>
<p>
<code>POSIXct</code> represents the number of seconds since the beginning of 1970 (UTC) and <code>ct</code> stands for calendar time. To store date/time as <code>POSIXct</code> objects, use <code>as.POSIXct()</code>. Let us now store the release date of R 3.6.2 as <code>POSIXct</code> as shown below
</p>
<pre class="r"><code>release_date &lt;- as.POSIXct("2019-12-12 08:05:03")
class(release_date)</code></pre>
<pre><code>## [1] "POSIXct" "POSIXt"</code></pre>
<pre class="r"><code>unclass(release_date) </code></pre>
<pre><code>## [1] 1576118103
## attr(,"tzone")
## [1] ""</code></pre>
</section>
<section id="posixlt" class="level4">
<h4 class="anchored" data-anchor-id="posixlt">
POSIXlt
</h4>
<p>
<code>POSIXlt</code> represents the following information in a list
</p>
<ul>
<li>
seconds
</li>
<li>
minutes
</li>
<li>
hour
</li>
<li>
day of the month
</li>
<li>
month
</li>
<li>
year
</li>
<li>
day of week
</li>
<li>
day of year
</li>
<li>
daylight saving time flag
</li>
<li>
time zone
</li>
<li>
offset in seconds from GMT
</li>
</ul>
<p>
The <code>lt</code> in <code>POSIXlt</code> stands for local time. Use <code>as.POSIXlt()</code> to store date/time as <code>POSIXlt</code> objects. Let us store the release date as a <code>POSIXlt</code> object as shown below
</p>
<pre class="r"><code>release_date &lt;- as.POSIXlt("2019-12-12 08:05:03")
release_date</code></pre>
<pre><code>## [1] "2019-12-12 08:05:03 IST"</code></pre>
<p>
As we said earlier, <code>POSIXlt</code> stores date/time components in a list and these can be extracted. Let us look at the date/time components returned by <code>POSIXlt</code> using <code>unclass()</code>.
</p>
<pre class="r"><code>release_date &lt;- as.POSIXlt("2019-12-12 08:05:03")
unclass(release_date)</code></pre>
<pre><code>## $sec
## [1] 3
## 
## $min
## [1] 5
## 
## $hour
## [1] 8
## 
## $mday
## [1] 12
## 
## $mon
## [1] 11
## 
## $year
## [1] 119
## 
## $wday
## [1] 4
## 
## $yday
## [1] 345
## 
## $isdst
## [1] 0
## 
## $zone
## [1] "IST"
## 
## $gmtoff
## [1] NA</code></pre>
<p>
Use <code>unlist()</code> if you want the components returned as a <code>vector</code>.
</p>
<pre class="r"><code>release_date &lt;- as.POSIXlt("2019-12-12 08:05:03")
unlist(release_date)</code></pre>
<pre><code>##    sec    min   hour   mday    mon   year   wday   yday  isdst   zone gmtoff 
##    "3"    "5"    "8"   "12"   "11"  "119"    "4"  "345"    "0"  "IST"     NA</code></pre>
<p>
To extract specific components, use <code><img src="https://latex.codecogs.com/png.latex?%3C/code%3E.%3C/p%3E%0A%3Cpre%20class=%22r%22%3E%3Ccode%3Erelease_date%20&amp;lt;-%20as.POSIXlt(&amp;quot;2019-12-12%2008:05:03&amp;quot;)%0Arelease_date">hour</code>

</p><pre><code>## [1] 8</code></pre>
<pre class="r"><code>release_date$mon</code></pre>
<pre><code>## [1] 11</code></pre>
<pre class="r"><code>release_date$zone</code></pre>
<pre><code>## [1] "IST"</code></pre>
<p>
Now, let us look at the components returned by <code>POSIXlt</code>. Some of them are intuitive
</p>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Component
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>sec</code>
</td>
<td style="text-align:left;">
Second
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>min</code>
</td>
<td style="text-align:left;">
Minute
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>hour</code>
</td>
<td style="text-align:left;">
Hour of the day
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>mon</code>
</td>
<td style="text-align:left;">
Month of the year (0-11
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>zone</code>
</td>
<td style="text-align:left;">
Timezone
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>wday</code>
</td>
<td style="text-align:left;">
Day of week
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>mday</code>
</td>
<td style="text-align:left;">
Day of month
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>year</code>
</td>
<td style="text-align:left;">
Years since 1900
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>yday</code>
</td>
<td style="text-align:left;">
Day of year
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>isdst</code>
</td>
<td style="text-align:left;">
Daylight saving flag
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>gmtoff</code>
</td>
<td style="text-align:left;">
Offset is seconds from GMT
</td>
</tr>
</tbody>
</table>
<p>
Great! We will end this section with a few tips/suggestions on when to use <code>Date</code> or <code>POSIXct/POSIXlt</code>.
</p>
<ul>
<li>
use <code>Date</code> when there is no time component
</li>
<li>
use <code>POSIX</code> when dealing with time and timezones
</li>
<li>
use <code>POSIXlt</code> when you want to access/extract the different components
</li>
</ul>
</section></section></section> ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-date-and-time-in-r/</guid>
  <pubDate>Fri, 17 Apr 2020 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/handling-date-and-time-in-r.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Date &amp; Time in R - Introduction</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/handling-date-and-time-in-r-part-1/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2020-04-17-handling-date-and-time-in-r-part-1.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/header-attrs/header-attrs.js"></script>
<script src="../../rmarkdown-libs/kePrint/kePrint.js"></script>
<p><link href="../../rmarkdown-libs/lightable/lightable.css" rel="stylesheet"></p>
<p>
<img src="https://blog.rsquaredacademy.com/img/handling-date-and-time-in-r.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
In this new series <a href="https://tutorials.rsquaredacademy.com/date-time/index.html">Handling Date &amp; Time in R</a>, we will learn to handle date &amp; time in R. We will start off by learning how to <strong>get current date &amp; time</strong> before moving on to understand <strong>how R handles date/time internally</strong> and the different classes such as <code>Date</code> &amp; <code>POSIXct/lt</code>. We will spend some time exploring <strong>time zones, daylight savings and ISO 8001 standard</strong> for representing date/time. We will look at all the <strong>weird formats in which date/time come in real world</strong> and learn to <strong>parse them using conversion specifications</strong>. After this, we will also <strong>learn how to handle date/time columns while reading external data into R</strong>. We will learn to <strong>extract and update different date/time components</strong> such as year, month, day, hour, minute etc., <strong>create sequence of dates</strong> in different ways and explore intervals, durations and period. We will end the tutorial by learning how to <strong>round/rollback dates</strong>. Throughout the series, we will also <strong>work through a case study</strong> to better understand the concepts we learn. Happy learning!
</p>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all the resources related to this tutorial:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/handling-date-and-time-in-r.pdf" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/1072419" target="_blank">RStudio Cloud Project</a>
</li>
<li>
<a href="https://wrangle-r.rsquaredacademy.com/lubridate.html" target="_blank">ebook</a>
</li>
</ul>
<p align="center">
<a href="https://rsquared-academy.thinkific.com/courses/handling-date-and-time-in-R" target="_blank"><img src="https://blog.rsquaredacademy.com/img/lubirdate-blog-course-ad.png" width="100%" alt="new courses ad" style="text-decoration: none;"></a>
</p>
</section>
<section id="intro" class="level2">
<h2 class="anchored" data-anchor-id="intro">
Introduction
</h2>
{{% youtube “322IcnZiYx4” %}}
<section id="date" class="level3">
<h3 class="anchored" data-anchor-id="date">
Date
</h3>
<p>
Let us begin by looking at the current date and time. <code>Sys.Date()</code> and <code>today()</code> will return the current date.
</p>
<pre class="r"><code>Sys.Date()</code></pre>
<pre><code>## [1] "2021-02-03"</code></pre>
<pre class="r"><code>lubridate::today()</code></pre>
<pre><code>## [1] "2021-02-03"</code></pre>
</section>
<section id="time" class="level3">
<h3 class="anchored" data-anchor-id="time">
Time
</h3>
<p>
<code>Sys.time()</code> and <code>now()</code> return the date, time and the timezone. In <code>now()</code>, we can specify the timezone using the <code>tzone</code> argument.
</p>
<pre class="r"><code>Sys.time()</code></pre>
<pre><code>## [1] "2021-02-03 18:54:23 IST"</code></pre>
<pre class="r"><code>lubridate::now()</code></pre>
<pre><code>## [1] "2021-02-03 18:54:23 IST"</code></pre>
<pre class="r"><code>lubridate::now(tzone = "UTC")</code></pre>
<pre><code>## [1] "2021-02-03 13:24:23 UTC"</code></pre>
</section>
<section id="am-or-pm" class="level3">
<h3 class="anchored" data-anchor-id="am-or-pm">
AM or PM?
</h3>
<p>
<code>am()</code> and <code>pm()</code> allow us to check whether date/time occur in the <code>AM</code> or <code>PM</code>? They return a logical value i.e.&nbsp;<code>TRUE</code> or <code>FALSE</code>
</p>
<pre class="r"><code>lubridate::am(now())</code></pre>
<pre><code>## [1] FALSE</code></pre>
<pre class="r"><code>lubridate::pm(now())</code></pre>
<pre><code>## [1] TRUE</code></pre>
</section>
<section id="leap-year" class="level3">
<h3 class="anchored" data-anchor-id="leap-year">
Leap Year
</h3>
<p>
We can also check if the current year is a leap year using <code>leap_year()</code>.
</p>
<pre class="r"><code>Sys.Date()</code></pre>
<pre><code>## [1] "2021-02-03"</code></pre>
<pre class="r"><code>lubridate::leap_year(Sys.Date())</code></pre>
<pre><code>## [1] FALSE</code></pre>
</section>
<section id="summary" class="level3">
<h3 class="anchored" data-anchor-id="summary">
Summary
</h3>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Function
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
<code>Sys.Date()</code>
</td>
<td style="text-align:left;">
Current Date
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::today()</code>
</td>
<td style="text-align:left;">
Current Date
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>Sys.time()</code>
</td>
<td style="text-align:left;">
Current Time
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::now()</code>
</td>
<td style="text-align:left;">
Current Time
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::am()</code>
</td>
<td style="text-align:left;">
Whether time occurs in am?
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::pm()</code>
</td>
<td style="text-align:left;">
Whether time occurs in pm?
</td>
</tr>
<tr>
<td style="text-align:left;">
<code>lubridate::leap_year()</code>
</td>
<td style="text-align:left;">
Check if the year is a leap year?
</td>
</tr>
</tbody>
</table>
</section>
<section id="your-turn" class="level3">
<h3 class="anchored" data-anchor-id="your-turn">
Your Turn
</h3>
<ul>
<li>
get current date
</li>
<li>
get current time
</li>
<li>
check whether the time occurs in am or pm?
</li>
<li>
check whether the following years were leap years
<ul>
<li>
2018
</li>
<li>
2016
</li>
</ul>
</li>
</ul>
</section>
</section>
<section id="casestudy" class="level2">
<h2 class="anchored" data-anchor-id="casestudy">
Case Study
</h2>
<p>
Throughout the tutorial, we will work on a case study related to transactions of an imaginary trading company. The data set includes information about invoice and payment dates.
</p>
<section id="data" class="level3">
<h3 class="anchored" data-anchor-id="data">
Data
</h3>
<pre class="r"><code>transact &lt;- readr::read_csv('https://raw.githubusercontent.com/rsquaredacademy/datasets/master/transact.csv')</code></pre>
<pre><code>## # A tibble: 2,466 x 3
##    Invoice    Due        Payment   
##    &lt;date&gt;     &lt;date&gt;     &lt;date&gt;    
##  1 2013-01-02 2013-02-01 2013-01-15
##  2 2013-01-26 2013-02-25 2013-03-03
##  3 2013-07-03 2013-08-02 2013-07-08
##  4 2013-02-10 2013-03-12 2013-03-17
##  5 2012-10-25 2012-11-24 2012-11-28
##  6 2012-01-27 2012-02-26 2012-02-22
##  7 2013-08-13 2013-09-12 2013-09-09
##  8 2012-12-16 2013-01-15 2013-01-12
##  9 2012-05-14 2012-06-13 2012-07-01
## 10 2013-07-01 2013-07-31 2013-07-26
## # ... with 2,456 more rows</code></pre>
<p>
We will explore more about reading data sets with date/time columns after learning how to parse date/time. We have shared the code for reading the data sets used in the practice questions both in the Learning Management System as well as in our GitHub repo.
</p>
</section>
<section id="data-dictionary" class="level3">
<h3 class="anchored" data-anchor-id="data-dictionary">
Data Dictionary
</h3>
<p>
The data set has 3 columns. All the dates are in the format (yyyy-mm-dd).
</p>
<table class="table table-striped table-hover table-condensed table-responsive" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Column
</th>
<th style="text-align:left;">
Description
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
Invoice
</td>
<td style="text-align:left;">
Invoice Date
</td>
</tr>
<tr>
<td style="text-align:left;">
Due
</td>
<td style="text-align:left;">
Due Date
</td>
</tr>
<tr>
<td style="text-align:left;">
Payment
</td>
<td style="text-align:left;">
Payment Date
</td>
</tr>
</tbody>
</table>
<p>
In the case study, we will try to answer a few questions we have about the <code>transact</code> data.
</p>
<ul>
<li>
extract date, month and year from Due
</li>
<li>
compute the number of days to settle invoice
</li>
<li>
compute days over due
</li>
<li>
check if due year is a leap year
</li>
<li>
check when due day in february is 29, whether it is a leap year
</li>
<li>
how many invoices were settled within due date
</li>
<li>
how many invoices are due in each quarter
</li>
</ul>
<p>
*As the reader of this blog, you are our most important critic and commentator. We value your opinion and want to know what we are doing right, what we could do better, what areas you would like to see us publish in, and any other words of wisdom you are willing to pass our way.
</p>
<p>
We welcome your comments. You can email to let us know what you did or did not like about our blog as well as what we can do to make our post better.*
</p>
<p>
<strong>Email: <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a></strong>
</p>
</section>
</section>



 ]]></description>
  <category>R</category>
  <category>data-wrangling</category>
  <guid>https://blog.rsquaredacademy.com/posts/handling-date-and-time-in-r-part-1/</guid>
  <pubDate>Thu, 16 Apr 2020 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/handling-date-and-time-in-r.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Introducing nse2r</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/nse2r-fetch-data-from-nse/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2020-01-17-nse2r-intro.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/github_nse2r.png" width="70%" style="display: block; margin: auto;">
</p>
<p>
We are excited to announce the <a href="https://nse2r.rsquaredacademy.com">nse2r</a> package. <a href="https://www.nseindia.com/">NSE</a> (National Stock Exchange) is the leading stock exchange of India, located in the city of Mumbai. While users can manually download data from NSE through a browser, importing this data into R becomes cumbersome. The nse2r R package implements the retrieval of data from NSE and aims to reduce the pre-processing steps needed in analyzing such data.
</p>
<p>
nse2r is inspired by and a port of the Python package <a href="https://nsetools.readthedocs.io/en/latest/">nsetools</a>. The authors and contributors for this R package are not affiliated with NSE and NSE does not offer support for this R package.
</p>
<p>
With nse2r, you can fetch the following data related to:
</p>
<ul>
<li>
stocks
<ul>
<li>
quote for a given stock
</li>
<li>
stock description
</li>
<li>
validate stock symbol/ticker
</li>
<li>
most actively traded stocks in a month
</li>
<li>
52 week high/low
</li>
<li>
top gainers/losers for the last trading session
</li>
</ul>
</li>
<li>
index
<ul>
<li>
list of NSE indices
</li>
<li>
validate index symbol/ticker
</li>
<li>
quote for a given index
</li>
</ul>
</li>
<li>
futures &amp; options
<ul>
<li>
top gainers/losers for the last trading session
</li>
</ul>
</li>
<li>
pre open market data
<ul>
<li>
nifty
</li>
<li>
nifty bank
</li>
</ul>
</li>
<li>
indices advances &amp; declines
</li>
</ul>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">
Installation
</h2>
<pre class="r"><code># Install release version from CRAN
install.packages("nse2r")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/nse2r")</code></pre>
</section>
<section id="usage" class="level2">
<h2 class="anchored" data-anchor-id="usage">
Usage
</h2>
<p>
nse2r uses consistent prefix <code>nse_</code> for easy tab completion.
</p>
<ul>
<li>
<code>nse_index_</code> for index
</li>
<li>
<code>nse_stock_</code> for stocks
</li>
<li>
<code>nse_fo_</code> for futures and options
</li>
<li>
<code>nse_preopen_</code> for preopen data
</li>
</ul>
</section>
<section id="preprocessing" class="level2">
<h2 class="anchored" data-anchor-id="preprocessing">
Preprocessing
</h2>
<p>
nse2r does basic data preprocessing which are listed below:
</p>
<ul>
<li>
modify column data types from <code>character</code> to <code>numeric</code> and <code>Date</code>
</li>
<li>
modify column names
<ul>
<li>
make them more descriptive
</li>
<li>
to <code>snake_case</code> from <code>camelCase</code>
</li>
</ul>
</li>
</ul>
<p>
Users can retain the names and format as returned by NSE using the <code>clean_names</code> argument and setting it to <code>FALSE</code>.
</p>
</section>
<section id="quick-overview" class="level2">
<h2 class="anchored" data-anchor-id="quick-overview">
Quick Overview
</h2>
<section id="fetch-indices-quote" class="level4">
<h4 class="anchored" data-anchor-id="fetch-indices-quote">
Fetch Indices Quote
</h4>
<pre class="r"><code>nse_index_quote()</code></pre>
<pre><code>## # A tibble: 55 x 4
##    index_name        last_traded_price change percent_change
##    &lt;chr&gt;                         &lt;dbl&gt;  &lt;dbl&gt;          &lt;dbl&gt;
##  1 NIFTY 50 Pre Open           10073.   26.0            0.26
##  2 NIFTY 50                    10116.   69.5            0.69
##  3 NIFTY NEXT 50               25154.   -8.9           -0.04
##  4 NIFTY100 LIQ 15              2587    17.3            0.67
##  5 NIFTY BANK                  21100.  375.             1.81
##  6 INDIA VIX                      29.4  -0.77          -2.54
##  7 NIFTY 100                   10292.   60.4            0.59
##  8 NIFTY 500                    8293.   55.2            0.67
##  9 NIFTY MIDCAP 100            14342   173.             1.22
## 10 NIFTY MIDCAP 50              3993.   52              1.32
## # ... with 45 more rows</code></pre>
<pre class="r"><code># retain original column names as returned by NSE
nse_index_quote(clean_names = FALSE)</code></pre>
<pre><code>## # A tibble: 55 x 4
##    name              lastPrice change pChange
##    &lt;chr&gt;                 &lt;dbl&gt;  &lt;dbl&gt;   &lt;dbl&gt;
##  1 NIFTY 50 Pre Open   10073.   26.0     0.26
##  2 NIFTY 50            10116.   69.5     0.69
##  3 NIFTY NEXT 50       25154.   -8.9    -0.04
##  4 NIFTY100 LIQ 15      2587    17.3     0.67
##  5 NIFTY BANK          21100.  375.      1.81
##  6 INDIA VIX              29.4  -0.77   -2.54
##  7 NIFTY 100           10292.   60.4     0.59
##  8 NIFTY 500            8293.   55.2     0.67
##  9 NIFTY MIDCAP 100    14342   173.      1.22
## 10 NIFTY MIDCAP 50      3993.   52       1.32
## # ... with 45 more rows</code></pre>
</section>
<section id="top-gainers-for-the-last-trading-session." class="level4">
<h4 class="anchored" data-anchor-id="top-gainers-for-the-last-trading-session.">
Top gainers for the last trading session.
</h4>
<pre class="r"><code>nse_stock_top_gainers()</code></pre>
<pre><code>## # A tibble: 10 x 12
##    symbol series last_corp_annou~ last_corp_annou~ open_price high_price
##    &lt;chr&gt;  &lt;chr&gt;  &lt;date&gt;           &lt;chr&gt;                 &lt;dbl&gt;      &lt;dbl&gt;
##  1 INDUS~ EQ     2019-08-08       Annual General ~       464        510.
##  2 HINDA~ EQ     2019-08-14       Annual General ~       144.       148 
##  3 AXISB~ EQ     2019-07-04       Annual General ~       422.       433.
##  4 SHREE~ EQ     2020-02-24       Interim Dividen~     21171.     21611.
##  5 KOTAK~ EQ     2019-07-12       Annual General ~      1304.      1344.
##  6 RELIA~ EQ     2020-05-13       Rights 1:15 @ P~      1543.      1580.
##  7 SBIN   EQ     2018-06-15       Annual General ~       185.       189 
##  8 ICICI~ EQ     2019-07-22       Annual General ~       349.       357.
##  9 HDFC   EQ     2020-07-09       Annual General ~      1784       1820 
## 10 TCS    EQ     2020-06-03       Dividend - Rs 6~      2077.      2132 
## # ... with 6 more variables: low_price &lt;dbl&gt;, last_traded_price &lt;dbl&gt;,
## #   prev_close_price &lt;dbl&gt;, percent_change &lt;dbl&gt;, traded_quantity &lt;dbl&gt;,
## #   turnover_in_lakhs &lt;dbl&gt;</code></pre>
<pre class="r"><code># retain original column names as returned by NSE
nse_stock_top_gainers(clean_names = FALSE)</code></pre>
<pre><code>## # A tibble: 10 x 12
##    symbol series lastCorpAnnounc~ lastCorpAnnounc~ openPrice highPrice lowPrice
##    &lt;chr&gt;  &lt;chr&gt;  &lt;date&gt;           &lt;chr&gt;                &lt;dbl&gt;     &lt;dbl&gt;    &lt;dbl&gt;
##  1 INDUS~ EQ     2019-08-08       Annual General ~      464       510.     463.
##  2 HINDA~ EQ     2019-08-14       Annual General ~      144.      148      142.
##  3 AXISB~ EQ     2019-07-04       Annual General ~      422.      433.     415.
##  4 SHREE~ EQ     2020-02-24       Interim Dividen~    21171.    21611.   21150.
##  5 KOTAK~ EQ     2019-07-12       Annual General ~     1304.     1344.    1304.
##  6 RELIA~ EQ     2020-05-13       Rights 1:15 @ P~     1543.     1580.    1540.
##  7 SBIN   EQ     2018-06-15       Annual General ~      185.      189      184.
##  8 ICICI~ EQ     2019-07-22       Annual General ~      349.      357.     345.
##  9 HDFC   EQ     2020-07-09       Annual General ~     1784      1820     1782 
## 10 TCS    EQ     2020-06-03       Dividend - Rs 6~     2077.     2132     2074.
## # ... with 5 more variables: ltp &lt;dbl&gt;, previousPrice &lt;dbl&gt;, netPrice &lt;dbl&gt;,
## #   tradedQuantity &lt;dbl&gt;, turnoverInLakhs &lt;dbl&gt;</code></pre>
</section>
<section id="stocks-that-have-touched-their-52-week-highs-during-the-day" class="level4">
<h4 class="anchored" data-anchor-id="stocks-that-have-touched-their-52-week-highs-during-the-day">
Stocks that have touched their 52 week highs during the day
</h4>
<pre class="r"><code>nse_stock_year_high()</code></pre>
<pre><code>## # A tibble: 34 x 10
##    symbol symbol_desc date       new_high   year last_traded_pri~ prev_high
##    &lt;chr&gt;  &lt;chr&gt;       &lt;date&gt;        &lt;dbl&gt;  &lt;dbl&gt;            &lt;dbl&gt;     &lt;dbl&gt;
##  1 ADANI~ Adani Gree~ 2020-06-09   328.   328.             328.      313.  
##  2 ALCHEM Alchemist ~ 2020-06-09     6.95   6.95             6.95      6.65
##  3 ALOKI~ Alok Indus~ 2020-06-09    25.8   25.8             25.8      24.6 
##  4 ANDHR~ Andhra Cem~ 2020-06-09     5.35   5.35             5.35      5.1 
##  5 AUROP~ Aurobindo ~ 2020-06-04   800.   800.             796.      791   
##  6 BCG    Brightcom ~ 2020-06-09     9.95   9.95             9.95      9.5 
##  7 BIRLA~ Birla Tyre~ 2020-06-09    23.5   23.5             23.5      22.4 
##  8 COMPU~ Compucom S~ 2020-06-09    10.8   10.8             10.8      10.6 
##  9 DAAWAT LT Foods L~ 2020-06-09    37.8   37.8             37.2      37   
## 10 DHANU~ Dhanuka Ag~ 2020-06-09   653.   653.             635.      620   
## # ... with 24 more rows, and 3 more variables: prev_close &lt;dbl&gt;, change &lt;dbl&gt;,
## #   percent_change &lt;dbl&gt;</code></pre>
<pre class="r"><code># retain original column names as returned by NSE
nse_stock_year_high(clean_names = FALSE)</code></pre>
<pre><code>## # A tibble: 34 x 10
##    symbol symbolDesc dt          value   year    ltp value_old   prev change
##    &lt;chr&gt;  &lt;chr&gt;      &lt;date&gt;      &lt;dbl&gt;  &lt;dbl&gt;  &lt;dbl&gt;     &lt;dbl&gt;  &lt;dbl&gt;  &lt;dbl&gt;
##  1 ADANI~ Adani Gre~ 2020-06-09 328.   328.   328.      313.   313.    15.6 
##  2 ALCHEM Alchemist~ 2020-06-09   6.95   6.95   6.95      6.65   6.65   0.3 
##  3 ALOKI~ Alok Indu~ 2020-06-09  25.8   25.8   25.8      24.6   24.6    1.2 
##  4 ANDHR~ Andhra Ce~ 2020-06-09   5.35   5.35   5.35      5.1    5.1    0.25
##  5 AUROP~ Aurobindo~ 2020-06-04 800.   800.   796.      791    775.    21.1 
##  6 BCG    Brightcom~ 2020-06-09   9.95   9.95   9.95      9.5    9.5    0.45
##  7 BIRLA~ Birla Tyr~ 2020-06-09  23.5   23.5   23.5      22.4   22.4    1.1 
##  8 COMPU~ Compucom ~ 2020-06-09  10.8   10.8   10.8      10.6   10.3    0.5 
##  9 DAAWAT LT Foods ~ 2020-06-09  37.8   37.8   37.2      37     34.7    2.55
## 10 DHANU~ Dhanuka A~ 2020-06-09 653.   653.   635.      620    594.    41.4 
## # ... with 24 more rows, and 1 more variable: pChange &lt;dbl&gt;</code></pre>
</section>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<ul>
<li>
<a href="https://nse2r.rsquaredacademy.com">Documentation</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy/nse2r">GitHub</a>
</li>
</ul>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
All feedback is welcome. Issues (bugs and feature requests) can be posted to <a href="https://github.com/rsquaredacademy/nse2r/issues">github tracker</a>. For help with code or other related questions, feel free to reach out to us at <a href="mailto:pkgs@rsquaredacademy.com" class="email">pkgs@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>packages</category>
  <guid>https://blog.rsquaredacademy.com/posts/nse2r-fetch-data-from-nse/</guid>
  <pubDate>Fri, 17 Jan 2020 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/github_nse2r.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>A Comprehensive Introduction to Working with Databases using R</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/working-with-databases-using-r/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-08-08-working-with-databases-using-r.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/dbi_cover_image.png" width="100%" style="display: block; margin: auto;">
</p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
In a previous <a href="https://blog.rsquaredacademy.com/quick-guide-r-sqlite/" target="_blank">post</a>, we had briefly looked at connecting to databases from R and using dplyr for querying data. In this new expanded post, we will focus on the following:
</p>
<ul>
<li>
connect to &amp; explore database
</li>
<li>
read &amp; write data
</li>
<li>
use RStudio SQL script &amp; knitr SQL engine
</li>
<li>
query data using dplyr
</li>
<li>
visualize data with dbplot
</li>
<li>
modeling data with modeldb &amp; tidypredict
</li>
<li>
explore RStudio connections pane
</li>
<li>
handling credentials
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all the resources related to this post:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/sql/sqlite.html#/section" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/430439" target="_blank">RStudio Cloud</a>
</li>
</ul>
<p>
You can try our <strong>free online course</strong> <a href="https://rsquared-academy.thinkific.com/courses/working-with-databases-using-r"><strong>Working with Databases using R</strong></a> if you prefer to learn through self paced online courses.
</p>
<p>{{% youtube “bPlGPmjPFeU” %}}</p>
</section>
<section id="libraries" class="level2">
<h2 class="anchored" data-anchor-id="libraries">
Libraries
</h2>
<p>
Before we connect to and explore the local SQLite database, let us take a look at the R packages we will use in this post.
</p>
<ul>
<li>
<a href="http://r-dbi.github.io/DBI/">DBI</a> a database interface for R
</li>
<li>
<a href="https://dbplyr.tidyverse.org/">dbplyr</a> a dplyr backend for databases
</li>
<li>
<a href="https://dplyr.tidyverse.org/">dplyr</a> for querying data
</li>
<li>
<a href="https://edgararuiz.github.io/dbplot/">dbplot</a> &amp; <a href="https://ggplot2.tidyverse.org/">ggplot2</a> for data visualization
</li>
<li>
<a href="https://tidymodels.github.io/modeldb/">modeldb</a> &amp; <a href="https://tidymodels.github.io/tidypredict/">tidypredict</a> for modeling &amp; prediction inside database
</li>
<li>
<a href="https://cran.r-project.org/package=config">config</a> for handling credentials
</li>
</ul>
<pre class="r"><code># install.packages(c("DBI", "dbplyr", "dplyr", "dbplot", "ggplot2", "modeldb",
# "tidypredict", "config"))
library(DBI)
library(dbplyr)
library(dplyr)
library(dbplot)
library(ggplot2)
library(modeldb)
library(tidypredict)
library(config)</code></pre>
<p>
If you do not have all the above packages installed, go ahead and install them. In the R script we are sharing with you, we have commented out the code for installing the packages. If you are using the RStudio Cloud project, we have already installed the packages in the project and you can just load them into the R session using <code>library()</code>.
</p>
<p>
As and when we come to the specific sections where we are using these packages, they will be reintroduced and we will look at their documentation and explore the functions we will use.
</p>
<hr>
<p align="center">
<a href="https://rsquared-academy.thinkific.com/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/ad_courses.png" width="100%" alt="new courses ad" style="text-decoration: none;"></a>
</p>
<hr>
</section> ]]></description>
  <category>R</category>
  <category>databases</category>
  <guid>https://blog.rsquaredacademy.com/posts/working-with-databases-using-r/</guid>
  <pubDate>Thu, 08 Aug 2019 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/dbi_cover_image.png" medium="image" type="image/png" height="81" width="144"/>
</item>
<item>
  <title>Customer Segmentation using RFM Analysis</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-07-22-customer-segmentation-using-rfm-analysis.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<script src="../../rmarkdown-libs/kePrint/kePrint.js"></script>
<p><link href="../../rmarkdown-libs/bsTable/bootstrapTable.min.css" rel="stylesheet"></p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_cover_image.png" width="100%" style="display: block; margin: auto;">
</p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
In a previous <a href="https://blog.rsquaredacademy.com/introducing-rfm/" target="_blank">post</a>, we had introduced our R package <a href="https://rfm.rsquaredacademy.com" target="_blank">rfm</a> but did not go into the conceptual details of RFM analysis. In this post, we will explore RFM in much more depth and work through a case study as well. <strong>RFM</strong> (Recency, Frequency &amp; Monetary) analysis is a behavior based technique used to segment customers by examining their transaction history such as:
</p>
<ul>
<li>
how recently a customer has purchased?
</li>
<li>
how often do they purchase?
</li>
<li>
how much the customer spends?
</li>
</ul>
<p>
It is based on the marketing axiom that <strong>80% of your business comes from 20% of your customers</strong>. RFM helps to identify customers who are more likely to respond to promotions by segmenting them into various categories.
</p>
<p>{{% youtube “275X7yaSsoQ” %}}</p>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all resources related to this post:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/rfm/rfm.html" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/416146" target="_blank">RStudio Cloud</a>
</li>
</ul>
<p>
You can try our free online course <a href="https://rsquared-academy.thinkific.com/courses/customer-segmentation-using-rfm-analysis" target="_blank">Customer Segmentation using RFM Analysis</a> if you like to learn through self paced online courses.
</p>
</section>
<section id="case-study" class="level2">
<h2 class="anchored" data-anchor-id="case-study">
Case Study
</h2>
<p>
We will work through a case study to better understand the underlying concepts of RFM analysis. To pique your curiosity, we will start with the results or the final outcome of the case study as shown below:
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_segments.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
The table has the following details:
</p>
<ul>
<li>
name or id of the customer
</li>
<li>
number of days since the last transaction of the customer
</li>
<li>
number of transactions of the customer
</li>
<li>
total value of the all the transactions of the customer
</li>
<li>
RFM score
</li>
<li>
customer segment
</li>
</ul>
<p>
The rest of this post will focus on generating a similar result for our case study and along the way we will learn to:
</p>
<ul>
<li>
structure data for RFM analysis
</li>
<li>
generate RFM score
</li>
<li>
and segment customers using RFM score
</li>
</ul>
</section>
<section id="applications" class="level2">
<h2 class="anchored" data-anchor-id="applications">
Applications
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_applications.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
Let us talk about applications. Though largely identified with retail or ecommerce, RFM analysis can be applied in a lot of other domains or industry as well. In social media and apps, RFM can be used to segment users as well. The only difference is instead of using monetary value as the third metric, we will use the amount of time spent (or some other metric based on it) on the site/app. The more time we spend on the platform and consume the content, the more ads can be displayed by the platform. So in those cases, the amount of time we spend will be the third metric.
</p>
</section>
<section id="rfm-workflow" class="level2">
<h2 class="anchored" data-anchor-id="rfm-workflow">
RFM Workflow
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_work_flow.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
The typical workflow for RFM analysis is shown above. It can be broadly divided into the following:
</p>
<ul>
<li>
<p>
In the first step of the workflow, we <strong>collect transaction data</strong>. This should include a unique customer id, transaction date and transaction amount. In case of ecommerce, we need to decide how to treat visits that did not result in a transaction. If data is aggregated and made available at the customer level, it must include a unique customer id, last transaction date and total revenue from the customer. The last transaction date may be replaced by days since last visit as well. The details available in data supplied depends on the data pipeline and the <code>rfm</code> package can handle any of the above 3 scenarios.
</p>
</li>
<li>
<p>
In the second step, we <strong>generate RFM table</strong> from the raw data available. The RFM table aggregates data at the customer level. It includes the unique customer id, days since last transaction/visit, frequency of transactions/visits and the total revenue from all the transactions of the customer.
</p>
</li>
<li>
<p>
In the third step, we <strong>generate scores</strong> for recency, frequency and monetary value, and use them to create the RFM score for each customer.
</p>
</li>
<li>
<p>
In the final step, we use the recency, frequency and monetary scores to <strong>define customer segments</strong> and design customised campaigns, promotions, offers &amp; discounts to retain and reactivate customers.
</p>
</li>
</ul>
</section>
<section id="rfm-table" class="level2">
<h2 class="anchored" data-anchor-id="rfm-table">
RFM Table
</h2>
<p>
Let us assume we have completed the first step in RFM analysis by collecting transaction data. Now, we have to generate the RFM table from the transaction data. In the transaction data, each row represents a transaction and we may get the transaction details in any of the following ways:
</p>
<ul>
<li>
according to transaction date
</li>
<li>
sorted by customer id
</li>
<li>
or in a random order
</li>
</ul>
<p>
The first two cases are more likely but if we get the transaction data in a random order, the first order of business is to sort them by customer id. In the below example, we have transaction details for 3 customers Lionel, Jaineel and Taj but they are not sorted by transaction date or customer id. Since we want to create the RFM table from this data, we sort it by customer id.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_raw_sorted.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
From the sorted data, we aggregate the transaction details at the customer level as shown below.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_raw_sorted_aggregate.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
From the above step, we have created the RFM table which contains recency (days since last visit), frequency (frequency of visits) and monetary (total revenue from the customer) data for each customer.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_table.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
<section id="metrics" class="level2">
<h2 class="anchored" data-anchor-id="metrics">
Metrics
</h2>
<p>
Time to talk about the key metrics <strong>R</strong>ecency, <strong>F</strong>requency and <strong>M</strong>onetary in more detail. In this section, we will understand how they are calculated, and in the next section, we will learn how they are used for generating the RFM score.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_segment_4.png" width="100%" style="display: block; margin: auto;">
</p>
<section id="recency" class="level3">
<h3 class="anchored" data-anchor-id="recency">
Recency
</h3>
<p>
Let us begin with recency. Earlier, we defined it as the number of days since the last transaction of the customer. How do we calculate this metric? Apart from the date of the last transaction of the customer, what other information do we need? In all the discussions till now, we have missed out on a key point i.e.&nbsp;the time frame of the analysis.
</p>
<p>
The most crucial step in RFM analysis is to select a time frame from which we use the transaction data. How do we decide on this time frame? It depends on the domain to which we are applying this analysis. Customers visit a grocery store more often than they visit a consumer durables store. Similarly, people consume content from news &amp; blogs more frequently while they may visit an e-commerce website only when they have to purchase something. Keeping in mind the domain to which the analysis is being applied, select an appropriate time frame. To calculate recency, compute the difference between the last transaction date and the analysis date i.e.&nbsp;the last date of the selected time frame.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_recency_computation.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
In the above example, the analysis date is <code>2016-12-31</code>. To compute recency, we first extract all the transaction date of customer Taj and then select the last transaction date, <code>2015-04-21</code>, and subtract if from the analysis date to get the number of days since the last transaction date, <code>620</code>.
</p>
</section>
<section id="frequency" class="level3">
<h3 class="anchored" data-anchor-id="frequency">
Frequency
</h3>
<p>
Frequency is the count of transactions. In the online/digital world, we need to decide whether we will consider all the visits to a website or app as the frequency or only those which resulted in a transaction/conversion. In the below example, we count the transactions for each customer and use it as frequency. Lionel has 6 transactions, Jaineel has 9 transactions and Taj has 4 transactions.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_frequency_computation.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
<section id="monetary-value" class="level3">
<h3 class="anchored" data-anchor-id="monetary-value">
Monetary Value
</h3>
<p>
Monetary value is the total revenue from each customer in the selected time frame. It is computed by summing up the transaction amount. In our case study, Jaineel has spent the highest amount of <code>$843</code>, followed by Lionel who has spent <code>$472</code> and Taj has spent the lowest, <code>$196</code>. As you can observe, we have arrived at these figures by summing the values in the third column of the second table, <code>Order Amount</code>.
</p>
<p>
If we are applying this analysis to the digital world, we may think of using a metric such as time spent on the website/app instead of transaction/order amount.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_monetary_computation.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
</section>
<section id="rfm-score" class="level2">
<h2 class="anchored" data-anchor-id="rfm-score">
RFM Score
</h2>
<p>
As shown in the workflow, the third step in RFM analysis is to generate the individual score for each metric and then use them to generate the RFM score. In this section, we will explain in detail how the scores are computed for recency, frequency and monetary. This section is slightly complex (we received a lot of mails from readers after we published the previous post) and we have tried our best to break down the complexity as much as possible. Still, if you have any questions feel free to write to us at <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a>.
</p>
<p>
We follow the below steps to create the score:
</p>
<ul>
<li>
use quantiles to generate cut off points
</li>
<li>
create intervals based on the cut off points
</li>
<li>
use the intervals to assign score
</li>
</ul>
<section id="monetary-score" class="level3">
<h3 class="anchored" data-anchor-id="monetary-score">
Monetary Score
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_monetary_score_interval.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
Let us generate the monetary score in our case study. The first step is to compute the quantiles using the <code>quantile()</code> function. We use the revenue column from the RFM table to compute the quantiles. If you look at the example, it gives us the cut off below which a certain percentage of customers are present.
</p>
<ul>
<li>
the bottom 20% of customers spend below <code>$254.8</code>.
</li>
<li>
the next 20% of customers spend between <code>$254.8</code> and <code>$381.0</code>.
</li>
<li>
the top 20% of customers spend above <code>$665.0</code>.
</li>
</ul>
<p>
Using these cut off points we have created intervals which can then be converted to <code>if else</code> statements. The intervals are then used to assign scores. For example, Lionel falls in the interval <code>&gt; 381.0 &amp; &lt;= 505.4</code> and hence is assigned the score <code>3</code>. Similarly, Jaineel and Taj are assigned the scores <code>5</code> and <code>1</code>. How do we interpret the scores? The score is more like a rank. A customer with a score of 3 is ranked higher than a customer with score of 1 as his transaction amount is higher. In the <code>rfm</code> package, we use the above method to assign the scores.
</p>
<p>
Some users reverse the order of the score i.e.&nbsp;top 20% customers by transaction amountare assigned the score 1 and the bottom 20% are assigned the score 5.
</p>
</section>
<section id="frequency-score" class="level3">
<h3 class="anchored" data-anchor-id="frequency-score">
Frequency Score
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_frequency_score_interval.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
The frequency score is computed in the same way as the monetary score. Instead of using the revenue column from the RFM table, we use the frequency column. Using quantiles, we arrive at the cut off points below which a certain percentage of customer are present. If you observe the example, the first table shows the quantiles and the associated cut off points.
</p>
<ul>
<li>
the bottom 20% of customers visit/transact less than 3 times.
</li>
<li>
the next 20% of customers visit/transact around 4 times.
</li>
<li>
the top 20% of customers visit/transact more than 7 times.
</li>
</ul>
<p>
The cut off points are then used to create the intervals and assign the scores as shown in the second table. We assign a higher score to those who visit more frequently and a lower score to those who visit less frequently.
</p>
<p>
In our case study, Jaineel has visited 9 times and hence assigned the score 5 where as Taj has visited only 4 times and hence the score 2.
</p>
</section>
<section id="recency-score" class="level3">
<h3 class="anchored" data-anchor-id="recency-score">
Recency Score
</h3>
<p>
The recency score follows the same methodology but uses a slightly different concept while assigning the score. If you look at the metrics, the higher the values of frequency and monetary, the better as we want customers to transact frequently and spend higher amount but it is not the case with recency. Since recency represents the number of days since the last transaction, the lower its value the better i.e.&nbsp;customers who visited in the recent past are more likely to visit again whereas customers who visited long back may be as good as lost. Hence in the case of recency, higher score is assigned to those with lower recency value and vice versa.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_recency_score_interval.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
In the above example, we have used quantiles to compute the cut off point for recency. The first table shows the quantiles and the associated cut off points:
</p>
<ul>
<li>
the bottom 20% of customers visited more than <code>481</code> days back.
</li>
<li>
the next 20% of customers visited between <code>296.4</code> and <code>481</code> days back.
</li>
<li>
the top 20% of customers visited less than 114 days back.
</li>
</ul>
<p>
The above statements will become clear if you study the second table which includes the interval and the score. We have assigned a higher score to those who visited in the recent past (&lt; 114 days) compared to those who visited way back (&gt; 481 days). In our case study, Jaineel visited in the past 3 months and hence the score of 5 where as Taj visited almost 20 months back and has been assigned the score 1.
</p>
</section>
<section id="rfm-score-1" class="level3">
<h3 class="anchored" data-anchor-id="rfm-score-1">
RFM Score
</h3>
<p>
Now that we have calculated the individual scores, let us compute the RFM score using the below formula:
</p>
<p>
<code>RFM Score = Recency Score * 100 + Frequency Score * 10 + Monetary Score</code>
</p>
<p>
The below table shows the individal scores of recency, frequency and monetary as well as the RFM score. All of them are computed from the RFM table which in itself is based on the transaction data.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_score_table.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
</section>
<section id="segments" class="level2">
<h2 class="anchored" data-anchor-id="segments">
Segments
</h2>
<p>
Great! We have finally computed the RFM score. Now what? How do we define the segments using this score? In this section, we will learn how to define customer segments using the RFM score. The below table is an example of how segments are defined. It has the following details:
</p>
<ul>
<li>
the name of the segment
</li>
<li>
the definition of the segment
</li>
<li>
the intervals for the recency, frequency &amp; monetary scores
</li>
</ul>
<p>
We should be careful while creating the intervals for the scores in the segments table. Look at <a href="https://en.wikipedia.org/wiki/Interval_(mathematics)#Classification_of_intervals" target="_blank">this</a> Wkipedia article to understand how intervals work.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_segments_table.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
Let us apply the above rules to our case study.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/rfm_segments.png" width="100%" style="display: block; margin: auto;">
</p>
<p>
Defining segments is another crucial step in RFM analysis. We need to ensure that there is no duplication or large number of customers get classified into <code>Others</code> segment. In the case study, we will show you some of the mistakes that can happen while defining the segments.
</p>
</section>
<section id="case-study-1" class="level2">
<h2 class="anchored" data-anchor-id="case-study-1">
Case Study
</h2>
<p>
It is time to work through the case study. Let us first load all the libraries we will use as shown below:
</p>
<pre class="r"><code>library(rfm)
library(dplyr)
library(magrittr)
library(lubridate)</code></pre>
<section id="data" class="level3">
<h3 class="anchored" data-anchor-id="data">
Data
</h3>
<p>
To calculate the RFM score for each customer we need transaction data which should include the following:
</p>
<ul>
<li>
a unique customer id
</li>
<li>
date of transaction/order
</li>
<li>
transaction/order amount
</li>
</ul>
<p>
<code>rfm</code> includes a sample data set <code>rfm_data_orders</code> which includes the above details:
</p>
<pre class="r"><code>rfm_data_orders</code></pre>
<pre><code>## # A tibble: 4,906 x 3
##    customer_id         order_date revenue
##    &lt;chr&gt;               &lt;date&gt;       &lt;dbl&gt;
##  1 Mr. Brion Stark Sr. 2004-12-20      32
##  2 Ethyl Botsford      2005-05-02      36
##  3 Hosteen Jacobi      2004-03-06     116
##  4 Mr. Edw Frami       2006-03-15      99
##  5 Josef Lemke         2006-08-14      76
##  6 Julisa Halvorson    2005-05-28      56
##  7 Judyth Lueilwitz    2005-03-09     108
##  8 Mr. Mekhi Goyette   2005-09-23     183
##  9 Hansford Moen PhD   2005-09-07      30
## 10 Fount Flatley       2006-04-12      13
## # ... with 4,896 more rows</code></pre>
</section>
<section id="rfm-score-2" class="level3">
<h3 class="anchored" data-anchor-id="rfm-score-2">
RFM Score
</h3>
<p>
Use <code>rfm_table_order()</code> to generate the score for each customer from the sample data set <code>rfm_data_orders</code>.
</p>
<p>
<code>rfm_table_order()</code> takes 8 inputs:
</p>
<ul>
<li>
<code>data</code>: a data set with
<ul>
<li>
unique customer id
</li>
<li>
date of transaction
</li>
<li>
and amount
</li>
</ul>
</li>
<li>
<code>customer_id</code>: name of the customer id column
</li>
<li>
<code>order_date</code>: name of the transaction date column
</li>
<li>
<code>revenue</code>: name of the transaction amount column
</li>
<li>
<code>analysis_date</code>: date of analysis
</li>
<li>
<code>recency_bins</code>: number of rankings for recency score (default is 5)
</li>
<li>
<code>frequency_bins</code>: number of rankings for frequency score (default is 5)
</li>
<li>
<code>monetary_bins</code>: number of rankings for monetary score (default is 5)
</li>
</ul>
</section>
<section id="rfm-table-1" class="level3">
<h3 class="anchored" data-anchor-id="rfm-table-1">
RFM Table
</h3>
<pre class="r"><code>analysis_date &lt;- lubridate::as_date("2006-12-31", tz = "UTC")
rfm_result &lt;- rfm_table_order(rfm_data_orders, customer_id, order_date, revenue, analysis_date)
rfm_result</code></pre>
<pre><code>## Warning: `tz` argument is ignored by `as_date()`</code></pre>
<table class="table" style="margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
customer_id
</th>
<th style="text-align:left;">
date_most_recent
</th>
<th style="text-align:right;">
recency_days
</th>
<th style="text-align:right;">
transaction_count
</th>
<th style="text-align:right;">
amount
</th>
<th style="text-align:right;">
recency_score
</th>
<th style="text-align:right;">
frequency_score
</th>
<th style="text-align:right;">
monetary_score
</th>
<th style="text-align:right;">
rfm_score
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
Abbey O’Reilly DVM
</td>
<td style="text-align:left;">
2006-06-09
</td>
<td style="text-align:right;">
205
</td>
<td style="text-align:right;">
6
</td>
<td style="text-align:right;">
472
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
343
</td>
</tr>
<tr>
<td style="text-align:left;">
Add Senger
</td>
<td style="text-align:left;">
2006-08-13
</td>
<td style="text-align:right;">
140
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
340
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
1
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
412
</td>
</tr>
<tr>
<td style="text-align:left;">
Aden Lesch Sr.&nbsp;
</td>
<td style="text-align:left;">
2006-06-20
</td>
<td style="text-align:right;">
194
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
405
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
323
</td>
</tr>
<tr>
<td style="text-align:left;">
Admiral Senger
</td>
<td style="text-align:left;">
2006-08-21
</td>
<td style="text-align:right;">
132
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
448
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
433
</td>
</tr>
<tr>
<td style="text-align:left;">
Agness O’Keefe
</td>
<td style="text-align:left;">
2006-10-02
</td>
<td style="text-align:right;">
90
</td>
<td style="text-align:right;">
9
</td>
<td style="text-align:right;">
843
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
555
</td>
</tr>
<tr>
<td style="text-align:left;">
Aileen Barton
</td>
<td style="text-align:left;">
2006-10-08
</td>
<td style="text-align:right;">
84
</td>
<td style="text-align:right;">
9
</td>
<td style="text-align:right;">
763
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
555
</td>
</tr>
<tr>
<td style="text-align:left;">
Ailene Hermann
</td>
<td style="text-align:left;">
2006-03-25
</td>
<td style="text-align:right;">
281
</td>
<td style="text-align:right;">
8
</td>
<td style="text-align:right;">
699
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
5
</td>
<td style="text-align:right;">
355
</td>
</tr>
<tr>
<td style="text-align:left;">
Aiyanna Bruen PhD
</td>
<td style="text-align:left;">
2006-04-29
</td>
<td style="text-align:right;">
246
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
157
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
1
</td>
<td style="text-align:right;">
321
</td>
</tr>
<tr>
<td style="text-align:left;">
Ala Schmidt DDS
</td>
<td style="text-align:left;">
2006-01-16
</td>
<td style="text-align:right;">
349
</td>
<td style="text-align:right;">
3
</td>
<td style="text-align:right;">
363
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
1
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
212
</td>
</tr>
<tr>
<td style="text-align:left;">
Alannah Borer
</td>
<td style="text-align:left;">
2005-04-21
</td>
<td style="text-align:right;">
619
</td>
<td style="text-align:right;">
4
</td>
<td style="text-align:right;">
196
</td>
<td style="text-align:right;">
1
</td>
<td style="text-align:right;">
2
</td>
<td style="text-align:right;">
1
</td>
<td style="text-align:right;">
121
</td>
</tr>
</tbody>
</table>
<p>
<code>rfm_table_order()</code> will return the following columns as seen in the above table:
</p>
<ul>
<li>
<code>customer_id</code>: unique customer id
</li>
<li>
<code>date_most_recent</code>: date of most recent visit
</li>
<li>
<code>recency_days</code>: days since the most recent visit
</li>
<li>
<code>transaction_count</code>: number of transactions of the customer
</li>
<li>
<code>amount</code>: total revenue generated by the customer
</li>
<li>
<code>recency_score</code>: recency score of the customer
</li>
<li>
<code>frequency_score</code>: frequency score of the customer
</li>
<li>
<code>monetary_score</code>: monetary score of the customer
</li>
<li>
<code>rfm_score</code>: RFM score of the customer
</li>
</ul>
</section>
<section id="segments-1" class="level3">
<h3 class="anchored" data-anchor-id="segments-1">
Segments
</h3>
<p>
Let us classify our customers based on the individual recency, frequency and monetary scores.
</p>
<table class="table" style="font-size: 12px; margin-left: auto; margin-right: auto;">
<thead>
<tr>
<th style="text-align:left;">
Segment
</th>
<th style="text-align:left;">
Description
</th>
<th style="text-align:left;">
R
</th>
<th style="text-align:left;">
F
</th>
<th style="text-align:left;">
M
</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">
Champions
</td>
<td style="text-align:left;">
Bought recently, buy often and spend the most
</td>
<td style="text-align:left;">
4 - 5
</td>
<td style="text-align:left;">
4 - 5
</td>
<td style="text-align:left;">
4 - 5
</td>
</tr>
<tr>
<td style="text-align:left;">
Loyal Customers
</td>
<td style="text-align:left;">
Spend good money. Responsive to promotions
</td>
<td style="text-align:left;">
2 - 4
</td>
<td style="text-align:left;">
3 - 4
</td>
<td style="text-align:left;">
4 - 5
</td>
</tr>
<tr>
<td style="text-align:left;">
Potential Loyalist
</td>
<td style="text-align:left;">
Recent customers, spent good amount, bought more than once
</td>
<td style="text-align:left;">
3 - 5
</td>
<td style="text-align:left;">
1 - 3
</td>
<td style="text-align:left;">
1 - 3
</td>
</tr>
<tr>
<td style="text-align:left;">
New Customers
</td>
<td style="text-align:left;">
Bought more recently, but not often
</td>
<td style="text-align:left;">
4 - 5
</td>
<td style="text-align:left;">
&lt; 2
</td>
<td style="text-align:left;">
&lt; 2
</td>
</tr>
<tr>
<td style="text-align:left;">
Promising
</td>
<td style="text-align:left;">
Recent shoppers, but haven’t spent much
</td>
<td style="text-align:left;">
3 - 4
</td>
<td style="text-align:left;">
&lt; 2
</td>
<td style="text-align:left;">
&lt; 2
</td>
</tr>
<tr>
<td style="text-align:left;">
Need Attention
</td>
<td style="text-align:left;">
Above average recency, frequency &amp; monetary values
</td>
<td style="text-align:left;">
3 - 4
</td>
<td style="text-align:left;">
3 - 4
</td>
<td style="text-align:left;">
3 - 4
</td>
</tr>
<tr>
<td style="text-align:left;">
About To Sleep
</td>
<td style="text-align:left;">
Below average recency, frequency &amp; monetary values
</td>
<td style="text-align:left;">
2 - 3
</td>
<td style="text-align:left;">
&lt; 3
</td>
<td style="text-align:left;">
&lt; 3
</td>
</tr>
<tr>
<td style="text-align:left;">
At Risk
</td>
<td style="text-align:left;">
Spent big money, purchased often but long time ago
</td>
<td style="text-align:left;">
&lt; 3
</td>
<td style="text-align:left;">
2 - 5
</td>
<td style="text-align:left;">
2 - 5
</td>
</tr>
<tr>
<td style="text-align:left;">
Can’t Lose Them
</td>
<td style="text-align:left;">
Made big purchases and often, but long time ago
</td>
<td style="text-align:left;">
&lt; 2
</td>
<td style="text-align:left;">
4 - 5
</td>
<td style="text-align:left;">
4 - 5
</td>
</tr>
<tr>
<td style="text-align:left;">
Hibernating
</td>
<td style="text-align:left;">
Low spenders, low frequency, purchased long time ago
</td>
<td style="text-align:left;">
2 - 3
</td>
<td style="text-align:left;">
2 - 3
</td>
<td style="text-align:left;">
2 - 3
</td>
</tr>
<tr>
<td style="text-align:left;">
Lost
</td>
<td style="text-align:left;">
Lowest recency, frequency &amp; monetary scores
</td>
<td style="text-align:left;">
&lt; 2
</td>
<td style="text-align:left;">
&lt; 2
</td>
<td style="text-align:left;">
&lt; 2
</td>
</tr>
</tbody>
</table>
</section>
<section id="segmented-customer-data" class="level3">
<h3 class="anchored" data-anchor-id="segmented-customer-data">
Segmented Customer Data
</h3>
<p>
We can use the segmented data to identify
</p>
<ul>
<li>
champion customers
</li>
<li>
loyal customers
</li>
<li>
at risk customers
</li>
<li>
and lost customers
</li>
</ul>
<p>
Once we have classified a customer into a particular segment, we can take appropriate action to increase his/her lifetime value.
</p>
<pre class="r"><code>segment_names &lt;- 
  c("Champions", "Loyal Customers", "Potential Loyalist",
    "New Customers", "Promising", "Need Attention", 
    "About To Sleep", "At Risk", "Can't Lose Them", 
    "Hibernating", "Lost")

recency_lower   &lt;- c(4, 2, 3, 4, 3, 3, 2, 1, 1, 2, 1)
recency_upper   &lt;- c(5, 4, 5, 5, 4, 4, 3, 2, 1, 3, 1)
frequency_lower &lt;- c(4, 3, 1, 1, 1, 3, 1, 2, 4, 2, 1)
frequency_upper &lt;- c(5, 4, 3, 1, 1, 4, 2, 5, 5, 3, 1)
monetary_lower  &lt;- c(4, 4, 1, 1, 1, 3, 1, 2, 4, 2, 1)
monetary_upper  &lt;- c(5, 5, 3, 1, 1, 4, 2, 5, 5, 3, 1)

segments &lt;- 
  rfm_segment(rfm_result, segment_names, recency_lower, 
              recency_upper, frequency_lower, frequency_upper, 
              monetary_lower, monetary_upper)

segments %&gt;% 
  select(customer_id, segment, rfm_score)</code></pre>
<pre><code>## # A tibble: 995 x 3
##    customer_id        segment            rfm_score
##    &lt;chr&gt;              &lt;chr&gt;                  &lt;dbl&gt;
##  1 Abbey O'Reilly DVM Need Attention           343
##  2 Add Senger         Potential Loyalist       412
##  3 Aden Lesch Sr.     Hibernating              323
##  4 Admiral Senger     Need Attention           433
##  5 Agness O'Keefe     Champions                555
##  6 Aileen Barton      Champions                555
##  7 Ailene Hermann     Others                   355
##  8 Aiyanna Bruen PhD  About To Sleep           321
##  9 Ala Schmidt DDS    About To Sleep           212
## 10 Alannah Borer      Others                   121
## # ... with 985 more rows</code></pre>
</section>
<section id="segment-size" class="level3">
<h3 class="anchored" data-anchor-id="segment-size">
Segment Size
</h3>
<p>
Now that we have defined and segmented our customers, let us examine the distribution of customers across the segments. If our segmentation logic is good, few or no customer should be categorized as <code>Others</code>.
</p>
<pre class="r"><code>segments %&gt;%
  count(segment) %&gt;%
  arrange(desc(n)) %&gt;%
  rename(Segment = segment, Count = n)</code></pre>
<pre><code>## # A tibble: 12 x 2
##    Segment            Count
##    &lt;chr&gt;              &lt;int&gt;
##  1 At Risk              157
##  2 Potential Loyalist   132
##  3 Others               128
##  4 Champions            116
##  5 Need Attention       100
##  6 Hibernating           97
##  7 About To Sleep        92
##  8 Lost                  75
##  9 Loyal Customers       43
## 10 Promising             21
## 11 Can't Lose Them       17
## 12 New Customers         17</code></pre>
<p>
We can also examine the median recency, frequency and monetary value across segments to ensure that the logic used for customer classification is sound and practical.
</p>
<section id="median-recency" class="level4">
<h4 class="anchored" data-anchor-id="median-recency">
Median Recency
</h4>
<pre class="r"><code>rfm_plot_median_recency(segments)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/avg_recency-1.png" width="576" style="display: block; margin: auto;">
</p>
</section>
<section id="median-frequency" class="level4">
<h4 class="anchored" data-anchor-id="median-frequency">
Median Frequency
</h4>
<pre class="r"><code>rfm_plot_median_frequency(segments)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/avg_frequency-1.png" width="576" style="display: block; margin: auto;">
</p>
</section>
<section id="median-monetary-value" class="level4">
<h4 class="anchored" data-anchor-id="median-monetary-value">
Median Monetary Value
</h4>
<pre class="r"><code>rfm_plot_median_monetary(segments)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/avg_monetary-1.png" width="576" style="display: block; margin: auto;">
</p>
</section>
</section>
<section id="heat-map" class="level3">
<h3 class="anchored" data-anchor-id="heat-map">
Heat Map
</h3>
<p>
The heat map shows the average monetary value for different categories of recency and frequency scores. Higher scores of frequency and recency are characterized by higher average monetary value as indicated by the darker areas in the heatmap.
</p>
<pre class="r"><code>rfm_heatmap(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/heatmap-1.png" width="768" style="display: block; margin: auto;">
</p>
</section>
<section id="bar-chart" class="level3">
<h3 class="anchored" data-anchor-id="bar-chart">
Bar Chart
</h3>
<p>
Use <code>rfm_bar_chart()</code> to generate the distribution of monetary scores for the different combinations of frequency and recency scores.
</p>
<pre class="r"><code>rfm_bar_chart(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/barchart-1.png" width="768" style="display: block; margin: auto;">
</p>
</section>
<section id="histogram" class="level3">
<h3 class="anchored" data-anchor-id="histogram">
Histogram
</h3>
<p>
Use <code>rfm_histograms()</code> to examine the relative distribution of
</p>
<ul>
<li>
monetary value (total revenue generated by each customer)
</li>
<li>
recency days (days since the most recent visit for each customer)
</li>
<li>
frequency (transaction count for each customer)
</li>
</ul>
<pre class="r"><code>rfm_histograms(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/rfmhist-1.png" width="768" style="display: block; margin: auto;">
</p>
</section>
<section id="customers-by-orders" class="level3">
<h3 class="anchored" data-anchor-id="customers-by-orders">
Customers by Orders
</h3>
<p>
Visualize the distribution of customers across orders.
</p>
<pre class="r"><code>rfm_order_dist(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/rfmorders-1.png" width="768" style="display: block; margin: auto;">
</p>
</section>
<section id="scatter-plots" class="level3">
<h3 class="anchored" data-anchor-id="scatter-plots">
Scatter Plots
</h3>
<p>
The best customers are those who:
</p>
<ul>
<li>
bought most recently
</li>
<li>
most often
</li>
<li>
and spend the most
</li>
</ul>
<p>
Now let us examine the relationship between the above.
</p>
<section id="recency-vs-monetary-value" class="level4">
<h4 class="anchored" data-anchor-id="recency-vs-monetary-value">
Recency vs Monetary Value
</h4>
<p>
Customers who visited more recently generated more revenue compared to those who visited in the distant past. The customers who visited in the recent past are more likely to return compared to those who visited long time ago as most of those would be lost customers. As such, higher revenue would be associated with most recent visits.
</p>
<pre class="r"><code>rfm_rm_plot(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/mr-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
<section id="frequency-vs-monetary-value" class="level4">
<h4 class="anchored" data-anchor-id="frequency-vs-monetary-value">
Frequency vs Monetary Value
</h4>
<p>
As the frequency of visits increases, the revenue generated also increases. Customers who visit more frquently are your champion customers, loyal customers or potential loyalists and they drive higher revenue.
</p>
<pre class="r"><code>rfm_fm_plot(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/fm-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
<section id="recency-vs-frequency" class="level4">
<h4 class="anchored" data-anchor-id="recency-vs-frequency">
Recency vs Frequency
</h4>
<p>
Customers with low frequency visited in the distant past while those with high frequency have visited in the recent past. Again, the customers who visited in the recent past are more likely to return compared to those who visited long time ago. As such, higher frequency would be associated with the most recent visits.
</p>
<pre class="r"><code>rfm_rf_plot(rfm_result)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/2019-07-22-customer-segmentation-using-rfm-analysis_files/figure-html/fr-1.png" width="672" style="display: block; margin: auto;">
</p>
</section>
</section>
<section id="your-turn" class="level3">
<h3 class="anchored" data-anchor-id="your-turn">
Your Turn…
</h3>
<ul>
<li>
<p>
if you look at the distribution of segments, around 13% of the customers are in the <code>Others</code> segment For segmentation to be effective and optimal, the <code>Others</code> segment should be eliminated or should have few customers only. Redefine the segments and try to reduce the number of customers in the <code>Others</code> segment.
</p>
</li>
<li>
<p>
we have defined 11 segments. Try to combine some of the existing segments and bring down the total segments to around 6 or 8.
</p>
</li>
<li>
<p>
the RFM score we generated uses score between 1 and 5. Try to create segments by using a score between 1 and 3 i.e.&nbsp;the lowest RFM should be 111 and the highest should be 333 instead of 555.
</p>
</li>
<li>
<p>
reverse the scores i.e.&nbsp;so far we have assigned a score of 5 to customers who visited recently, frequently and had higher transaction amount and 1 to customers who visited way back, rarely and have low transaction amount. Reverse this score pattern and create the segments.
</p>
</li>
</ul>
</section>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<p>
The <a href="https://rfm.rsquaredacademy.com/index.html" target="_blank">rfm website</a> includes comprehensive documentation on using the package, including the following articles that cover various aspects of using rfm:
</p>
<ul>
<li>
<p>
<a href="https://rfm.rsquaredacademy.com/articles/rfm-customer-level-data.html" target="_blank">RFM Customer Level Data</a> - shows you how to handle customer level data.
</p>
</li>
<li>
<p>
<a href="https://rfm.rsquaredacademy.com/articles/rfm-transaction-level-data.html" target="_blank">RFM Transaction Level Data</a> - shows you how to handle transaction level data.
</p>
</li>
</ul>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
If you see mistakes or want to suggest changes, please create an issue on the <a href="https://github.com/rsquaredacademy-infra/blog" target="_blank">source repository</a> or reach out to us at <a href="mailto:support@rsquaredacademy.com" class="email">support@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>rfm</category>
  <guid>https://blog.rsquaredacademy.com/posts/customer-segmentation-using-rfm-analysis/</guid>
  <pubDate>Mon, 22 Jul 2019 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/rfm_cover_image.png" medium="image" type="image/png" height="89" width="144"/>
</item>
<item>
  <title>pkginfo: Tools for Retrieving R Package Information</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/introducing-pkginfo/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-07-05-pkginfo.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/github_pkginfo.png" width="100%" style="display: block; margin: auto;">
</p>
<section id="motivation" class="level2">
<h2 class="anchored" data-anchor-id="motivation">
Motivation
</h2>
<p>
There are several wonderful tools for retrieving information about R packages, some of which are listed below:
</p>
<ul>
<li>
<strong><a href="https://github.com/r-hub/cranlogs">cranlogs</a></strong>, <strong><a href="https://github.com/GuangchuangYu/dlstats">dlstats</a></strong> and <strong><a href="https://github.com/lindbrook/packageRank">packageRank</a></strong> for R package download stats
</li>
<li>
<strong><a href="https://github.com/r-hub/pkgsearch">pkgsearch</a></strong> and <strong><a href="https://github.com/jsugarelli/packagefinder/">packagefinder</a></strong> for searching CRAN R packages
</li>
<li>
<strong><a href="https://github.com/r-hub/crandb">crandb</a></strong> provides API for programatically accessing meta-data
</li>
<li>
<strong><a href="https://github.com/ropenscilabs/cchecks">cchecks</a></strong> for CRAN check results
</li>
</ul>
<p>
We have used some or all of these to track/monitor our own <a href="https://pkgs.rsquaredacademy.com">R packages</a> available on CRAN. Over time, we wanted to have a single interface which would retrieve information from different places including:
</p>
<ul>
<li>
check results from <a href="https://cran.r-project.org/">CRAN</a>
</li>
<li>
issues &amp; pull requests from <a href="https://github.com/">GitHub</a>
</li>
<li>
build status from <a href="https://travis-ci.org/">Travis</a> and <a href="https://ci.appveyor.com/">Appveyor</a>
</li>
<li>
code coverage from <a href="https://codecov.io/">Codecov</a>
</li>
<li>
questions tagged with the package name in <a href="https://stackoverflow.com/">Stack Overflow</a>
</li>
</ul>
<p>
And below is a demo of what we came up with…
</p>
{{% youtube “g7Lcmgiyzmo” %}}
<p>
The above shiny app can be launched with:
</p>
<pre class="r"><code>pkginfo::pkginfo_dashboard()</code></pre>
</section>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
<a href="https://github.com/rsquaredacademy/pkginfo">pkginfo</a> will allow you to retrieve information from multiple places. The shiny app is especially useful if you are looking for information about multiple R packages. <a href="https://github.com/rsquaredacademy/pkginfo">pkginfo</a> provides two APIs, one is object oriented, the other one is procedural, and in this post, we will show you how to use them.
</p>
</section>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">
Installation
</h2>
<p>
You can install the development version from <a href="https://github.com/rsquaredacademy/pkginfo">GitHub</a> with:
</p>
<pre class="r"><code>devtools::install_github("rsquaredacademy/pkginfo")</code></pre>
<hr>
<p>
<a href="https://www.rsquaredacademy.com/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/ad.png" width="100%" alt="course ad" style="text-decoration: none;"></a>
</p>
<hr>
</section> ]]></description>
  <category>R</category>
  <category>packages</category>
  <guid>https://blog.rsquaredacademy.com/posts/introducing-pkginfo/</guid>
  <pubDate>Fri, 05 Jul 2019 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/github_pkginfo.png" medium="image" type="image/png" height="72" width="144"/>
</item>
<item>
  <title>Demystifying Regular Expressions in R</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/regular-expression-in-r/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-05-27-regex.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_cover_image.png" width="100%" style="display: block; margin: auto;">
</p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
In this post, we will learn about using regular expressions in R. While it is aimed at absolute beginners, we hope experienced users will find it useful as well. The post is broadly divided into 3 sections. In the first section, we will introduce the pattern matching functions such as <code>grep</code>, <code>grepl</code> etc. in base R as we will be using them in the rest of the post. Once the reader is comfortable with the above mentioned pattern matching functions, we will learn about regular expressions while exploring the names of R packages by probing the following:
</p>
<ul>
<li>
how many package names include the letter <code>r</code>?
</li>
<li>
how many package names begin or end with the letter <code>r</code>?
</li>
<li>
how many package names include the words <code>data</code> or <code>plot</code>?
</li>
</ul>
<p>
In the final section, we will go through 4 case studies including simple email validation. If you plan to try the case studies, please do not skip any of the topics in the second section.
</p>
<section id="what" class="level3">
<h3 class="anchored" data-anchor-id="what">
What
</h3>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_what.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
<section id="why" class="level3">
<h3 class="anchored" data-anchor-id="why">
Why
</h3>
<p>
Regular expressions can be used for:
</p>
<ul>
<li>
search
</li>
<li>
replace
</li>
<li>
validation
</li>
<li>
extraction
</li>
</ul>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_why.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
<section id="use-cases" class="level3">
<h3 class="anchored" data-anchor-id="use-cases">
Use Cases
</h3>
<p>
Regular expressions have applications in a wide range of areas. We list some of the most popular ones below:
</p>
<ul>
<li>
email validation
</li>
<li>
password validation
</li>
<li>
date validation
</li>
<li>
phone number validation
</li>
<li>
search and replace in text editors
</li>
<li>
web scraping
</li>
</ul>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_use_case_blog.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
</section>
<section id="learning" class="level2">
<h2 class="anchored" data-anchor-id="learning">
Learning
</h2>
<p>
The below steps offer advice on the best way to learn or use regular expressions:
</p>
<ul>
<li>
describe the pattern in layman terms
</li>
<li>
break down the pattern into individual components<br>
</li>
<li>
match each component to a regular expression
</li>
<li>
build incrementally
</li>
<li>
test
</li>
</ul>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all the resources related to this post:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/regex/regex.html" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/tree/master/regular-expressions-in-r" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/356612" target="_blank">RStudio Cloud</a>
</li>
</ul>
</section>
<section id="libraries" class="level2">
<h2 class="anchored" data-anchor-id="libraries">
Libraries
</h2>
<p>
We will use the following libraries in this post:
</p>
<pre class="r"><code>library(dplyr)
library(readr)</code></pre>
</section>
<section id="data" class="level2">
<h2 class="anchored" data-anchor-id="data">
Data
</h2>
<p>
We will use two data sets in this post. The first one is a list of all R packages on CRAN and is present in the <code>package_names.csv</code> file, and the second one, <code>top_downloads</code>, is the most downloaded packages from the <a href="https://cran.rstudio.com/" target="_blank">RStudio CRAN mirror</a> in the first week of May 2019, and extracted using the <a href="https://cranlogs.r-pkg.org/#rpackage" target="_blank">cranlogs</a> pacakge.
</p>
<section id="r-packages-data" class="level3">
<h3 class="anchored" data-anchor-id="r-packages-data">
R Packages Data
</h3>
<pre class="r"><code>read_csv("package_names.csv") %&gt;%
  pull(1) -&gt; r_packages </code></pre>
</section>
<section id="top-r-packages" class="level3">
<h3 class="anchored" data-anchor-id="top-r-packages">
Top R Packages
</h3>
<pre class="r"><code>top_downloads &lt;- c("devtools", "rlang", "dplyr", "Rcpp", "tibble",    
                   "ggplot2", "glue", "pillar", "cli", "data.table")

top_downloads</code></pre>
<pre><code>##  [1] "devtools"   "rlang"      "dplyr"      "Rcpp"       "tibble"    
##  [6] "ggplot2"    "glue"       "pillar"     "cli"        "data.table"</code></pre>
</section>
</section>
<section id="pattern-matching-functions" class="level2">
<h2 class="anchored" data-anchor-id="pattern-matching-functions">
Pattern Matching Functions
</h2>
<p>
Before we get into the nitty gritty of regular expressions, let us explore a few functions from base R for pattern matching. We will learn about using regular expressions with the stringr package in an upcoming post.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_base_functions.png" width="100%" style="display: block; margin: auto;">
</p>
</section>
<section id="grep" class="level2">
<h2 class="anchored" data-anchor-id="grep">
grep
</h2>
<p>
The first function we will learn is <code>grep()</code>. It can be used to find elements that match a given pattern in a character vector. It will return the elements or the index of the elements based on your specification. In the below example, <code>grep()</code> returns the index of the elements that match the given pattern.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grep_index_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grep(x = top_downloads, pattern = "r")</code></pre>
<pre><code>## [1] 2 3 8</code></pre>
<p>
Now let us look at the inputs:
</p>
<ul>
<li>
<code>pattern</code>
</li>
<li>
<code>x</code>
</li>
<li>
<code>ignore.case</code>
</li>
<li>
<code>value</code>
</li>
<li>
<code>invert</code>
</li>
</ul>
</section>
<section id="grep---value" class="level2">
<h2 class="anchored" data-anchor-id="grep---value">
grep - Value
</h2>
<p>
If you want <code>grep()</code> to return the element instead of the index, set the <code>value</code> argument to <code>TRUE</code>. The default is <code>FALSE</code>. In the below example, <code>grep()</code> returns the elements and not their index.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grep_value_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grep(x = top_downloads, pattern = "r", value = TRUE)</code></pre>
<pre><code>## [1] "rlang"  "dplyr"  "pillar"</code></pre>
</section>
<section id="grep---ignore-case" class="level2">
<h2 class="anchored" data-anchor-id="grep---ignore-case">
grep - Ignore Case
</h2>
<p>
If you have carefully observed the previous examples, have you noticed that the pattern <code>r</code> did not match the element <code>Rcpp</code> i.e.&nbsp;regular expressions are case sensitive. The <code>ignore.case</code> argument will ignore case while matching the pattern as shown below.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grep_ignore_case_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grep(x = top_downloads, pattern = "r", value = TRUE, ignore.case = TRUE)</code></pre>
<pre><code>## [1] "rlang"  "dplyr"  "Rcpp"   "pillar"</code></pre>
<pre class="r"><code>grep(x = top_downloads, pattern = "R", value = TRUE)</code></pre>
<pre><code>## [1] "Rcpp"</code></pre>
<pre class="r"><code>grep(x = top_downloads, pattern = "R", value = TRUE, ignore.case = TRUE)</code></pre>
<pre><code>## [1] "rlang"  "dplyr"  "Rcpp"   "pillar"</code></pre>
</section>
<section id="grep---invert" class="level2">
<h2 class="anchored" data-anchor-id="grep---invert">
grep - Invert
</h2>
<p>
In some cases, you may want to retrieve elements that do not match the pattern specified. The <code>invert</code> argument will return the elements that do not match the pattern. In the below example, the elements returned do not match the pattern specified by us.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grep_invert_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grep(x = top_downloads, pattern = "r", value = TRUE, invert = TRUE)</code></pre>
<pre><code>## [1] "devtools"   "Rcpp"       "tibble"     "ggplot2"    "glue"      
## [6] "cli"        "data.table"</code></pre>
<pre class="r"><code>grep(x = top_downloads, pattern = "r", value = TRUE, 
     invert = TRUE, ignore.case = TRUE)</code></pre>
<pre><code>## [1] "devtools"   "tibble"     "ggplot2"    "glue"       "cli"       
## [6] "data.table"</code></pre>
</section>
<section id="grepl" class="level2">
<h2 class="anchored" data-anchor-id="grepl">
grepl
</h2>
<p>
<code>grepl()</code> will return only logical values. If the elements match the pattern specified, it will return <code>TRUE</code> else <code>FALSE</code>.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grepl_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grepl(x = top_downloads, pattern = "r")</code></pre>
<pre><code>##  [1] FALSE  TRUE  TRUE FALSE FALSE FALSE FALSE  TRUE FALSE FALSE</code></pre>
<section id="ignore-case" class="level3">
<h3 class="anchored" data-anchor-id="ignore-case">
Ignore Case
</h3>
<p>
To ignore the case, use the <code>ignore.case</code> argument and set it to <code>TRUE</code>.
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/regex_grepl_ignore_case_canva.png" width="100%" style="display: block; margin: auto;">
</p>
<pre class="r"><code>grepl(x = top_downloads, pattern = "r", ignore.case = TRUE)</code></pre>
<pre><code>##  [1] FALSE  TRUE  TRUE  TRUE FALSE FALSE FALSE  TRUE FALSE FALSE</code></pre>
<p>
The next 3 functions that we explore differ from the above 2 in the format of and amount of details in the results. They all return the following additional details:
</p>
<ul>
<li>
the starting position of the first match
</li>
<li>
the length of the matched text
</li>
<li>
whether the match position and length are in chracter or bytes
</li>
</ul>
</section>
<section id="regexpr" class="level3">
<h3 class="anchored" data-anchor-id="regexpr">
regexpr
</h3>
<pre class="r"><code>rr_pkgs &lt;- c("purrr", "olsrr", "blorr")
regexpr("r", rr_pkgs)</code></pre>
<pre><code>## [1] 3 4 4
## attr(,"match.length")
## [1] 1 1 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE</code></pre>
</section>
<section id="gregexpr" class="level3">
<h3 class="anchored" data-anchor-id="gregexpr">
gregexpr
</h3>
<pre class="r"><code>gregexpr("r", rr_pkgs)</code></pre>
<pre><code>## [[1]]
## [1] 3 4 5
## attr(,"match.length")
## [1] 1 1 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[2]]
## [1] 4 5
## attr(,"match.length")
## [1] 1 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[3]]
## [1] 4 5
## attr(,"match.length")
## [1] 1 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE</code></pre>
</section>
<section id="regexec" class="level3">
<h3 class="anchored" data-anchor-id="regexec">
regexec
</h3>
<pre class="r"><code>regexec("r", rr_pkgs)</code></pre>
<pre><code>## [[1]]
## [1] 3
## attr(,"match.length")
## [1] 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[2]]
## [1] 4
## attr(,"match.length")
## [1] 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE
## 
## [[3]]
## [1] 4
## attr(,"match.length")
## [1] 1
## attr(,"index.type")
## [1] "chars"
## attr(,"useBytes")
## [1] TRUE</code></pre>
<hr>
<p>
<a href="https://www.youtube.com/user/rsquaredin/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/ad_youtube.png" width="100%" alt="youtube ad" style="text-decoration: none;"></a>
</p>
<hr>
</section></section> ]]></description>
  <category>R</category>
  <category>regex</category>
  <guid>https://blog.rsquaredacademy.com/posts/regular-expression-in-r/</guid>
  <pubDate>Mon, 27 May 2019 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/regex_cover_image.png" medium="image" type="image/png" height="89" width="144"/>
</item>
<item>
  <title>Practical Introduction to Market Basket Analysis - Asociation Rules</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/market-basket-analysis-in-r/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-05-02-mba.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/cover_image.jpg" width="80%" style="display: block; margin: auto;">
</p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/mba_intro.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Ever wondered why items are displayed in a particular way in retail/online stores. Why certain items are suggested to you based on what you have added to the cart? Blame it on market basket analysis or association rule mining.
</p>
</section>
<section id="resources" class="level2">
<h2 class="anchored" data-anchor-id="resources">
Resources
</h2>
<p>
Below are the links to all the resources related to this post:
</p>
<ul>
<li>
<a href="https://slides.rsquaredacademy.com/mba/mba.html" target="_blank">Slides</a>
</li>
<li>
<a href="https://github.com/rsquaredacademy-education/online-courses/tree/master/association-rule-mining-in-r" target="_blank">Code &amp; Data</a>
</li>
<li>
<a href="https://rstudio.cloud/project/335377" target="_blank">RStudio Cloud</a>
</li>
</ul>
</section>
<section id="what" class="level2">
<h2 class="anchored" data-anchor-id="what">
What?
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/mba_what.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Market basket analysis uses association rule mining under the hood to identify products frequently bought together. Before we get into the nitty gritty of market basket analysis, let us get a basic understanding of association rule mining. It finds association between different objects in a set. In the case of market basket analysis, the objects are the products purchased by a cusomter and the set is the transaction. In short, market basket analysis
</p>
<ul>
<li>
is a unsupervised data mining technique
</li>
<li>
that uncovers products frequently bought together
</li>
<li>
and creates if-then scenario rules
</li>
</ul>
</section>
<section id="why" class="level2">
<h2 class="anchored" data-anchor-id="why">
Why ?
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/mba_why.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Market basket analysis creates actionable insights for:
</p>
<ul>
<li>
designing store layout
</li>
<li>
online recommendation engines
</li>
<li>
targeted marketing campaign/sales promotion/email campaign
</li>
<li>
cross/up selling
</li>
<li>
catalogue design
</li>
</ul>
</section>
<section id="advantages" class="level2">
<h2 class="anchored" data-anchor-id="advantages">
Advantages
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/mba_advantages.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Market basket analysisis is <strong>cost effective</strong> as data required is readily available through electronic point of sale systems. It generates <strong>actionable insights</strong> for product placement, cross/up selling strategies, targeted marketing campaigns, catalogue design, pricing strategies, inventory control etc.
</p>
</section>
<section id="use-cases" class="level2">
<h2 class="anchored" data-anchor-id="use-cases">
Use Cases
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/mba_use_cases.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Association rule mining has applications in several industries including retail, telecommunications, banking, insurance, manufacturing and medical. Let us look at its applications in more detail in the following industries:
</p>
<section id="retail" class="level3">
<h3 class="anchored" data-anchor-id="retail">
Retail
</h3>
<p>
The introduction of electronic point of sale systems have allowed the collection of immense amounts of data and retail organizations make prolifc use of market basket analysis for
</p>
<ul>
<li>
designing store layout so that consumers can more easily find items that are frequently purchased together
</li>
<li>
recommending associated products that are frequently bought together, <strong>“Customers who purchased this product also viewed this product…”</strong>
</li>
<li>
emailing customers who bought products specific products with other products and offers on those products that are likely to be interesting to them.
</li>
<li>
grouping products that customers purchase frequently together in the store’s product placement
</li>
<li>
designing special promotions that combine or discount certain products
</li>
<li>
optimizing the layout of the catalog of an eCommerce site
</li>
<li>
controlling inventory based on product demands and what products sell better together
</li>
</ul>
</section>
<section id="banks" class="level3">
<h3 class="anchored" data-anchor-id="banks">
Banks
</h3>
<p>
Banks and financial institutions use market basket analysis to analyze credit card purchases for fraud detection and cross sell insurance products, investment products (mutual funds etc.), tax preparation, retirement planning, wealth management etc. It can also be used for next best offer, sequence and seasonal offers.
</p>
</section>
<section id="telecommunications" class="level3">
<h3 class="anchored" data-anchor-id="telecommunications">
Telecommunications
</h3>
<p>
The telecommunications industry is characterized by high volatility and low customer loyalty due to lucrative offers for new customers from other service providers. The more services a customer uses from a particular operator, the harder it gets for him/her to switch to another operator. Market basket analysis is used to bundle mobile, landline, TV and internet services to customers to increase stickiness and reduce churn.
</p>
<hr>
<a href="https://www.rsquaredacademy.com/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/ad.png" width="100%" alt="course ad" style="text-decoration: none;"></a>
<hr>
</section></section> ]]></description>
  <category>R</category>
  <category>arules</category>
  <guid>https://blog.rsquaredacademy.com/posts/market-basket-analysis-in-r/</guid>
  <pubDate>Thu, 02 May 2019 00:00:00 GMT</pubDate>
  <media:content url="https://blog.rsquaredacademy.com/img/cover_image.jpg" medium="image" type="image/jpeg"/>
</item>
<item>
  <title>A follow up note on our web scraping tutorial</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/web-scraping-note/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-04-13-web-scraping-note.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
We had published a web scraping <a href="https://blog.rsquaredacademy.com/web-scraping/" target="_blank">tutorial</a> a couple of days back and it had received a good response from the #rstats community. While we thank you for that, we made a mistake in choosing one of the case study as pointed out by <a href="https://twitter.com/hrbrmstr" target="_blank"><span class="citation"><span class="citation" data-cites="hrbrmstr">@hrbrmstr</span></span></a> in this tweet:
</p>
<p>
We choose the case study to appeal to a wide audience but in doing so we set a bad example. We have removed the case study from our post and apologize to the #rstats community for our oversight and promise to be more responsible in the future.
</p>
<p>
<strong>We reiterate it is very important to read the terms and conditions before scraping data from a website and checking the robots.txt file is not sufficient</strong>.
</p>
<p>
Team Rsquared Academy
</p>



 ]]></description>
  <category>R</category>
  <guid>https://blog.rsquaredacademy.com/posts/web-scraping-note/</guid>
  <pubDate>Sat, 13 Apr 2019 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Practical Introduction to Web Scraping in R</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/web-scraping/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-04-11-web-scraping.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/ws_meme.jpg" width="80%" style="display: block; margin: auto;">
</p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">
Introduction
</h2>
<p>
Are you trying to compare price of products across websites? Are you trying to monitor price changes every hour? Or planning to do some text mining or sentiment analysis on reviews of products or services? If yes, how would you do that? How do you get the details available on the website into a format in which you can analyse it?
</p>
<ul>
<li>
Can you copy/paste the data from their website?
</li>
<li>
Can you see some save button?
</li>
<li>
Can you download the data?
</li>
</ul>
<p>
Hmmm.. If you have these or similar questions on your mind, you have come to the right place. In this post, we will learn about web scraping using R. Below is a video tutorial which covers the intial part of this post.
</p>
{{% youtube “l37n_HDD1qs” %}}
<p>
The slides used in the above video tutorial can be found <a href="https://slides.rsquaredacademy.com/web-scraping/web-scraping.html#/section" target="_blank">here</a>.
</p>
</section>
<section id="the-what" class="level2">
<h2 class="anchored" data-anchor-id="the-what">
The What?
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_what_border.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
What exactly is web scraping or web mining or web harvesting? It is a technique for extracting data from websites. Remember, websites contain wealth of useful data but designed for human consumption and not data analysis. The goal of web scraping is to take advantage of the pattern or structure of web pages to extract and store data in a format suitable for data analysis.
</p>
</section>
<section id="the-why" class="level2">
<h2 class="anchored" data-anchor-id="the-why">
The Why?
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_why_border.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Now, let us understand why we may have to scrape data from the web.
</p>
<ul>
<li>
<strong>Data Format</strong>: As we said earlier, there is a wealth of data on websites but designed for human consumption. As such, we cannot use it for data analysis as it is not in a suitable format/shape/structure.
</li>
<li>
<strong>No copy/paste</strong>: We cannot copy &amp; paste the data into a local file. Even if we do it, it will not be in the required format for data analysis.
</li>
<li>
<strong>No save/download</strong>: There are no options to save/download the required data from the websites. We cannot right click and save or click on a download button to extract the required data.
</li>
<li>
<strong>Automation</strong>: With web scraping, we can automate the process of data extraction/harvesting.
</li>
</ul>
</section>
<section id="the-how" class="level2">
<h2 class="anchored" data-anchor-id="the-how">
The How?
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_how_border.png" width="80%" style="display: block; margin: auto;">
</p>
<ul>
<li>
<strong>robots.txt</strong>: One of the most important and overlooked step is to check the <strong>robots.txt</strong> file to ensure that we have the permission to access the web page without violating any terms or conditions. In R, we can do this using the <a href="https://cran.r-project.org/package=robotstxt" target="_blank">robotstxt</a> by <a href="https://ropensci.org/" target="_blank">rOpenSci</a>.
</li>
<li>
<strong>Fetch</strong>: The next step is to fetch the web page using the <a href="https://cran.r-project.org/package=xml2" target="_blank">xml2</a> package and store it so that we can extract the required data. Remember, you fetch the page once and store it to avoid fetching multiple times as it may lead to your IP address being blocked by the owners of the website.
</li>
<li>
<strong>Extract/Store/Analyze</strong>: Now that we have fetched the web page, we will use <a href="https://rvest.tidyverse.org/" target="_blank">rvest</a> to extract the data and store it for further analysis.
</li>
</ul>
</section>
<section id="use-cases" class="level2">
<h2 class="anchored" data-anchor-id="use-cases">
Use Cases
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_use_cases_border.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
Below are few use cases of web scraping:
</p>
<ul>
<li>
<strong>Contact Scraping</strong>: Locate contact information including email addresses, phone numbers etc.
</li>
<li>
<strong>Monitoring/Comparing Prices</strong>: How your competitors price their products, how your prices fit within your industry, and whether there are any fluctuations that you can take advantage of.
</li>
<li>
<strong>Scraping Reviews/Ratings</strong>: Scrape reviews of product/services and use it for text mining/sentiment analysis etc.
</li>
</ul>
</section>
<section id="things-to-keep-in-mind" class="level2">
<h2 class="anchored" data-anchor-id="things-to-keep-in-mind">
Things to keep in mind…
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_keep_in_mind_border.png" width="80%" style="display: block; margin: auto;">
</p>
<ul>
<li>
<strong>Static &amp; Well Structured</strong>: Web scraping is best suited for static &amp; well structured web pages. In one of our case studies, we demonstrate how badly structured web pages can hamper data extraction.
</li>
<li>
<strong>Code Changes</strong>: The underling HTML code of a web page can change anytime due to changes in design or for updating details. In such case, your script will stop working. It is important to identify changes to the web page and modify the web scraping script accordingly.
</li>
<li>
<strong>API Availability</strong>: In many cases, an API (application programming interface) is made available by the service provider or organization. It is always advisable to use the API and avoid web scraping. The <a href="https://httr.r-lib.org/" target="_blank">httr</a> package has a nice introduction on interacting with APIs.
</li>
<li>
<strong>IP Blocking</strong>: Do not flood websites with requests as you run the risk of getting blocked. Have some time gap between request so that your IP address in not blocked from accessing the website.
</li>
<li>
<strong>robots.txt</strong>: We cannot emphasize this enough, always review the <strong>robots.txt</strong> file to ensure you are not violating any terms and conditions.
</li>
</ul>
</section>
<section id="case-studies" class="level2">
<h2 class="anchored" data-anchor-id="case-studies">
Case Studies
</h2>
<p>
<img src="https://blog.rsquaredacademy.com/img/blog_ws_case_studies_border.png" width="80%" style="display: block; margin: auto;">
</p>
<ul>
<li>
<strong>IMDB top 50 movies</strong>: In this case study we will scrape the IMDB website to extract the title, year of release, certificate, runtime, genre, rating, votes and revenue of the top 50 movies.
</li>
<li>
<strong>Most visited websites</strong>: In this case study, we will look at the 50 most visited websites in the world including the category to which they belong, average time on site, average pages browsed per vist and bounce rate.
</li>
<li>
<strong>List of RBI governors</strong> : In this final case study, we will scrape the list of RBI Governors from Wikipedia, and analyze the background from which they came i.e whether there were more economists or bureaucrats?
</li>
</ul>
<hr>
<p>
<a href="https://www.rsquaredacademy.com/" target="_blank"><img src="https://blog.rsquaredacademy.com/img/ad.png" width="100%" alt="course ad" style="text-decoration: none;"></a>
</p>
<hr>
</section> ]]></description>
  <category>R</category>
  <category>rvest</category>
  <guid>https://blog.rsquaredacademy.com/posts/web-scraping/</guid>
  <pubDate>Thu, 11 Apr 2019 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Shiny Apps for Interactive Data Analysis</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/shiny-apps/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-04-01-shiny-apps.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/apps_banner.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
We are excited and happy to share a set of <a href="https://shiny.rstudio.com/" target="_blank">shiny</a> apps built for interactive data analysis and teaching at <a href="https://www.rsquaredacademy.com" target="_blank">Rsquared Academy</a>. The apps are part of our <a href="https://pkgs.rsquaredacademy.com" target="_blank">R packages</a> and presently cover the following topics:
</p>
<ul>
<li>
Descriptive Statistics
</li>
<li>
Probability Distributions
</li>
<li>
Hypothesis Testing
</li>
<li>
Linear Regression
</li>
<li>
Logistic Regression
</li>
<li>
RFM Analysis
</li>
<li>
Data Visualization
</li>
</ul>
<p>
We would suggest that you explore the apps using sample data sets available within the app before using your own data set so that you get comfortable with the user interface.
</p>
<section id="demo" class="level2">
<h2 class="anchored" data-anchor-id="demo">
Demo
</h2>
<p>{{% youtube “7lBEqrZLUOU” %}}</p>
</section>
<section id="details" class="level2">
<h2 class="anchored" data-anchor-id="details">
Details
</h2>
<p>
You can try the live version of the apps <a href="https://apps.rsquaredacademy.com" target="_blank">here</a> or locally launch these apps from the R packages mentioned in the below table. <strong>It will be easier to use the apps if you refer to the documentation of the respective packages.</strong>
</p>
<table class="table">
<thead>
<tr class="header">
<th>
Topic
</th>
<th>
Try App
</th>
<th>
R Package
</th>
<th>
API
</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>
Descriptive Statistics
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/olsrr/" target="_blank">Launch</a>
</td>
<td>
<a href="https://descriptr.rsquaredacademy.com/" target="_blank">descritpr</a>
</td>
<td>
<code>ds_launch_shiny_app()</code>
</td>
</tr>
<tr class="even">
<td>
Visualize Distributions
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/vistributions/" target="_blank">Launch</a>
</td>
<td>
<a href="https://vistributions.rsquaredacademy.com/" target="_blank">vistributions</a>
</td>
<td>
<code>vdist_launch_app()</code>
</td>
</tr>
<tr class="odd">
<td>
Hypothesis Testing
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/inferr/" target="_blank">Launch</a>
</td>
<td>
<a href="https://inferr.rsquaredacademy.com/" target="_blank">inferr</a>
</td>
<td>
<code>infer_launch_shiny_app()</code>
</td>
</tr>
<tr class="even">
<td>
Linear Regression
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/olsrr/" target="_blank">Launch</a>
</td>
<td>
<a href="https://olsrr.rsquaredacademy.com/" target="_blank">olsrr</a>
</td>
<td>
<code>ols_launch_app()</code>
</td>
</tr>
<tr class="odd">
<td>
Logistic Regression
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/blorr/" target="_blank">Launch</a>
</td>
<td>
<a href="https://blorr.rsquaredacademy.com/" target="_blank">blorr</a>
</td>
<td>
<code>blr_launch_app()</code>
</td>
</tr>
<tr class="even">
<td>
RFM Analysis
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/rfm/" target="_blank">Launch</a>
</td>
<td>
<a href="https://rfm.rsquaredacademy.com/" target="_blank">rfm</a>
</td>
<td>
<code>rfm_launch_app()</code>
</td>
</tr>
<tr class="odd">
<td>
Data Visualization
</td>
<td>
<a href="https://www.rsquaredcomputing.com/shiny/xplorerr/" target="_blank">Launch</a>
</td>
<td>
<a href="https://xplorerr.rsquaredacademy.com/" target="_blank">xplorerr</a>
</td>
<td>
<code>app_visualizer()</code>
</td>
</tr>
</tbody>
</table>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
All feedback is welcome. Issues (bugs and feature requests) can be posted to <a href="https://github.com/rsquaredacademy/xplorerr/issues">github tracker</a>. For help with code or other related questions, feel free to reach out to us at <a href="mailto:pkgs@rsquaredacademy.com" class="email">pkgs@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>shiny</category>
  <guid>https://blog.rsquaredacademy.com/posts/shiny-apps/</guid>
  <pubDate>Mon, 01 Apr 2019 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Visually explore Probability Distributions with vistributions</title>
  <dc:creator>Aravind Hebbali</dc:creator>
  <link>https://blog.rsquaredacademy.com/posts/introducting-vistributions/</link>
  <description><![CDATA[ 




<!-- Migrated from content/post/2019-03-14-introducing-vistributions.Rmd. -->
<!-- Day-1 static bundle: body reuses the pre-rendered .html fragment. -->
<p>
<img src="https://blog.rsquaredacademy.com/img/vistributions_banner.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
We are happy to introduce the <strong>vistributions</strong> package, a set of tools for visually exploring probability distributions.
</p>
<section id="installation" class="level2">
<h2 class="anchored" data-anchor-id="installation">
Installation
</h2>
<pre class="r"><code># Install release version from CRAN
install.packages("vistributions")

# Install development version from GitHub
# install.packages("devtools")
devtools::install_github("rsquaredacademy/vistributions")</code></pre>
</section>
<section id="shiny-app" class="level2">
<h2 class="anchored" data-anchor-id="shiny-app">
Shiny App
</h2>
<p>
<strong>vistributions</strong> includes a shiny app which can be launched using
</p>
<pre class="r"><code>vdist_launch_app()</code></pre>
<p>
or try the live version <a href="https://www.rsquaredcomputing.com/shiny/vistributions/">here</a>.
</p>
{{% youtube “Wa44jxAJvTE” %}}
<p>
Read on to learn more about the features of <strong>vistributions</strong>, or see the <a href="https://vistributions.rsquaredacademy.com/">vistributions website</a> for detailed documentation on using the package.
</p>
</section>
<section id="tab-completion" class="level2">
<h2 class="anchored" data-anchor-id="tab-completion">
Tab Completion
</h2>
<p>
The common <code>vdist_</code> prefix will trigger autocomplete, allowing you to see all <strong>vistributions</strong> functions:
</p>
<p>
<img src="https://blog.rsquaredacademy.com/img/tab_gif.png" width="80%" style="display: block; margin: auto;">
</p>
<p>
In exploring statistical distributions, we focus on the following:
</p>
<ul>
<li>
what influences the shape of a distribution
</li>
<li>
calculate probability from a given quantile
</li>
<li>
calculate quantiles out of given probability
</li>
</ul>
<p>
for the following distributions:
</p>
<ul>
<li>
Normal
</li>
<li>
Binomial
</li>
<li>
Chi Square
</li>
<li>
F
</li>
<li>
t
</li>
</ul>
</section>
<section id="normal-distribution" class="level2">
<h2 class="anchored" data-anchor-id="normal-distribution">
Normal Distribution
</h2>
<section id="distribution-shape" class="level3">
<h3 class="anchored" data-anchor-id="distribution-shape">
Distribution Shape
</h3>
<p>
Visualize how changes in mean and standard deviation affect the shape of the normal distribution.
</p>
<section id="input" class="level4">
<h4 class="anchored" data-anchor-id="input">
Input
</h4>
<ul>
<li>
mean: mean of the normal distribution
</li>
<li>
sd: standard deviation of the normal distribution
</li>
</ul>
<pre class="r"><code>vdist_normal_plot(mean = 2, sd = 0.6)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_plot-1.png" width="672">
</p>
</section>
</section>
<section id="percentiles" class="level3">
<h3 class="anchored" data-anchor-id="percentiles">
Percentiles
</h3>
<section id="calculate-and-visualize-quantiles-out-of-given-probability." class="level4">
<h4 class="anchored" data-anchor-id="calculate-and-visualize-quantiles-out-of-given-probability.">
Calculate and visualize quantiles out of given probability.
</h4>
<section id="input-1" class="level5">
<h5 class="anchored" data-anchor-id="input-1">
Input
</h5>
<ul>
<li>
probs: a probability value
</li>
<li>
mean: mean of the normal distribution
</li>
<li>
sd: standard deviation of the normal distribution
</li>
<li>
type: lower/upper tail
</li>
</ul>
<p>
Suppose X, the grade on a exam, is normally distributed with mean 60 and standard deviation 3. The teacher wants to give 10% of the class an A. What should be the cutoff to determine who gets an A?
</p>
<pre class="r"><code>vdist_normal_perc(0.10, 60, 3, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_per1-1.png" width="672">
</p>
<p>
The teacher wants to give lower 15% of the class a D. What cutoff should the teacher use to determine who gets an D?
</p>
<pre class="r"><code>vdist_normal_perc(0.85, 60, 3, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_per2-1.png" width="672">
</p>
<p>
The teacher wants to give middle 50% of the class a B. What cutoff should the teacher use to determine who gets an B?
</p>
<pre class="r"><code>vdist_normal_perc(0.5, 60, 3, 'both')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_per3-1.png" width="672">
</p>
</section>
</section>
</section>
<section id="probabilities" class="level3">
<h3 class="anchored" data-anchor-id="probabilities">
Probabilities
</h3>
<section id="calculate-and-visualize-probability-from-a-given-quantile" class="level4">
<h4 class="anchored" data-anchor-id="calculate-and-visualize-probability-from-a-given-quantile">
Calculate and visualize probability from a given quantile
</h4>
<section id="input-2" class="level5">
<h5 class="anchored" data-anchor-id="input-2">
Input
</h5>
<ul>
<li>
perc: a quantile value
</li>
<li>
mean: mean of the normal distribution
</li>
<li>
sd: standard deviation of the normal distribution
</li>
<li>
type: lower/upper/both tail
</li>
</ul>
<p>
Let X be the IQ of a randomly selected student of a school. Assume X ~ N(90, 4). What is the probability that a randomly selected student has an IQ below 80?
</p>
<pre class="r"><code>vdist_normal_prob(80, mean = 90, sd = 4)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_prob1-1.png" width="672">
</p>
<p>
What is the probability that a randomly selected student has an IQ above 100?
</p>
<pre class="r"><code>vdist_normal_prob(100, mean = 90, sd = 4, type = 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_prob2-1.png" width="672">
</p>
<p>
What is the probability that a randomly selected student has an IQ between 85 and 100?
</p>
<pre class="r"><code>vdist_normal_prob(c(85, 100), mean = 90, sd = 4, type = 'both')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/norm_prob3-1.png" width="672">
</p>
</section>
</section>
</section>
</section>
<section id="binomial-distribution" class="level2">
<h2 class="anchored" data-anchor-id="binomial-distribution">
Binomial Distribution
</h2>
<section id="distribution-shape-1" class="level3">
<h3 class="anchored" data-anchor-id="distribution-shape-1">
Distribution Shape
</h3>
<p>
Visualize how changes in number of trials and the probability of success affect the shape of the binomial distribution.
</p>
<pre class="r"><code>vdist_binom_plot(10, 0.3)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_plot-1.png" width="672">
</p>
</section>
<section id="percentiles-1" class="level3">
<h3 class="anchored" data-anchor-id="percentiles-1">
Percentiles
</h3>
<section id="calculate-and-visualize-quantiles-out-of-given-probability" class="level4">
<h4 class="anchored" data-anchor-id="calculate-and-visualize-quantiles-out-of-given-probability">
Calculate and visualize quantiles out of given probability
</h4>
<section id="input-3" class="level5">
<h5 class="anchored" data-anchor-id="input-3">
Input
</h5>
<ul>
<li>
p: a single aggregated probability of multiple trials
</li>
<li>
n: the number of trials
</li>
<li>
tp: the probability of success in a trial
</li>
<li>
type: lower/upper tail
</li>
</ul>
<pre class="r"><code>vdist_binom_perc(10, 0.5, 0.05)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_per1-1.png" width="672">
</p>
<pre class="r"><code>vdist_binom_perc(10, 0.5, 0.05, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_per2-1.png" width="672">
</p>
</section>
</section>
</section>
<section id="probabilities-1" class="level3">
<h3 class="anchored" data-anchor-id="probabilities-1">
Probabilities
</h3>
<section id="calculate-and-visualize-probability-from-a-given-quantile-1" class="level4">
<h4 class="anchored" data-anchor-id="calculate-and-visualize-probability-from-a-given-quantile-1">
Calculate and visualize probability from a given quantile
</h4>
<section id="input-4" class="level5">
<h5 class="anchored" data-anchor-id="input-4">
Input
</h5>
<ul>
<li>
p: probability of success
</li>
<li>
n: the number of trials
</li>
<li>
s: number of success in a trial
</li>
<li>
type: lower/upper/interval/exact tail
</li>
</ul>
<p>
Assume twenty-percent (20%) of Magemill have no health insurance. Randomly sample n = 12 Magemillians. Let X denote the number in the sample with no health insurance. What is the probability that exactly 4 of the 15 sampled have no health insurance?
</p>
<pre class="r"><code>vdist_binom_prob(12, 0.2, 4, type = 'exact')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_prob1-1.png" width="672">
</p>
<p>
What is the probability that at most one of those sampled has no health insurance?
</p>
<pre class="r"><code>vdist_binom_prob(12, 0.2, 1, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_prob2-1.png" width="672">
</p>
<p>
What is the probability that more than seven have no health insurance?
</p>
<pre class="r"><code>vdist_binom_prob(12, 0.2, 8, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_prob3-1.png" width="672">
</p>
<p>
What is the probability that fewer than 5 have no health insurance?
</p>
<pre class="r"><code>vdist_binom_prob(12, 0.2, c(0, 4), 'interval')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/binom_prob4-1.png" width="672">
</p>
</section>
</section>
</section>
</section>
<section id="chi-square-distribution" class="level2">
<h2 class="anchored" data-anchor-id="chi-square-distribution">
Chi Square Distribution
</h2>
<section id="distribution-shape-2" class="level3">
<h3 class="anchored" data-anchor-id="distribution-shape-2">
Distribution Shape
</h3>
<p>
Visualize how changes in degrees of freedom affect the shape of the chi square distribution.
</p>
<pre class="r"><code>vdist_chisquare_plot(df = 5)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_plot-1.png" width="672">
</p>
<pre class="r"><code>vdist_chisquare_plot(df = 5, normal = TRUE)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_plot-2.png" width="672">
</p>
</section>
<section id="percentiles-2" class="level3">
<h3 class="anchored" data-anchor-id="percentiles-2">
Percentiles
</h3>
<section id="calculate-quantiles-out-of-given-probability" class="level4">
<h4 class="anchored" data-anchor-id="calculate-quantiles-out-of-given-probability">
Calculate quantiles out of given probability
</h4>
<section id="input-5" class="level5">
<h5 class="anchored" data-anchor-id="input-5">
Input
</h5>
<ul>
<li>
probs: a probability value
</li>
<li>
df: degrees of freedom
</li>
<li>
type: lower/upper tail
</li>
</ul>
<p>
Let X be a chi-square random variable with 8 degrees of freedom. What is the upper fifth percentile?
</p>
<pre class="r"><code>vdist_chisquare_perc(0.05, 8, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_per1-1.png" width="672">
</p>
<p>
What is the tenth percentile?
</p>
<pre class="r"><code>vdist_chisquare_perc(0.10, 8, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_per2-1.png" width="672">
</p>
</section>
</section>
</section>
<section id="probability" class="level3">
<h3 class="anchored" data-anchor-id="probability">
Probability
</h3>
<section id="calculate-probability-from-a-given-quantile." class="level4">
<h4 class="anchored" data-anchor-id="calculate-probability-from-a-given-quantile.">
Calculate probability from a given quantile.
</h4>
<section id="input-6" class="level5">
<h5 class="anchored" data-anchor-id="input-6">
Input
</h5>
<ul>
<li>
perc: a quantile value
</li>
<li>
df: degrees of freedom
</li>
<li>
type: lower/upper tail
</li>
</ul>
<p>
What is the probability that a chi-square random variable with 12 degrees of freedom is greater than 8.79?
</p>
<pre class="r"><code>vdist_chisquare_prob(8.79, 12, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_prob1-1.png" width="672">
</p>
<p>
What is the probability that a chi-square random variable with 12 degrees of freedom is greater than 8.62?
</p>
<pre class="r"><code>vdist_chisquare_prob(8.62, 12, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/chi_prob2-1.png" width="672">
</p>
</section>
</section>
</section>
</section>
<section id="f-distribution" class="level2">
<h2 class="anchored" data-anchor-id="f-distribution">
F Distribution
</h2>
<section id="distribution-shape-3" class="level3">
<h3 class="anchored" data-anchor-id="distribution-shape-3">
Distribution Shape
</h3>
<p>
Visualize how changes in degrees of freedom affect the shape of the F distribution.
</p>
<pre class="r"><code>vdist_f_plot()</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_plot-1.png" width="672">
</p>
<pre class="r"><code>vdist_f_plot(6, 10, normal = TRUE)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_plot-2.png" width="672">
</p>
</section>
<section id="percentiles-3" class="level3">
<h3 class="anchored" data-anchor-id="percentiles-3">
Percentiles
</h3>
<section id="calculate-quantiles-out-of-given-probability-1" class="level4">
<h4 class="anchored" data-anchor-id="calculate-quantiles-out-of-given-probability-1">
Calculate quantiles out of given probability
</h4>
<section id="input-7" class="level5">
<h5 class="anchored" data-anchor-id="input-7">
Input
</h5>
<ul>
<li>
probs: a probability value
</li>
<li>
num_df: nmerator degrees of freedom
</li>
<li>
den_df: denominator degrees of freedom
</li>
<li>
type: lower/upper tail
</li>
</ul>
<p>
Let X be an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom. What is the upper twenth percentile?
</p>
<pre class="r"><code>vdist_f_perc(0.20, 4, 5, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_per1-1.png" width="672">
</p>
<p>
What is the 35th percentile?
</p>
<pre class="r"><code>vdist_f_perc(0.35, 4, 5, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_per2-1.png" width="672">
</p>
</section>
</section>
</section>
<section id="probabilities-2" class="level3">
<h3 class="anchored" data-anchor-id="probabilities-2">
Probabilities
</h3>
<section id="calculate-probability-from-a-given-quantile.-1" class="level4">
<h4 class="anchored" data-anchor-id="calculate-probability-from-a-given-quantile.-1">
Calculate probability from a given quantile.
</h4>
<section id="input-8" class="level5">
<h5 class="anchored" data-anchor-id="input-8">
Input
</h5>
<ul>
<li>
perc: a quantile value
</li>
<li>
num_df: nmerator degrees of freedom
</li>
<li>
den_df: denominator degrees of freedom
</li>
<li>
type: lower/upper tail
</li>
</ul>
<p>
What is the probability that an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom is greater than 3.89?
</p>
<pre class="r"><code>vdist_f_prob(3.89, 4, 5, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_prob1-1.png" width="672">
</p>
<p>
What is the probability that an F random variable with 4 numerator degrees of freedom and 5 denominator degrees of freedom is less than 2.63?
</p>
<pre class="r"><code>vdist_f_prob(2.63, 4, 5, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/f_prob2-1.png" width="672">
</p>
</section>
</section>
</section>
</section>
<section id="t-distribution" class="level2">
<h2 class="anchored" data-anchor-id="t-distribution">
t Distribution
</h2>
<section id="distribution-shape-4" class="level3">
<h3 class="anchored" data-anchor-id="distribution-shape-4">
Distribution Shape
</h3>
<p>
Visualize how degrees of freedom affect the shape of t distribution.
</p>
<pre class="r"><code>vdist_t_plot(df = 8)</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_plot-1.png" width="672">
</p>
</section>
<section id="percentiles-4" class="level3">
<h3 class="anchored" data-anchor-id="percentiles-4">
Percentiles
</h3>
<section id="calculate-quantiles-out-of-given-probability-2" class="level4">
<h4 class="anchored" data-anchor-id="calculate-quantiles-out-of-given-probability-2">
Calculate quantiles out of given probability
</h4>
<section id="input-9" class="level5">
<h5 class="anchored" data-anchor-id="input-9">
Input
</h5>
<ul>
<li>
probs: a probability value
</li>
<li>
df: degrees of freedom
</li>
<li>
type: lower/upper/both tail
</li>
</ul>
<p>
What is the upper fifteenth percentile?
</p>
<pre class="r"><code>vdist_t_perc(0.15, 8, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_per1-1.png" width="672">
</p>
<p>
What is the eleventh percentile?
</p>
<pre class="r"><code>vdist_t_perc(0.11, 8, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_per2-1.png" width="672">
</p>
<p>
What is the area of the curve that has 95% of the t values?
</p>
<pre class="r"><code>vdist_t_perc(0.8, 8, 'both')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_per3-1.png" width="672">
</p>
</section>
</section>
</section>
<section id="probabilities-3" class="level3">
<h3 class="anchored" data-anchor-id="probabilities-3">
Probabilities
</h3>
<section id="calculate-probability-from-a-given-quantile.-2" class="level4">
<h4 class="anchored" data-anchor-id="calculate-probability-from-a-given-quantile.-2">
Calculate probability from a given quantile.
</h4>
<section id="input-10" class="level5">
<h5 class="anchored" data-anchor-id="input-10">
Input
</h5>
<ul>
<li>
perc: a quantile value
</li>
<li>
df: degrees of freedom
</li>
<li>
type: lower/upper/interval/both tail
</li>
</ul>
<p>
Let T follow a t-distribution with r = 6 df.
</p>
<p>
What is the probability that the value of T is less than 2?
</p>
<pre class="r"><code>vdist_t_prob(2, 6, 'lower')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_prob1-1.png" width="672">
</p>
<p>
What is the probability that the value of T is greater than 2?
</p>
<pre class="r"><code>vdist_t_prob(2, 6, 'upper')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_prob2-1.png" width="672">
</p>
<p>
What is the probability that the value of T is between -2 and 2?
</p>
<pre class="r"><code>vdist_t_prob(2, 6, 'both')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_prob3-1.png" width="672">
</p>
<p>
What is the probability that the absolute value of T is greater than 2?
</p>
<pre class="r"><code>vdist_t_prob(2, 6, 'interval')</code></pre>
<p>
<img src="https://blog.rsquaredacademy.com/posts/introducting-vistributions/2019-03-14-introducing-vistributions_files/figure-html/t_prob4-1.png" width="672">
</p>
</section>
</section>
</section>
</section>
<section id="learning-more" class="level2">
<h2 class="anchored" data-anchor-id="learning-more">
Learning More
</h2>
<p>
The <a href="https://vistributions.rsquaredacademy.com/">vistributions website</a> includes comprehensive documentation on using the package, including the following article that gives a brief introduction to vistributions:
</p>
<ul>
<li>
<a href="https://vistributions.rsquaredacademy.com/articles/introduction-to-vistributions.html">Introduction to vistributions</a>
</li>
</ul>
</section>
<section id="feedback" class="level2">
<h2 class="anchored" data-anchor-id="feedback">
Feedback
</h2>
<p>
<strong>vistributions</strong> has been on CRAN for a few months now while we were fixing bugs and making the API stable. All feedback is welcome. Issues (bugs and feature requests) can be posted to <a href="https://github.com/rsquaredacademy/vistributions/issues">github tracker</a>. For help with code or other related questions, feel free to reach out to us at <a href="mailto:pkgs@rsquaredacademy.com" class="email">pkgs@rsquaredacademy.com</a>.
</p>
</section>



 ]]></description>
  <category>R</category>
  <category>distributions</category>
  <guid>https://blog.rsquaredacademy.com/posts/introducting-vistributions/</guid>
  <pubDate>Thu, 14 Mar 2019 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
