How environments work in R and what lazy evaluation is

Reading time:

time

min

January 8, 2018

Knowledge of the way how R evaluates expressions is crucial to avoid hours of staring at the screen or hitting unexpected and difficult bugs. We’ll start with an example of an issue I came accross a few months ago when using the <code class="highlighter-rouge">purrr::map</code> function. To simplify, the issue I had: <label class="margin-toggle" for="wat">⊕</label><input id="wat" class="margin-toggle" type="checkbox" /><img class="fullwidth" src="/blog-old/assets/article_images/2017-10-30-r-function-evaluation/wat.jpg" /> wat <figure class="highlight"> <pre><code class="language-r" data-lang="r">makePrintFunction <- function(index) { function() { print(index) } } printFunctions <- lapply(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2 printFunctions <- purrr::map(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 3</code></pre> </figure> Since I came across the issue, <code class="highlighter-rouge">purrr::map</code> <a href="https://github.com/tidyverse/purrr/commit/b041e7897bc882037b7b5044a53e585c217a9b5a" target="_blank" rel="noopener noreferrer">has changed</a> and this example no longer applies. To simulate it, let’s use a simplified implementation of <code class="highlighter-rouge">map</code> function. You should be able to just copy-paste the code in this article and run it: <figure class="highlight"> <pre><code class="language-r" data-lang="r">map <- function(range, functionToApply) { result <- vector("list", length(range)) for (i in range) { result[[i]] <- functionToApply(i) } return(result) } makePrintFunction <- function(index) { function() { print(index) } } printFunctions <- lapply(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2 printFunctions <- map(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 3</code></pre> </figure> <hr /> How to fix that? If you don’t already know to fix that issue, you’ll quickly find out. This is quite a common problem and the solution is to use the <code class="highlighter-rouge">force</code> function as follows: <figure class="highlight"> <pre><code class="language-r" data-lang="r">makePrintFunction <- function(index) { force(index) function() { print(index) } } printFunctions <- lapply(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2 printFunctions <- map(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2</code></pre> </figure> <hr /> It works! But … why? This could be a great moment to just carry on - the problem is solved. You’ve heard about lazy evaluation and know that <code class="highlighter-rouge">force()</code> is useful in fixing such issues. But then again, what does lazy evaluation mean in this context? Let’s take a look at the magical <code class="highlighter-rouge">force</code> function. It consists of two lines: <label class="margin-toggle" for="Huh?">⊕</label><input id="Huh?" class="margin-toggle" type="checkbox" /><img class="fullwidth" src="/blog-old/assets/article_images/2017-10-30-r-function-evaluation/whaa.jpg" /> Huh? <figure class="highlight"> <pre><code class="language-r" data-lang="r">force # function (x) # x # <bytecode: 0x18e0920> # <environment: namespace:base></code></pre> </figure> Wait, what’s going on here? Does this mean that I can simply call <code class="highlighter-rouge">index</code> instead of <code class="highlighter-rouge">force(index)</code> and it will still work? <figure class="highlight"> <pre><code class="language-r" data-lang="r">makePrintFunction <- function(index) { index function() { print(index) } } printFunctions <- lapply(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2 printFunctions <- map(1:3, function(i) makePrintFunction(i)) printFunctions[[2]]() # 2</code></pre> </figure> <hr /> Let’s get to the bottom of this There are two factors that cause the issue we are facing. The first one is lazy evaluation. The second is the way environments work in R. Lazy evaluation The way R works is that it doesn’t evaluate an expression when it is not used. Let’s take a look at an example that you can find in Hadley’s book <a href="http://adv-r.had.co.nz/Functions.html" target="_blank" rel="noopener noreferrer">http://adv-r.had.co.nz/Functions.html</a>: <figure class="highlight"> <pre><code class="language-r" data-lang="r">f <- function(x) { 42 } f(stop("This is an error!")) # 42 f <- function(x) { force(x) 42 } f(stop("This is an error!")) # Error in force(x): This is an error!</code></pre> </figure> Another useful example to better understand that expressions are evaluated at the moment they are used: <figure class="highlight"> <pre><code class="language-r" data-lang="r">printLabel <- function(x, label = toupper(x)) { x <- "changed" print(label) } printLabel("original") # CHANGED printLabel <- function(x, label = toupper(x)) { force(label) x <- "changed" print(label) } printLabel("original") # ORIGINAL</code></pre> </figure> <label class="margin-toggle" style="font-size: 0.8em; text-decoration: underline;" for="promises-note"> sticky note</label><input id="promises-note" class="margin-toggle" type="checkbox" />Please note that promises mentioned here are something different than promises package used to handle concurrent computations. These semantics are described in R language definition <a href="https://cran.r-project.org/doc/manuals/r-patched/R-lang.html#Argument-evaluation" target="_blank" rel="noopener noreferrer">R language definition</a>: <blockquote>The mechanism is implemented via promises. When a function is being evaluated the actual expression used as an argument is stored in the promise together with a pointer to the environment the function was called from. When (if) the argument is evaluated the stored expression is evaluated in the environment that the function was called from. Since only a pointer to the environment is used any changes made to that environment will be in effect during this evaluation. The resulting value is then also stored in a separate spot in the promise. Subsequent evaluations retrieve this stored value (a second evaluation is not carried out).</blockquote> <hr /> How environments work Every function object has an environment assigned when it is created. Let’s call it environment A. When the function is invoked, a new environment is created and used in the function call. This new environment inherits from environment A. <figure class="highlight"> <pre><code class="language-r" data-lang="r">a <- 1 f <- function(a) { a <- a + 1 a # <-- debug here } f(5) # Browse[2]> environment(f) # <environment: R_GlobalEnv> # # Browse[2]> environment(f)[["a"]] # [1] 1 # # Browse[2]> environment() # <environment: 0x3fa2db0> # # Browse[2]> environment()[["a"]] # [1] 6</code></pre> </figure> This is what the environments hierarchy is at this point: <img src="/blog-old/assets/article_images/2017-10-30-r-function-evaluation/f_environment.png" alt="Environments hierarchy" /> <hr /> How does our example work without force <iframe src="https://docs.google.com/presentation/d/e/2PACX-1vSKFPfZYFTrwk80iVABBOR-FFhHMkRWqmTNK3F7Nh4tHI2HKqX2whfhdWtz_nbZuDCWye_ieY89twAG/embed?start=false&loop=false&delayms=3000" width="100%" height="400px" frameborder="0" allowfullscreen="allowfullscreen"></iframe> Environment 0x3fa2db2 inherits from mpfEnv and points to <code class="highlighter-rouge">index</code> variable which is stored in 0x3fa2db0. <code class="highlighter-rouge">index</code> variable is not going to be copied to environment 0x3fa2db2 until it is used there. <hr /> How does our example work with force <iframe src="https://docs.google.com/presentation/d/e/2PACX-1vTrd_zP2jASGGStgE3afxNNoiroWOVp_QHkbriEgs58ARDdvENemU1iMoEGReUPUbRIcY6NnDrUmrDr/embed?start=false&loop=false&delayms=3000" width="100%" height="400px" frameborder="0" allowfullscreen="allowfullscreen"></iframe> <hr /> You shouldn’t come across this issue while using most high-order functions: R 3.2.0 (2015) changelog: <ul><li>Higher-order functions such as the apply functions and Reduce()now force arguments to the functions they apply in order toeliminate undesirable interactions between lazy evaluation andvariable capture in closures. This resolves PR#16093.</li></ul> Purrr issue fixed in March 2017: https://github.com/tidyverse/purrr/issues/191 I hope this knowledge will save you some time if you stumble upon such issues in the future. <hr /> Until next time!