👋Learning to code? Check out our new dedicated beginners platform -Jiki!
1 exercise
In other parts of the syllabus, we have seen various data types with different characteristics.
This Concept will look at ways to store multi-dimensional, heterogenous data. In practice, most real-world data is like this, so we are now getting to the heart of how R is (mostly) used in practice.
Over the decades, R has added multiple data types to handle tabular data.
This syllabus will focus mainly on tibbles, but it is useful to know about some alternatives.
data.table
Tibbles (described below) and data.tables are both well-respected attempts to improve on the traditional data.frame in Base R.
There is inevitably much argument about which is "better".
Maybe there is some degree of consensus around the following points (even if they will be criticized as simplistic).
tibble is optimized mainly for ease of use, and integration with the Tidyverse ecosystem.data.table is optimized mainly for raw power and scalability, especially when working with very large datasets.In any case, data.table is not available within Exercism, so it is mentioned here just for completeness.
data.frame
In Base R, a data.frame is a list of equal-length vectors.
This can be thought of as a rectangular table of data, in which each column is homogeneous, but each row can (and usually does) contain different types of data.
An example to illustrate this:
# create the column vectors
languages <- c("Fortran", "R", "Python", "Julia")
created <- c(1957, 1993, 1991, 2012)
has.syllabus <- c(FALSE, TRUE, TRUE, TRUE)
# join columns to create the dataframe
df <- data.frame(languages, created, has.syllabus)
df
#> languages created has.syllabus
#> 1 Fortran 1957 FALSE
#> 2 R 1993 TRUE
#> 3 Python 1991 TRUE
#> 4 Julia 2012 TRUE
# look at the structure
str(df)
#> 'data.frame': 4 obs. of 3 variables:
#> $ languages : chr "Fortran" "R" "Python" "Julia"
#> $ created : num 1957 1993 1991 2012
#> $ has.syllabus: logi FALSE TRUE TRUE TRUE
We have a column of character strings, a column of numbers and a column of booleans. When scaled up, this is an intuitive way to represent many collections of real world data.
tibble
The data.frame design is old.
Multi-decade experience, plus changing patterns of how R is used, led to a redesign, creating a modernized alternative in the Tidyverse: tibbles.
Compared to Base R, tibbles have:
In short, a tibble aims to "do less and complain more", also described as "lazy and surly".
The types are usually interchangeable: any function which accepts a data.frame will also accept a tibble, and vice versa.
Conversions between the types are easy, with as_tibble(df) and as.data.frame(tbl).
For new work, using tibbles will probably help you create more robust code.
However, legacy code and legacy data is very plentiful in the R world, so the data.frame is likely to remain common for a long time.
# column vectors are same as for data.frame
library(tibble)
tbl <- tibble(languages, created, has.syllabus)
tbl
# A tibble: 4 × 3
#> languages created has.syllabus
#> <chr> <dbl> <lgl>
#> 1 Fortran 1957 FALSE
#> 2 R 1993 TRUE
#> 3 Python 1991 TRUE
#> 4 Julia 2012 TRUE
str(tbl)
#> tibble [4 × 3] (S3: tbl_df/tbl/data.frame)
#> $ languages : chr [1:4] "Fortran" "R" "Python" "Julia"
#> $ created : num [1:4] 1957 1993 1991 2012
#> $ has.syllabus: logi [1:4] FALSE TRUE TRUE TRUE
Note the default print format: the comment line with dimensions is printed automatically, and column types are also displayed.
Tibbles are a core part of the Tidyverse, so add them with either library(tibble) or library(tidyverse).
Documentation is fairly extensive, in the Tidyverse style:
Most simply, we can use the tibble() function to join column vectors, as for data.frame().
An example of this was shown in a previous section.
If it is more convenient to enter values row-wise, the corresponding function is tribble().
tbl_r <- tribble(
# column names marked with tilde prefix
~languages, ~created, ~has.syllabus,
"Fortran", 1957, FALSE,
"R", 1993, TRUE,
"Python", 1991, TRUE,
"Julia", 2012, TRUE
)
tbl_r
# A tibble: 4 × 3
#> languages created has.syllabus
#> <chr> <dbl> <lgl>
#> 1 Fortran 1957 FALSE
#> 2 R 1993 TRUE
#> 3 Python 1991 TRUE
#> 4 Julia 2012 TRUE
In practice, there are dozens of ways to create tibbles, as they are the default output format from a diverse range of Tidyverse functions. We will return to this in a future Concept.
The Functional Programming Concept discussed the purrr library to manipulate vectors and lists (1-D data structures).
For dataframes (whether traditional or tibbles), the corresponding library to use is dplyr.
We introduced dplyr previously, in the Switch Concept.
That just used a few utility functions, but now we can start to explore the rest of this large library.
Dataframes, including tibbles, can be treated as lists of column vectors, so list indexing recovers a specified column.
tbl
# A tibble: 4 × 3
#> languages created has.syllabus
#> <chr> <dbl> <lgl>
#> 1 Fortran 1957 FALSE
#> 2 R 1993 TRUE
#> 3 Python 1991 TRUE
#> 4 Julia 2012 TRUE
tbl$created
#> [1] 1957 1993 1991 2012
A dataframe can also be indexed with matrix-style indexing.
tbl[c(2, 4), 1:2]
# A tibble: 2 × 2
#> languages created
#> <chr> <dbl>
#> 1 R 1993
#> 2 Julia 2012
In modern R with the Tidyverse ecosystem, dplyr functions are generally more flexible and convenient, and will be the focus for the rest of this Concept.
Get a single column with pull() with the name or sequential number (negative numbers to count right-to-left).
tbl |> pull(created)
#> [1] 1957 1993 1991 2012
This is the same result as tbl$created, but using a pipeline-friendly function.
To get multiple columns, the appropriate function is select(), which is highly versatile.
Get (or drop) columns based on properties of their name or type.
# Range with position and/or name
tbl |> select(1:created)
# A tibble: 4 × 2
#> languages created
#> <chr> <dbl>
#> 1 Fortran 1957
#> 2 R 1993
#> 3 Python 1991
#> 4 Julia 2012
# Exclude a column
tbl |> select(!created)
# A tibble: 4 × 2
#> languages has.syllabus
#> <chr> <lgl>
#> 1 Fortran FALSE
#> 2 R TRUE
#> 3 Python TRUE
#> 4 Julia TRUE
# Use type of column
tbl |> select(where(is.numeric))
# A tibble: 4 × 1
#> created
#> <dbl>
#> 1 1957
#> 2 1993
#> 3 1991
#> 4 2012
Multiple criteria are allowed, using Boolean operators &, | and ! (and, or not).
Column names that are valid R identifiers do not need quotes within a select().
Invalid names (e.g. those including spaces) can be enclosed in backticks, though renaming them might be better.
The select() function can work with a range of helper functions to pick column names: starts_with, contains, num_range and various others.
matches allows full RegEx matching.
See the documentation for details.
Such power seems quite silly with our toy dataframe of languages.
Fortunately, the starwars tibble is included in dplyr, giving us something bigger to practice with.
# limit display to top 3 rows of non-list columns
starwars |>
select(!where(is.list)) |>
head(3)
# A tibble: 3 × 11
#> name height mass hair_color skin_color eye_color birth_year sex gender homeworld species
#> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr>
#> 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human
#> 2 C-3PO 167 75 NA gold yellow 112 none masculine Tatooine Droid
#> 3 R2-D2 96 32 NA white, blue red 33 none masculine Naboo Droid
# pick a subset of columns
starwars |>
select(name | ends_with("color")) |>
head(5)
# A tibble: 5 × 4
#> name hair_color skin_color eye_color
#> <chr> <chr> <chr> <chr>
#> 1 Luke Skywalker blond fair blue
#> 2 C-3PO NA gold yellow
#> 3 R2-D2 NA white, blue red
#> 4 Darth Vader none white yellow
#> 5 Leia Organa brown light brown
Clearly, dplyr provides powerful ways to select columns by name.
Can we do similar things with row names?
No!
Traditional R dataframes can have row names, but (after a history of bugs and performance issues) row names are not allowed in tibbles.
If you want names, put them in a <chr> column (typically column 1), used like any other column.
Import functions such as as_tibble() will create this automatically when importing data with named rows.
If this row-name limitation seems oddly restrictive, remember that most large database systems handle tables the same way: Oracle, SQL Server, PostgreSQL, MySQL...
Get rows matching some criteria with filter(), or exclude them with filter_out().
starwars |>
select(name:mass) |>
filter(between(height, 150, 165) & !is.na(mass))
# A tibble: 4 × 3
#> name height mass
#> <chr> <int> <dbl>
#> 1 Leia Organa 150 49
#> 2 Beru Whitesun Lars 165 75
#> 3 Nien Nunb 160 68
#> 4 Ben Quadinaros 163 65
Filter criteria can be arbitrarily complex, but always based on row contents.
If row numbers are known, we can use a variety of slice() functions.
starwars |>
select(name | homeworld) |>
slice(20:25)
# A tibble: 6 × 2
#> name homeworld
#> <chr> <chr>
#> 1 Palpatine Naboo
#> 2 Boba Fett Kamino
#> 3 IG-88 NA
#> 4 Bossk Trandosha
#> 5 Lando Calrissian Socorro
#> 6 Lobot Bespin
# random sample of rows
starwars |>
select(name | homeworld) |>
slice_sample(n = 4)
# A tibble: 4 × 2
#> name homeworld
#> <chr> <chr>
#> 1 Shaak Ti Shili
#> 2 Luminara Unduli Mirial
#> 3 Grievous Kalee
#> 4 Palpatine Naboo
To remove duplicate rows, use distinct().
First caveat: the copy-on-modify default means that the original tibble usually remains unchanged.
Most modifications are applied column-wise.
Column names can be changed with rename(newname = oldname), or rename_with() to apply a function.
A typical use would be cleaning up imported names to make them easier to work with in R, by removing whitespace and forcing a consistent format for related names.
Note the syntax within rename().
The contents of column oldname are bound to name newname, hence the order.
Column order can be changed with relocate().
Specified column(s) are moved to the left-most position(s) by default, but a .before or .after argument can be used for finer positioning.
sw <- starwars |> select(name:species) |> slice(1:4)
sw
# A tibble: 4 × 11
#> name height mass hair_color skin_color eye_color birth_year sex gender homeworld species
#> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr>
#> 1 Luke Skywalker 172 77 blond fair blue 19 male masculine Tatooine Human
#> 2 C-3PO 167 75 NA gold yellow 112 none masculine Tatooine Droid
#> 3 R2-D2 96 32 NA white, blue red 33 none masculine Naboo Droid
#> 4 Darth Vader 202 136 none white yellow 41.9 male masculine Tatooine Human
sw |> relocate(c(species, homeworld), .after = name)
# A tibble: 4 × 11
#> name species homeworld height mass hair_color skin_color eye_color birth_year sex gender
#> <chr> <chr> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr>
#> 1 Luke Skywalker Human Tatooine 172 77 blond fair blue 19 male masculine
#> 2 C-3PO Droid Tatooine 167 75 NA gold yellow 112 none masculine
#> 3 R2-D2 Droid Naboo 96 32 NA white, blue red 33 none masculine
#> 4 Darth Vader Human Tatooine 202 136 none white yellow 41.9 male masculine
For bigger changes, mutate() lets you:
NULL
Clearly, mutate() is powerful, potentially confusing, and a reason to be very grateful for copy-on-modify.
There is no obvious reason to care about the Body Mass Index of Star Wars characters, but just in case:
starwars |>
select(c(name, species, height, mass)) |>
mutate(BMI = mass / (height / 100)^2) |>
head(4)
# A tibble: 4 × 5
#> name species height mass BMI
#> <chr> <chr> <int> <dbl> <dbl>
#> 1 Luke Skywalker Human 172 77 26.0
#> 2 C-3PO Droid 167 75 26.9
#> 3 R2-D2 Droid 96 32 34.7
#> 4 Darth Vader Human 202 136 33.3
When you want to operate on a subset of the columns with functions such as mutate(), the select() |> mutate() sequence in the above example is one option.
Only the selected columns will be in the result.
Alternatively, it can be convenient to use pick() within the mutate() call:
starwars |> mutate(pick(c(name, species, height, mass)), BMI = mass / (height / 100)^2) |> head(4)
When you want to operate on a subset of the columns with functions such as `mutate()`, the `select() |> mutate()` sequence in the above example is one option.
Only the selected columns will be in the result.
Alternatively, it can be convenient to use `pick()` _within_ the `mutate()` call:
```R
starwars |> mutate(pick(c(name, species, height, mass)), BMI = mass / (height / 100)^2) |> head(4)
# A tibble: 4 × 15
#> name height mass hair_color skin_color eye_color birth_year sex gender homeworld species films vehicles
#> <chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr> <chr> <chr> <lis> <list>
#> 1 Luke Skywal… 172 77 blond fair blue 19 male mascu… Tatooine Human <chr> <chr>
#> 2 C-3PO 167 75 NA gold yellow 112 none mascu… Tatooine Droid <chr> <chr>
#> 3 R2-D2 96 32 NA white, bl… red 33 none mascu… Naboo Droid <chr> <chr>
#> 4 Darth Vader 202 136 none white yellow 41.9 male mascu… Tatooine Human <chr> <chr>
# ℹ 2 more variables: starships <list>, BMI <dbl>
Only the picked columns are used in the mutation, but all columns are returned.
Row-wise operations are less common for modifying single tibbles (merging multiple tibbles will be discussed in a later concept).
One exception: arrange() sorts rows by the values in one or more columns.
tbl
# A tibble: 4 × 3
#> languages created has.syllabus
#> <chr> <dbl> <lgl>
#> 1 Fortran 1957 FALSE
#> 2 R 1993 TRUE
#> 3 Python 1991 TRUE
#> 4 Julia 2012 TRUE
tbl |> arrange(languages)
# A tibble: 4 × 3
#> languages created has.syllabus
#> <chr> <dbl> <lgl>
#> 1 Fortran 1957 FALSE
#> 2 Julia 2012 TRUE
#> 3 Python 1991 TRUE
#> 4 R 1993 TRUE
Dataframes, whether traditional or tibbles, are central to the way modern R is typically used.
Most of the Tidyverse functions (not just dplyr) take tibbles as input and/or create them as output.
This concept just provided a brief introduction, barely scratching the surface of what is possible.
Later concepts will discuss several other aspects of dataframes (within the technical contraints of Exercism).
Edit via GitHub
Learn Dataframes