Questions are below. My solutions will be available after the tutorials are all finished. The whole point of these worksheets is for you to use your lecture notes to figure out what to do. In tutorial, the TAs are available to guide you if you get stuck. Once you have figured out how to do this worksheet, you will be prepared to tackle the assignment that depends on it.
There is probably too much here for you to finish in an hour, but I encourage you to continue later with what you were unable to finish in tutorial.
Packages
library(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.2.1 ✔ readr 2.2.0
✔ forcats 1.0.1 ✔ stringr 1.6.0
✔ ggplot2 4.0.3 ✔ tibble 3.3.1
✔ lubridate 1.9.5 ✔ tidyr 1.3.2
✔ purrr 1.2.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
The Boat Race
Each year, the Oxford and Cambridge University rowing teams race against each other on the River Thames in London (England). For the 1992 race, the weights (in pounds) of the participants on each team were recorded, and can be found in https://ritsokiguess.site/datafiles/boat_race.txt.
Take a look at the data file, and describe how the data values are separated one from the next.
Solution
Click on the URL, and see that the data values are separated by a single space. First comes the name of the university the rower comes from, then a single space, then the rower’s weight in pounds.
\(\blacksquare\)
Read the file into a dataframe and display at least some of it.
Solution
The data values are separated by a single space, so you need read_delim with a single space as the second input. My habit is to save the (often long) URL into a variable first:
Rows: 18 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: " "
chr (1): university
dbl (1): weight
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
rowers
You can check, by looking under the column heading, that the university names are text and the weights really are numbers (dbl means “double-precision decimal number”).
The alternative below works, but you have some extra work to do to explain why it works:
rowers0 <-read_delim(my_url)
Rows: 18 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: " "
chr (1): university
dbl (1): weight
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
rowers0
To get full credit for doing it this way (if it were on an assignment), you would need to draw the reader’s attention to this line:
Delimiter: " "
This means that read_delim guessed, by looking at the file before reading it in, that the data values seemed to be separated by single spaces. It had to guess, because you didn’t say what to look for.
This also works, but in this course is wrong:
rowers0 <-read.table(my_url, header =TRUE)rowers0
If you do it this way, it reveals that you are not paying attention. In the course outline it says “I expect you to do things as they are done in this course”, and in the lecture it says that read_delim is how to read a file of this type. You may have learned things differently elsewhere, but I do not use read.table in this course at all (and indeed, not very many base R ideas).
If you know anything about rowing, you might be a bit suspicious about there being nine observations, since a rowing team usually has eight people. See the last part of the question.
Extra (very long; you can read this in detail later): in this course, there is quite often a story of how the data came from the world to be a nice tidy data file for you. This is one of those cases. I don’t know how much of the story you’ll understand now, but I will do what I can to explain. Some of this comes from “choosing data” and some of it comes from “tidying data”, which is a bit further down the road.
The first bit is what is known as “web scraping”, which is not otherwise in the course, but you might be interested anyway. The data came from Wikipedia, to be precise here.1 Wikipedia articles are nice because the tables in them are easy to scrape once you know how. Web-scraping stuff lives in a package called rvest, and the first bit is not unlike reading data from a file:
library(rvest)
Attaching package: 'rvest'
The following object is masked from 'package:readr':
guess_encoding
This reads the entire HTML code of that web page into a variable I called html. This is structured as two things: the header of the web page, containing things like the title, and the body, containing all the rest of it.
The next part turns the HTML into something we can work with in R. This bit says “take the HTML and pull out all the things in it that are tables” (specifically, that are HTML TABLEs). We are going to be making a lengthy pipeline, rather like an extended group-by and summarize:
There are four tables in the document, and they display as an R list, the details of which you don’t need to concern yourself with now. The table we want is the third one. I explain the header thing (below) in a moment. Let’s grab just the third table (we don’t need the others at all):
html %>%html_table(header =FALSE) %>%pluck(3)
Now you see why I said there were no headers: there are actually two rows of headers, one saying which university the rower is from (first Oxford, and then scroll right to see Cambridge), and the second row is what we would normally think of as headers. I didn’t want to be dealing with that, so I let html_table choose some column names for me, the things beginning with X. I can be pretty relaxed about dealing with this, because all I want is the weights, which are in X4 for the Oxford rowers and X7 for the Cambridge ones. So those are the only two columns I need.
I don’t need all the rows, either; the first two are headers, and the last one is notes, so I only want rows 3 through 11. select chooses columns and slice chooses rows by number (these are from the “choosing things” lecture):
html %>%html_table(header =FALSE) %>%pluck(3) %>%select(X4, X7) %>%slice(3:11)
The weights are in funky units (that we will have to deal with shortly), but the immediate problem for making boxplots is that this is the wrong shape. We want one column of weights, and a second column saying which university that rower comes from. This is what pivot_longer makes. This comes from the “tidying data” lecture later, so don’t worry about making sense of it now:
There are now 18 rows, one row per person, with that person’s weight, and the text in col saying which column that weight came from.
You can go back and check that the first row here is the first Oxford rower, and the second one is the first Cambridge rower, and so on. Let’s identify the things in col with the proper university next:
mutate creates a new column in terms of an old one (from the “choosing things” lecture). Our new column is called university, and the value in it is a “recoding” of the values in col: X4 in col needs to be turned into Oxford in university, and X7 into Cambridge.
All right, those weights in weight_txt.2 What on earth does “st” mean? If you are in the UK, you measure weights of humans in “stones”, with 1 stone being 14 pounds. So the first rower weighs this many pounds:
13*14+4
[1] 186
So we now have to get those numbers out for each rower, and then we have to use them to calculate weights in pounds. The extraction is a bit fiddly, using something called “regular expressions”:
In words: take the column weight_txt and divide it up into a column called stone and a column called lbs. The recipe for doing that is the second input to separate_wider_regex. It says:
grab any number of any character and call it stone.
look for the letters st with a space either side, but don’t call it anything.
grab any number of any character and call it lbs.
look for the letters lb with a space in front, but don’t call it anything.
It is probably not obvious how this works. The idea is that the regular expression parser finds a way to make the whole thing match, and the only way to match the st with a space either side is to have the first “any number of any character” match only the 13 or 14 before st, and the only way to match the lb at the end as well is to have the second “any number of any character” match the decimal number of pounds in between the text st and the text lb.
The last-but-one thing to say here is that separate_wider_delim makes the column weight_txtdisappear, on the basis that after you’ve done the extraction, you usually don’t need the original any more (as here), though there is an option cols_remove = FALSE by which you can keep it if you need to. The last thing to say here is that separate_wider_regex is a more complicated version of separate_wider_delim, which we will see later; we could have used that if the weights had been written like 13:4 for 13 stone 4 lbs, with a predictable something separating each number.
From here, a mutate to calculate the weights in pounds, allowing for the fact that stone and lbs are actually still text, even though they look like numbers, so we need to convert them to numbers before doing arithmetic with them:3
and the final steps are to grab only the columns we need now, and to save it to a file for you. There is a write_delim that is the opposite of read_delim:4
First take a look at the type of variables you have: one categorical (the name of the university) and one quantitative (the weight of the rower). An appropriate graph for variables of this type is a side-by-side boxplot.
In ggplot the categorical variable is x because it goes horizontally, and the quantitative one is y (vertically):
ggplot(rowers, aes(x = university, y = weight)) +geom_boxplot()
This is the best plot. About the only other useful plot at all is above and below histograms. To get those, make a histogram of weights, and then facet by university, displaying the results in one column:
If you go this way, you will almost certainly have to experiment with the number of bins. One of the automatic bin choices is unlikely to help you here, since so many of the bins are empty. Also, the purpose of the plot is to compare the distributions, like the boxplot, so you really need the histograms to be above and below, with a common \(x\)-scale, not left and right with a common count scale.
\(\blacksquare\)
Would you say, based on your plot, that the average or typical weights of the rowers on the two teams are similar or different? Explain briefly.
Solution
The horizontal lines across the boxes on a boxplot are the medians of the distributions of weights. These medians are, I would say, very similar, especially given the amount of variability. (Have an opinion and defend it.)
From a boxplot, you cannot say anything about means, because they do not appear on a boxplot. With the kinds of distributions you have here, the mean is not a very sensible summary anyway, because of the outliers.
The use of the word “typical” in the question is meant to guide you towards a measure of centre, which might be mean, median, or even mode. A boxplot only shows you the median, which, as discussed, is a sensible measure of centre here anyway, so discuss that. (If your plot was the over-and-under histograms, you can try to figure out where, say, the medians are, or even use the mode. I care mostly about your thought process, not so much about the precise answer you get.)
\(\blacksquare\)
Each rowing team consists of eight rowers plus a cox, whose job is to keep the rowers in tempo. The cox does not row themselves. Which of the nine individuals in each team do you think is the cox? Explain briefly.
Solution
The obvious guess is “the low outlier”. But you also need to say something about why: if the cox does not row, this means that the other rowers are expending energy moving the cox as well as the boat. Thus it is an advantage to have a cox who is as light in weight as possible. Hence the low outlier is most probably the cox. (This was indeed the case.)
Aside: The positions in an 8-person rowing team are, from the front of the boat, “bow”, numbered positions 2 through 7, and “stroke”. The cox sits at the back and is the only one facing forwards! This means that the cox is also responsible for making sure the boat stays straight, or, in the Boat Race which is on a real river, following the curves of the river.
The cox on a rowing team has a similar role to the conductor in an orchestra, except that the musicians in an orchestra are not trying to move the conductor across the water as fast as possible!6
\(\blacksquare\)
Diet Wars 1
312 subjects took part in a comparative study of three diets: a low-fat diet (labelled 1 in the dataset), a Mediterranean diet (labelled 2), and a low-carbohydrate diet (labelled 3). Each subject was randomly allocated to one of the diets. One measure of success of a diet program is (of course) how much weight a subject lost, but another is whether they stayed with their diet until the end of the study (1 in the Status column) or whether they dropped out of the study (0 in the Status column). The data are in http://datafiles.ritsokiguess.site/diet-wars-1.txt, one row per subject.
Take a look at the data file (in your web browser). How is it laid out? How, therefore, will you read it in to a dataframe?
The data file having a .txt extension should make you wonder whether there is something special about it that will need care reading it in. As it turns out, the data are in aligned columns with more than one space between (at least for the data values themselves if not for the column headers), so read_table is going to be the thing.
There are indeed 312 rows (one per subject), and the two variables promised. Give the dataframe a suitable name; diets is better than diet because one of the columns is called Diet and it is not helpful to be confusing yourself.
Extra: this is the ex1921 data set from the Sleuth3 package, but it came to me like this:
Sleuth3::ex1921
as a frequency table, and I had to do a certain amount of re-formatting to get it in shape for you. I’ll talk about that later, when we see this data set again.
Make a suitable graph of these data. Hint 1: are those numbers in your dataset meaningful as numbers? If not, what kind of variables are they? Hint 2: to make a variable categorical even though it looks like a number, put it inside factor, ie. use something like factor(x) in your graph
Take the hint first: the numbers in the dataset are actually labels for categories (for both variables). (When we return to this data set, we will work on making these values look like categories.)
Hence, despite appearances, what we have here is a pair of categorical variables, and the right graph is a grouped bar chart. When you have two categorical variables, one of them is x and one of them is fill. To decide which is which, think about what you would like to have next to each other for comparison. I think you want the number of completed and dropped (Status) next to each other within each diet, which means that Diet needs to be x and Status needs to be fill. Attempt 1 (bad):
ggplot(diets, aes(x = Diet, fill = Status)) +geom_bar(position ="dodge")
Warning: The following aesthetics were dropped during statistical transformation: fill.
ℹ This can happen when ggplot fails to infer the correct grouping structure in
the data.
ℹ Did you forget to specify a `group` aesthetic or to convert a numerical
variable into a factor?
There is a bar for each Diet, but nothing for each Status. To figure out why not, read the warning message. The key is “did you forget […] to convert a numerical variable into a factor?”, and hint 2 tells you how to do that:
ggplot(diets, aes(x = Diet, fill =factor(Status))) +geom_bar(position ="dodge")
Better. Try it the other way around and see how it compares. The thing that is fill is the one that needs the factor() around it. The logic to this one is that completing the study is an outcome (response), and the Diet is explanatory: something that might affect whether or not the subject completes the study or drops out. If you have a variable that’s a response, that’s the one that belongs in fill.
Extra: if you think your numerical variables are genuinely quantitative, a scatterplot is the thing, but this happens:
ggplot(diets, aes(x = Diet, y = Status)) +geom_point()
and there a lot more subjects than that. The problem is that there were (say) a lot of people on diet 2 that completed the study, so that one point at \((2, 1)\) is actually a lot of points all plotted on top of each other. To see them all, you can “jitter” the points:
ggplot(diets, aes(x = Diet, y = Status)) +geom_jitter()
What this does is to plot each point “near” to where it belongs, but not exactly at where it belongs, so that all the points are in slightly different places. The rule geom_jitter uses is that each point is no more than halfway to the next value, so that all the points with Status above 0.5 on the graph are actually 1’s, that is, people who completed the study.
This is not the best graph, so we are not going to interpret it (below), but one thing you do see now is that no matter which diet, more people completed the study than dropped out.
Interpret your graph, bearing in mind what we want to compare. If you wish, comment on something that made your graph difficult to interpret.
What we want to compare is completion rates across the diets. As you go across the page, the number of people dropping out increases (that what 0 for Status means) and the number of people completing the study is more or less constant (in fact it is highest for diet 1). So the proportion of people completing the study is lowest for diet 1 (low-fat) and highest for diet 3 (low-carbohydrate).
Something that probably annoyed you while you were thinking about this is all the effort you had to put into converting the numbers into diets and what the status numbers meant. Wouldn’t it have been better for me to give you the data with named rather than numbered categories? Well, yes, but one thing we’ll be learning on a later worksheet is how you can do that conversion yourself before you draw the graph, so when the time comes to interpret it, you have a much easier job (and so does your reader).
Extra: another way of doing grouped bar charts is to stack the bars, but carefully. This is especially the case when you are comparing proportions, as we are here:
ggplot(diets, aes(x = Diet, fill =factor(Status))) +geom_bar(position ="fill")
This scales all the stacked bars to have the same height in total, so that to compare the proportions of people that dropped out on each diet, you compare the size of the red bits: lowest for diet 1, highest for diet 3.
This of course still does not get around the issue of mapping the numbers into something meaningful for your reader.
Hummingbirds and flowers
The tropical flower Heliconia is fertilized by hummingbirds, a different species for each variety of Heliconia. Over time, the lengths of the flowers and the form of the hummingbirds’ beaks have evolved to match each other. The length of the Heliconia flower is therefore an important measurement. Does it differ among varieties?
The data set at http://ritsokiguess.site/datafiles/heliconia_long.csv contains the lengths (in millimetres) of samples of flowers from each of three varieties of Heliconia: bihai, caribaea red, and caribaea yellow.
Read in and display (some of) the dataset.
Solution
This is a .csv, so there is no particular difficulty:
Rows: 54 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): variety
dbl (1): length
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
heliconia
54 observations of flower variety and length, with the varieties as described in the question.
\(\blacksquare\)
Why would a boxplot be a suitable graph for these data? Explain briefly.
Solution
In short, one categorical variable variety and one quantitative variable length. Make sure you say which variable is quantitative and which categorical, or else it looks as if you don’t know. Saying a bit more, the categorical variable variety denotes groups that we want to compare for length. (A categorical variable might only identify the observations; in that case, each “group” on a boxplot would only contain one observation!)
This is nothing special apart from getting the right variables in the right places. On a boxplot, the grouping variable is x and the quantitative one is y:
ggplot(heliconia, aes(x = variety, y = length)) +geom_boxplot()
\(\blacksquare\)
What do you learn from your boxplot? Explain briefly.
Solution
There are three things to say about how these distributions compare: centre (here median), spread (IQR) and shape:
centre: bihai has clearly the largest median (furthest up the page), and caribaea yellow has the lowest.
spread: caribaea red has the largest spread (IQR) because its box is the tallest. The other two distributions have about the same spread.
shape: bihai is skewed to the right (longer upper whisker), while the other two distributions are roughly symmetric.
\(\blacksquare\)
An alternative graph in this situation is a set of three histograms next to each other. Draw this graph. Consider your aims in drawing the graph; you will need to think about what “next to each other” most usefully means for you. Hint: problem 7.9(c) in PASIAS.
Solution
The idea of three histograms in one plot is meant to make you think about facets. Your first attempt might look something like this:
the point of this plot, like that of the boxplot, is to compare the three distributions. Having the distributions left and right like this makes it difficult to compare centres and spreads; it would be much easier if the graphs were above each other, in one column.
each histogram appears to have only two or three bins, even though I specified six bins in the graph. (My choice of six was based on there being about 20 observations in each group, 54 observations divided by 3 groups being an average of 18 per group.) The problem here is that the number of bins is shared among the three histograms; the lengths overall go from about 32 to about 50, so the six bins are about 32–35, 35–38, 38–41, … , 47–50 and they are used on all three graphs. The bihai length values are higher than the others, as we saw on the boxplot, so they use only the highest two of these bins. To fix this, use more bins so that each histogram has about the right number. You can experiment (I found this way that 20 bins was pretty good), or note that if the three sets of lengths didn’t overlap at all, you would multiply the number of groups by the number of bins to get 18. These three groups do overlap a little, so 18 bins might be a little too many.
With all this in mind, the way to get the distributions side by side so that you can compare them is to put them in one column. There are actually two ways to do that. If you like facet_wrap, use that with ncol = 1 to get one column of graphs. You don’t get specific control over the layout with facet_wrap, but the use of ncol to get all the graphs in one column achieves the desired effect:
Note that each histogram now has something like six bins. The second one, for caribaea red, has more because this distribution is more spread out than the others (and all the bins are the same width), but each distribution has at least five bins (in the case of bihai, one of them is empty), which for me is enough to give at least some sense of shape. If it is not for you, use slightly more bins overall, maybe 20.
In the PASIAS question referred to in the hint, I also talk about facet_grid. Use that if it makes more sense to you. The idea there is that you specify the way to arrange the sub-graphs in the plot (the three histograms, in this case). The overall arrangement has an x and a y, distinct from the x and y in the aes() (the ones in aes() refer to each individual graph, rather than the overall arrangement). What you do is to say which (categorical) column refers to y first (before a squiggle) and x second (after the squiggle). If you don’t have anything to go in either slot, you use a dot.
Thus, the facet_grid version of the above graph is this:
variety is determining how the histograms are arranged up and down, and there is nothing arranging them left and right.
Extra: facet_grid can be useful when you have two categorical variables, since then you can make a two-dimensional array of subplots, with one categorical variable going across and the other one going up and down. For example, you could use this if you have two quantitative and two categorical variables; you make the sub-plots scatterplots and then you can see how the trends change if either or both of the categorical variables change.
\(\blacksquare\)
Footnotes
It is called “The Boat Race” because, like many British things, it was the first organized rowing race, starting in 1829.↩︎
I used a “disposable” name for these, saving weight for the actual weights in pounds that we are going to calculate.↩︎
Note the chr at the top of each of those columns.↩︎
The default delimiter is one space, which I am happy with, so I don’t have to specify it, but if you want a different delimiter, like say _, you would add "_" as the second input to your write_delim, as you would for read_delim when you read it back in.↩︎
in the bash shell, which is the one that R Studio’s “terminal” uses, cat is how you display a file. You can also have a bash code chunk, or even a python one.↩︎
It’s up to you whether you call this one boxplot, or one graph containing three side-by-side boxplots. Think about what makes more sense to you, and what will make more sense to your reader.↩︎
