STAC32 Assignment 2

Packages

library(tidyverse)

You are expected to complete this assignment on your own: that is, you may discuss general ideas with others, but the writeup of the work must be entirely your own. If your assignment is unreasonably similar to that of another student, you can expect to be asked to explain yourself.

If you run into problems on this assignment, it is up to you to figure out what to do. The only exception is if it is impossible for you to complete this assignment, for example a data file cannot be read. (There is a difference between you not knowing how to do something, which you have to figure out, and something being impossible, which you are allowed to contact me about.)

You must hand in a rendered document that shows your code, the output that the code produces, and your answers to the questions. This should be a file with .html on the end of its name. There is no credit for handing in your unrendered document (ending in .qmd), because the grader cannot then see whether the code in it runs properly. After you have handed in your file, you should be able to see (in Attempts) what file you handed in, and you should make a habit of checking that you did indeed hand in what you intended to, and that it displays as you expect.

Hint: render your document frequently, and solve any problems as they come up, rather than trying to do so at the end (when you may be close to the due date). If your document will not successfully render, it is because of an error in your code that you will have to find and fix. The error message will tell you where the problem is, but it is up to you to sort out what the problem is.

Baseball players

A baseball fan has 59 baseball cards each containing information about a different player, specifically these:

height (inches)
weight (pounds)
bat: L = bats left-handed, R = bats right-handed, S = switch-hitter (can bat either left-handed or right-handed)
throw: L = throws left-handed, R = throws right-handed
field: is a pitcher or a fielder (who also bats)
average: the earned-run average for a pitcher, the batting average for a fielder

The data are in http://datafiles.ritsokiguess.site/baseball-players.csv.

(1 point) Read in and display (some of) the data.

As ever:

my_url <- "http://datafiles.ritsokiguess.site/baseball-players.csv"
baseball <- read_csv(my_url)

Rows: 59 Columns: 6
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): bat, throw, field
dbl (3): height, weight, average

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

baseball

You should have 59 rows and these six columns. Eyeballing the average column, you see that the earned run average (for the pitchers) is a very different kind of number from the batting average (for the fielders).

Give your dataframe a sensible name like baseball or players or something else that tells you what the data are about.

Extra: this is the baseball data set from the Rfit package, but in the original, the column field was 0 or 1, with 0 being the pitchers. I prefer my categorical variables to look like categorical variables (for example, it makes no sense to average these 0 and 1 values), so I straightened this out for you. (I used recode_values, which we will see in the course shortly.)

(3 points) Work out the median value of average for each of the fielders and pitchers.

What determines whether a player is a fielder or a pitcher is their value in the column field, so group by that first, and then work out the median value of average:

baseball %>% 
  group_by(field) %>% 
  summarize(med_average = median(average))

The median batting average of the fielders is 0.270 (batting averages are usually given to three decimals, since fielders bat a lot of times during the season), and the median earned run average of the pitchers is 3.56.

I realize that an average of an average is a weird concept, but let that pass for now.

(2 points) Based on what you know about these data, why would it not make sense to work out the median of all the values in average for all the players combined?

In the data description, you see that the value in average is a different kind of thing for pitchers and for fielders, one that has a different meaning beyond being an “average”. So it makes no sense to combine a batting average, a number like 0.300, with an earned run average, a number like 4.

Extra: in baseball, a batting average is the number of times a batter gets a hit (that is, reaches at least first base by hitting the ball) divided by the number of times the batter went up to bat. This is a number less than 1, and a batting average bigger than 0.300 is considered good.

An earned run average is the number of earned runs given up by a pitcher divided by the number of innings pitched, then multiplied by 9 to scale it up to runs per complete game. (Things like runs scored due to fielding errors do not count as earned runs, to make it a better measure of effectiveness of just a pitcher and not his teammates as well.)

To confuse the issue further, a high batting average is good, but a low earned run average is good (since the aim of a pitcher is to give up as few runs as possible). A number like about 3 is good.

(3 points) How many players bat with one hand and throw with the other? (Do not count switch-hitters in this calculation.)

Asking “how many” indicates that we are counting something, namely the combinations of things in bat and throw, which are both categorical. So start there:

baseball %>% 
  count(bat, throw)

From here, you can pick out what you need. Ignore the last two rows (switch hitters) and look at the second and third rows: 4 players bat left and throw right, and 3 players bat right and throw left. This is not very many out of the total of 59 players; most of the players bat with the same hand they throw.

Perhaps better is to recognize that this output is itself a dataframe, and you can use what you know about choosing rows on this output. There are two things to do: ignore the switch-hitters, and ignore the people who bat and throw with the same hand. This could be two filter_outs one after the other:

baseball %>% 
  count(bat, throw) %>% 
  filter_out(bat == "S") %>% 
  filter_out(bat == throw)

or with when_any, which also works with filter_out:

baseball %>% 
  count(bat, throw) %>% 
  filter_out(when_any(
    bat == "S",
    bat == throw
  ))

I got this wrong the first time (negative conditions are tricky). I realized that I needed to omit the rows when any of those two conditions are true, not when they are all true.

You can even go a step further and summarize how many mismatched handers there are altogether:

baseball %>% 
  count(bat, throw) %>% 
  filter_out(when_any(
    bat == "S",
    bat == throw
  )) %>% 
  summarize(wrong_handers = sum(n))

that is, seven of them.

It also works (and is perhaps more efficient) to do the filter first and then do the counting:

baseball %>% 
  filter_out(when_any(
    bat == "S",
    bat == throw
  )) %>% 
  count(bat, throw)

or add them up as above.

If you don’t have filter_out or when_any, you’ll need to work around that. You can use filter with negative conditions (“what not to include”) and you will need to decide whether “and” or “or” is what you need as a result:

baseball %>% 
  filter(bat != "S", bat != throw) %>% 
  count(bat, throw)

It turns out that you need “and”. This is not easy to think about: one way to get here is to note that you only want left-handers and right-handers, and you only want the ones where bat and throw are different.

(2 points) Display all the columns whose names have “ht” as their last two letters, without naming or numbering those columns.

This is ends_with:

baseball %>% 
  select(ends_with("ht"))

Height and weight, as you might have guessed.

(3 points) Work out the mean of each of the columns whose names have “ht” as their last two letters, again without naming or numbering those columns.

Doing something with each of some columns you haven’t named is across with an anonymous function:

baseball %>% 
  summarize(across(ends_with("ht"), \(x) mean(x)))

The tricky part with these is getting the anonymous function in the right place. It should be inside the across, so it is after the comma after the thing that chooses the columns, and you will typically have a lot of close-brackets at the end. Here there are three: the first one closes mean, the second closes the across, and the last one closes the summarize.

(3 points) Display all the pitchers who weigh (strictly) more than 220 pounds. (You may display all the columns.)

This is choosing rows, so is a filter. We have two things that we want to be both true: the player is a pitcher rather than a fielder, and their weight is more than 220 pounds, so we need a when_all:

baseball %>% 
  filter(when_all(
    field == "pitcher",
    weight > 220
  ))

There are four of them.

If you don’t have when_all, do this as one filter with two conditions, or as two filters one after the other. The thought process is “I only want pitchers, and out of those, I only want the ones whose weight is bigger than 220”:

baseball %>% 
  filter(field == "pitcher") %>% 
  filter(weight > 220)

(3 points) For the fielders whose height is 72 inches or less, what is their median (batting) average?

This sort of thing is best done as a filter followed by a summarize to figure out the summary you want. The filter is the same idea as in the previous question: two things that both have to be true, so it is again a filter with a when_all:

baseball %>% 
  filter(when_all(
    field == "fielder",
    height <= 72
  )) %>% 
  summarize(med_avg = median(average))

Without when_all, a filter with two conditions (as below), or two filters one after the other:

baseball %>%
  filter(field == "fielder", height <= 72) %>%
  summarize(med_avg = median(average))

There are undoubtedly different ways to do this one, but getting the correct answer by a sensible method is good.

There is some credit for trying to do it with group_by, but then you’ll get more output than is desirable:

baseball %>% 
  group_by(field, height <= 72) %>% 
  summarize(med_avg = median(average))

`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by field and height <= 72.
ℹ Output is grouped by field.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(field, height <= 72))` for per-operation grouping
  (`?dplyr::dplyr_by`) instead.

The answer is on the second row (where field is fielder and height <= 72 is TRUE), but you and your reader have to wade through some irrelevant output that it would be better not to show. You might also be surprised that you can group by something that is not a column.

Another way to do this is to create a new column that is TRUE for the players that are 72 inches or less:

baseball %>% 
  mutate(short = (height <= 72)) %>% 
  group_by(field, short) %>% 
  summarize(med_avg = median(average))

`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by field and short.
ℹ Output is grouped by field.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(field, short))` for per-operation grouping
  (`?dplyr::dplyr_by`) instead.

I like to use extra brackets when I am creating a column based on a logical condition (otherwise you have to stare at short = height <= 72 to figure out exactly what it means: are you creating a new column of true and false, or are you asking whether short and height are the same thing, and then what?)

The advantage to this approach of defining a new column is that you can then tidy up after yourself, picking out only the row that you care about:

baseball %>% 
  mutate(short = (height <= 72)) %>% 
  group_by(field, short) %>% 
  summarize(med_avg = median(average)) %>% 
  filter(when_all(
    field == "fielder",
    short == TRUE
  ))

`summarise()` has regrouped the output.
ℹ Summaries were computed grouped by field and short.
ℹ Output is grouped by field.
ℹ Use `summarise(.groups = "drop_last")` to silence this message.
ℹ Use `summarise(.by = c(field, short))` for per-operation grouping
  (`?dplyr::dplyr_by`) instead.

and you get to the right answer, but at the expense of doing some extra calculation (three medians that you didn’t need to find, because you calculate them and then throw them away). A filter with two conditions is another way to do the last step if you don’t have when_all.

Houses in Watauga County

Watauga County is in North Carolina (US). A professor at Appalachian State University collected a sample of asking prices for houses for sale in Watauga County with three bedrooms and two bathrooms. The data are in https://datafiles.ritsokiguess.site/watauga.csv. The two columns are the name of the Neighbourhood where the house is, and the asking Price (in thousands of dollars).

(1 point) Read in and display (some of) the data.

No surprises here (it’s a .csv file):

my_url <- "https://datafiles.ritsokiguess.site/watauga.csv"
watauga <- read_csv(my_url)

Rows: 14 Columns: 2
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (1): Neighborhood
dbl (1): Price

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

watauga

I named the dataframe after the county where the houses are. You could just as well call the dataframe houses or something similar.

Extra: the major city in Watauga County is Boone, which is home to Appalachian State University. The dataset is House in the PASWR package, which contains datasets for a textbook Probability and Statistics with R by Ugarte, Militino, and Arnholt. The third author, who also maintains the R package, works at Appalachian State University.

(3 points) One of the neighbourhoods is called Valley Crucis in the dataset, but should actually be Valle Crucis (no “y”) everywhere it appears. Update your dataframe to make this correction, and save the result, showing the code that you used to make the correction.

The wording of the question is to suggest that you do not find a way to hand-correct the data (such as by editing the .csv file and reading it in again). This is dangerous because you have to be alert enough to see and correct every instance of “Valley Crucis” and it is far too easy to go wrong.

The best way here is to recognize that you are making a small correction to one column, and so replace_values is what you want. You can save back into the original dataframe, but I am going to do this a second way below so I need the original to work with again:

watauga %>% 
  mutate(Neighborhood = replace_values(
    Neighborhood,
    "Valley Crucis" ~ "Valle Crucis"
  )) -> watauga1

and display what you have now, to show that it came out right:

watauga1

and it looks as if all the changes got made.

replace_values runs inside a mutate (typically). Here, we’re creating a “new” column Neighborhood (that is to say, overwriting the old one), whose values are coming from Neighborhood, but wherever you see “Valley Crucis” you replace it with “Valle Crucis”.

If you don’t have replace_values, you’ll need to use case_when and handle the default values (that do not change):

watauga %>% 
  mutate(Neighborhood = case_when(
    Neighborhood == "Valley Crucis" ~ "Valle Crucis",
    .default                        = Neighborhood
  )) -> watauga

The logic here is “any other neighbourhoods, set them equal to what they were before.” This also seems to have worked:

watauga

Extra 1: another reason to make changes like this in code rather than by hand is that Dr Arnholt may add some houses to the dataset, some of which may also be labelled as from “Valley Crucis”, and you want to be able to run the same code to make the corrections no matter how often Dr Arnholt updates the dataset or how many houses it ends up containing.

Extra 2: you might also be thinking of rows_update for this one, but that only works when the update depends on another column. In the lecture example, we wanted to change the state of certain locations (that is to say, whether or not the state got updated depends on which location it was). Here, the update depends on only one column: the Neighborhood “Valley Crucis” needs to become the Neighborhood “Valle Crucis”, and so replace_values is called for.

(3 points) Blowing Rock is in the South of Watauga County. Valle Crucis and Green Valley are in the West. The other three Neighborhoods are in Boone (the largest city in the county). Create and save a dataframe with new column called Region that puts each neighbourhood into its correct region (West, South, or Boone as appropriate).

This is different from the previous question in that we are reclassifying all the houses, not just some of them. Hence recode_values is what you need now.

“The other three neighbourhoods are in Boone” is a clue that you can make this a default and not have to find out the names of those other neighbourhoods. So the best way is this:

watauga %>% 
  mutate(Region = recode_values(
    Neighborhood,
    "Blowing Rock"                    ~ "South",
    c("Valle Crucis", "Green Valley") ~ "West",
    default                           = "Boone"
  )) -> watauga
watauga

It’s nicest if you line up the ~ and =, so that it’s clear which ones are the old values (from Neighborhood) and which ones are the new values (to go into Region). Note that default needs a =, not a ~, because it is a special name rather than one of the old values.

Like replace_values, recode_values works within a mutate: define a new column (a genuinely new one here), in terms of an old one, and then specify the changes you want to make: “every instance of Blowing Rock in Neighborhood needs to become South in Region”, and so on.

It is slightly less good to name the old neighbourhoods one at a time, and repeat values on the right:

watauga %>% 
  mutate(Region = recode_values(
    Neighborhood,
    "Blowing Rock"  ~ "South",
    "Valle Crucis"  ~ "West",
    "Green Valley"  ~ "West",
    default         = "Boone"
  )) -> watauga
watauga

It still works, but you have repeated West for no apparent reason, and you have made it more difficult for your reader to check that all the neighbourhoods in the West have been correctly recoded.

If you don’t do the default thing, you need to find out the names of the other three neighbourhoods, like this:

watauga %>% 
  mutate(Region = recode_values(
    Neighborhood,
    "Blowing Rock"                              ~ "South",
    c("Valle Crucis", "Green Valley")           ~ "West",
    c("Cove Creek", "Park Valley", "Parkway")   ~ "Boone",
  )) -> watauga
watauga

If you don’t have recode_values, you will need one of the slightly awkward workarounds listed in the workarounds document, either the one using %in%:

watauga %>% 
  mutate(Region = case_when(
    Neighborhood == "Blowing Rock"                              ~ "South",
    Neighborhood %in% c("Valle Crucis", "Green Valley")         ~ "West",
    Neighborhood %in% c("Cove Creek", "Park Valley", "Parkway") ~ "Boone",
  )) -> watauga
watauga

or the one using fct_collapse, in which the new and old values are the other way around:

watauga %>% 
  mutate(Region = fct_collapse(
    Neighborhood,
    South =  "Blowing Rock",
    West = c("Valle Crucis", "Green Valley"),
    Boone = c("Cove Creek", "Park Valley", "Parkway")
  )) -> watauga
watauga

Extra 1: You may need to look back at the original data to see what the “other” neighbourhoods were. If you miss any, you’ll get an error:

watauga %>% 
  mutate(Region = recode_values(
    Neighborhood,
    "Blowing Rock"                    ~ "South",
    c("Valle Crucis", "Green Valley") ~ "West",
    c("Cove Creek", "Park Valley")    ~ "Boone",
  ))

I missed Parkway, which you can tell because it has an NA in its Region.

Extra 2: another way to make sure you catch any missing ones is to make recode_values throw an error if you forgot any, like this:

watauga %>% 
  mutate(Region = recode_values(
    Neighborhood,
    "Blowing Rock"                    ~ "South",
    c("Valle Crucis", "Green Valley") ~ "West",
    c("Cove Creek", "Park Valley")    ~ "Boone",
    unmatched                         = "error"
  ))

Error in `mutate()`:
ℹ In argument: `Region = recode_values(...)`.
Caused by error in `recode_values()`:
! Each location must be matched.
✖ Location 5 is unmatched.

This is the sort of thing to do “in development” rather than “in production”: that is to say, while you are still figuring out what to do, have it throw an error that you then fix. When you are happy that you have caught everything, then you can take out the unmatched line. (Or, you can leave it in, in case you will later have to deal with a new set of houses, some of which might be in different neighbourhoods.)

This error message tells you that the fifth Neighborhood is unmatched:

watauga0 %>% slice(5)

namely, Parkway.

Extra 3: this also works with left_join, but that should be a last resort for you if you cannot get anything else to work. The way you do that is to create a lookup table. The column with the neighbourhoods in it is most easily called Neighborhood as in the original dataframe, and you need to match the neighbourhoods to the regions one at a time (repeating the Regions):

lookup <- tribble(
  ~Neighborhood,  ~Region,
  "Blowing Rock", "South",
  "Valle Crucis", "West",
  "Green Valley", "West",
  "Cove Creek",   "Boone",
  "Park Valley",  "Boone",
  "Parkway",      "Boone"
)

Once again, it is clearer to line up the columns so that you can see which are Neighborhoods and which are Regions.

Then, left-join that onto the original data. I made a copy of the data we read in from the file so that there is no confusion with already having a Region column:

watauga0 %>% 
  left_join(lookup, join_by("Neighborhood"))

This is more work (you have to make the lookup table), but once you have done that, you might argue that the join is easier than the recode-values.

(2 points) Make a table showing the number of houses, and the mean and SD of asking price, for each Region.

This one is meant to be easy now that you have done the work of creating the Regions. The only thing to remember is to use n() to get the number of observations when you are also calculating other summaries:

watauga %>% 
  group_by(Region) %>% 
  summarize(n = n(),
            mean_price = mean(Price),
            sd_price = sd(Price))

The houses in Boone are actually cheaper than the houses in the South or West.

Extra: decide for yourself whether this is a low-cost-of-living area, or whether these data are some years old! As far as I can tell, the textbook from which these data come dates from 2015.