Easily access APIs with {httr2}

️✅ Learning objectives

• Use API docs to explore available endpoints.
• Fetch data from an API with {httr2}.
• Build a {httr2} request piece-by-piece.

library(jsonlite)
library(tibblify)
library(tidyr)
library(dplyr)
library(httr2)

“After we finish this chapter, we’ll be able to…” (objectives) - Not every possible API, but we’ll get the basics. - I’m also hoping to make it clear why httr2 is useful!

What can we do so far?

We can:

• Access APIs that can be accessed in a web browser
• Copy/paste or manually build URLs
• Pass an authentication key as part of a URL
• Fetch JSON data with {jsonlite}
• Fetch YAML data with {yaml}

We can’t yet:

• Access APIs that can’t be accessed in a web browser
• Build API requests systematically
• Access APIs that require authentication
• Fetch other types of data (raw text, images, videos, etc)

• Browser: As long as it isn’t using our login information without us realizing it.

• No browser: There are technically 9 types of requests (HTTP methods), we can do 1 of them so far.

• A lot of APIs use something called “oauth”, can be really confusing

• The endpoints we’ve hit haven’t had arguments, but almost all do

• We can get some specific things with download.file, but need to parse each with different packages

• …and other things we don’t yet know that we wish we could do

• We’ll use httr2 to deal with all of these things

How can I learn about API endpoints?

Motivating Example: OpenFEC

• OpenFEC: Federal Election Commission API
• API documentation usually describes endpoints
  • endpoint: URL for specific API function

2020 Presidential Candidates

• We’ll focus on the /candidates/ endpoint
• https://api.open.fec.gov/v1/candidates/?election_year=2020&office=P&api_key=DEMO_KEY

openfec_response <- jsonlite::read_json("https://api.open.fec.gov/v1/candidates/?election_year=2020&office=P&api_key=DEMO_KEY")

tibblify(openfec_response$results) |> 
  select(name, party_full, has_raised_funds) |> 
  head()

#> # A tibble: 6 × 3
#>   name                       party_full       has_raised_funds
#>   <chr>                      <chr>            <lgl>           
#> 1 753, JO                    NONE             FALSE           
#> 2 ABRAMSON, MAX              VETERANS PARTY   FALSE           
#> 3 ABRAUGH, MATTHEW M. MR.    NON-PARTY        FALSE           
#> 4 ACKER, BRANDON W           DEMOCRATIC PARTY FALSE           
#> 5 ACKER, RYAN                NON-PARTY        TRUE            
#> 6 ACORD, ROBERT BRADFORD LEE DEMOCRATIC PARTY FALSE

• What if we want a different year? Or a different office? Or any of the other 20 or so available parameters?

What is {httr2}?

• We’ll dig into what the letters mean in a couple slides.
• I learned how to pronounce httr2 from this logo

What do {httr2} calls look like?

Pipe-based API calls

candidates <- 
  request("https://api.open.fec.gov/v1") |> 
  req_url_path_append("candidates") |> 
  req_url_query(api_key = "DEMO_KEY") |> 
  req_url_query(
    election_year = 2020,
    office = "P"
  ) |> 
  req_perform() |> # Chapter 5 (next)
  resp_body_json() # Chapter 7

waldo::compare(candidates, openfec_response)

#> ✔ No differences

• I use |> base R pipe here. Pronounce it “and then”.
• httr2 calls usually involve a request, which you perform, and then you parse the response.
• We’re hitting the free, open api from the Federal Elections Commission.
• (step through)
• Here we build the request piece-wise
  • We’ll get into details of the functions in a few slides
• Then we perform the request. This actually hits the server.
• Then we parse what we get back with a resp_body function.
• Same result as the thing we got using jsonlite.
  • Because resp_body_json uses jsonlite.

Why “httr2”?

HTTP = HyperText Transfer Protocol

• “HyperText” = web content
• “Transfer” = exchange
• “Protocol” = rules
• “rules for exchanging web content”
• HTTP(S) = most of internet communication

• You may have seen http or https at the start of URLs (web addresses)
• “web content” =
  • originally text and links (HTML = HyperText Markup Language)
  • now data, images, videos, etc.
• “exchange” or “move”
• “S” means “secure”
• If anyone asks: 1.1 = most, 2.0 = 2-way
• 2 = he rewrote it for piping

How do I use {httr2}?

req_*() functions return httr2_request objects

req_fec <- request("https://api.open.fec.gov/v1")
req_fec_auth <- req_url_query(req_fec, api_key = "DEMO_KEY")
req_candidates <- req_url_path_append(req_fec_auth, "candidates")
req_candidates_president <- req_url_query(req_candidates, office = "P")

pres_2024 <- req_url_query(req_candidates_president, election_year = 2024) |> 
  req_perform() |> resp_body_json()

candidates_2022 <- req_url_query(req_candidates, election_year = 2022) |> 
  req_perform() |> resp_body_json()

req_calendar <- req_url_path_append(req_fec_auth, "calendar-dates")

• Imagine you’re working with an API, like the FEC API
  • What you’ll very often do.
• We’ll go through function specifics in a moment, just soak in the utility of separate objects
• This is super important, so let’s pause to sink this in
• I’m going to drink some water while you make sure you have that.

How do I build {httr2} requests?

request() & req_path_append()

request("https://api.open.fec.gov/v1/candidates/")

Cleaner: “main” request object + specific path

req_fec <- request("https://api.open.fec.gov/v1")
req_candidates <- req_fec |> 
  req_url_path_append("candidates")

req_candidates$url

#> [1] "https://api.open.fec.gov/v1/candidates"

• httr2 “thinks” in pieces
• You don’t have to think about the /
• We have to be careful about the “v1” in our request, though!
  • More on next slide.

Don’t use req_url_path()!

req_path_bad <- req_fec |> 
  req_url_path("candidates")
req_path_bad$url

#> [1] "https://api.open.fec.gov/candidates"

req_candidates$url

#> [1] "https://api.open.fec.gov/v1/candidates"

• You probably never want req_url_path()
• Notably the help docs don’t show an example for this one

req_url_query()

https://api.open.fec.gov/v1/candidates/?api_key=DEMO_KEY&office=P

req_pres <- req_candidates |> 
  req_url_query(
    api_key = "DEMO_KEY",
    office = "P"
  )

• Can add query parameters piecewise, even before path!

req_fec_auth <- req_fec |> 
  req_url_query(api_key = "DEMO_KEY")
req_candidates_auth <- req_fec_auth |> 
  req_url_path_append("candidates")
req_pres2 <- req_candidates_auth |> 
  req_url_query(office = "P")

identical(req_pres$url, req_pres2$url)

#> [1] TRUE

• “Query” = stuff after “?”
• Arguments for the endpoint

req_url_query(.multi)

req_url_query(req_candidates, office = c("H", "S"))

#> Error in `req_url_query()`:
#> ! All vector elements of `...` must be length 1.
#> ℹ Use `.multi` to choose a strategy for handling vectors.

.multi = "pipe"

req_url_query(req_candidates, office = c("H", "S"), .multi = "pipe")$url

#> [1] "https://api.open.fec.gov/v1/candidates?office=H|S"

.multi = "comma"

req_url_query(req_candidates, office = c("H", "S"), .multi = "comma")$url

#> [1] "https://api.open.fec.gov/v1/candidates?office=H,S"

.multi = "explode"

req_url_query(req_candidates, office = c("H", "S"), .multi = "explode")$url

#> [1] "https://api.open.fec.gov/v1/candidates?office=H&office=S"

• APIs often can’t handle multiple params with same name
• default = “error”
• . in .multi so it doesn’t collide with a parameter named “multi”
• “pipe” or “comma” to list them out in same parameter
• “explode” to separate them as separate copies of that parameter (what this API actually wants)

More httr2 to come!

• Chapter 4 = you are here
• Chapter 5 = “How can I get a lot of data from an API?”
• Chapter 6 = “How do I tell the API who I am?
• Chapter 7 = “How can I process API responses?”
• Chapter 8 = “How can I do other things with APIs?”

• 5 = Pagination, retries, throttling, parallelization, etc
• 6 = Authentication & other headers
• 7 = Parsing responses
• 6 = Methods (notably POST)

What do you think?

Please complete this survey!

• https://forms.gle/Wk9YWAHybRigfg7e8

Chapter To-Do

• Review existing slides.
• Update LOs.
• Write exercises.
• Write exercise solutions.
• Finalize LOs.
• Add slide(s) per exercise.
  • Link to relevant content slide(s) from exercise slides.
• Update content slides.
• Write speaker-note questions to ask cohort.

This is transient (the checks will clear on reload), and it’s meant to be.