Tell APIs who you are

TODO

• Emphasize difference between working with what people have done vs understanding how things SHOULD work (knowing the OAuth2 specification backwards and forwards isn’t helpful when people use nonstandard terminology or implement something weird; PKCE often doesn’t work, for example)
• Make sure this makes sense before pagination.

️✅ Learning objectives

• Provide information in your request about how you’re accessing the API.
• Find authentication information in API docs.
• Authenticate a request with an API key.
• Authenticate a request with OAuth.

library(httr2)

How can I tell the API how I’m accessing it?

What is a user agent?

• “User agent” = software that accesses the web
• User-Agent request header = string IDing the “user agent”
  • Request header = extra info sent with request
• {httr2} attaches a default User-Agent header

httr2::request("https://example.com") |> httr2::req_dry_run()
#> GET / HTTP/1.1
#> Host: example.com
#> User-Agent: httr2/1.0.1 r-curl/5.2.1 libcurl/8.3.0
#> Accept: */*
#> Accept-Encoding: deflate, gzip

• It’s arguable whether your code is the user agent or httr2 is.
• Headers are the things at the top of a request, outside the request itself
  • Usually what you’re expecting back + user agent

Should I override the default user agent?

• Are you hitting the API a lot?
• Are you providing reusable code for others to hit the API?
• Does the API documentation mention it?

• Default is fine almost all the time
• Include UA in package (with option for user to expand)
• If you use your code repeatedly/automatically, provide a UA
• Documentation might ask for it (like Jim found in api.crossref.org docs)

How do I override the default user agent?

httr2::req_user_agent(req, string = NULL)

httr2::request("https://example.com") |> 
  httr2::req_user_agent("MyUserAgent/1.0 (more details; separate with ;)") |> 
  httr2::req_dry_run()
#> GET / HTTP/1.1
#> Host: example.com
#> User-Agent: MyUserAgent/1.0 (more details; separate with ;)
#> Accept: */*
#> Accept-Encoding: deflate, gzip

• Convention is Software slash version,
• Details in parentheses,
• Separate with ;
• But can be any string

An experimental .Rprofile user agent function

.httr2_ua <- function() {
  # Recreate the default httr2 string.
  versions <- c(
    httr2 = as.character(utils::packageVersion("httr2")),
    `r-curl` = as.character(utils::packageVersion("curl")),
    libcurl = curl::curl_version()$version
  )
  paste0(names(versions), "/", versions, collapse = " ")
}
.req_ua <- function(req) {
  httr2_string <- .httr2_ua()
  me <- "Jon Harmon"
  url <- "https://wapir.io"
  email <- "jonthegeek+useragent@gmail.com"
  string <- glue::glue(
    "{httr2_string} ({me}; {url}; mailto:{email})"
  )
  httr2::req_user_agent(req, string = string)
}

• usethis::edit_r_profile()
• Start with . so it doesn’t show in my environment.
• In general, UA should tell the API how you’re accessing it
• I’ll likely put a version of this into {nectar} soon.

How can I find authentication information?

What is authentication?

• Authentication: Verifying who you are.
• Authorization: Granting permissions to do things (based on authentication)
• Auth: Used interchangeably for both

• Internet makes a big deal about this distinction if you Google
• For users, the difference doesn’t matter
  • You auth with the server so the server will auth you

What are some types of authentication?

🔴 HTTP Basic: username + password sent with request

        📜 deed to your house

🟠 API Key: password-like thing sent with request

        🔑 key to your house

🟡 Bearer Token: shorter-lived, limited key

        💳 keycard

🟢 OAuth: multistep process to generate a bearer token

        🕵️ background check to issue keycard

• Almost nobody uses HTTP basic for APIs anymore.
• If you gave an ex a key to your house, you can change the lock without disrupting your life too much
• Name “bearer token” is stupid; anybody can use an API key, too, so the “bearer” part is weird to highlight
• Nothing enforces difference between bearer tokens and api keys; it’s a vocabulary thing, which API makers might screw up
• Keycard can be limited to open certain doors, not open others
• End result of OAuth is a bearer token
• There are other things, but these four cover almost everything.
• Other schemes like api keys in cookies, SSL certs not discussed here

API Auth Documentation

• Sadly, no standard
• OpenFEC
• Google Calendar

• Usually either at top of docs or in each relevant endpoint
• Sometimes different endpoints have different requirements (particularly OAuth scopes, more below)
  • This endpoint only requires read, this one requires read and write
• Often docs give the basics, but might not have all details for actually using OAuth in code, for example.

OpenAPI: securitySchemes

If you have the APID, use it!

• components$securitySchemes = ways to auth
• security (top level) = default schemes
• OpenFEC
• Google Calendar

• APID = API description in OpenAPI format
• Often still necessary to dig through docs for details
• I suspect the 2 “apiKey in query” schemes for OpenFEC are the default key vs a user-specific key, their way of implementing scopes.

How can I prepare my system for authentication?

Practice safe git

• Run usethis::git_vaccinate()
• Often usethis::use_git_ignore(".Renviron")

• git_vaccinate() “Adds Rproj.user, .Rhistory, .Rdata, .httr-oauth, .DS_Store, and .quarto to your global (a.k.a. user-level) .gitignore.”
• httr2 actually puts auth info in a more-secure, harder-to-check-in place.
• We’ll often put keys in personal .Renviron, but you might want a project-specific one for special keys.
• Even better: use {keyring} (TODO: Go down this rabbit hole!)

Use keyring

• install.packages("keyring")
• keyring::key_set(service)
  • keyring::key_set("FEC_API_KEY")
• keyring::key_set_with_value(service, password = NULL)
• keyring::key_get(service)
• May need to copy keyring to env for packages
  • Sys.setenv(FEC_API_KEY = keyring::key_get("FEC_API_KEY"))

• {keyring} works with your operating system’s key manager
• key_set() has you enter the key in a secure password window
• key_set_with_value() useful if a function fetches a key (you never need to see it)
• key_get() returns that key

How can I authenticate a request using API keys?

Where do I send API keys?

• in: query
  • httr2::req_url_query(.req, ...)
• in: header
  • httr2::req_headers(.req, ..., .redact = NULL)
  • .redact = character vector of headers to hide in print
• in: cookie
  • httr2::req_headers(.req, Cookie = "name=val1; name2=val2", .redact = "Cookie")

• APID does a good job of describing where these can go.
• Header more secure than query, so use that when available
• ... = name-value pairs, case-insensitive (but usually just copy-paste from docs).
• Cookies are meaningful in the browser, but for your code it’s easiest just to think of them as a type of header.

How can I authenticate FECAPI requests?

OpenFEC APID

request("https://api.open.fec.gov/v1") |> 
  req_headers("X-Api-Key" = "DEMO_KEY", .redact = "X-Api-Key")

#> <httr2_request>
#> GET https://api.open.fec.gov/v1
#> Headers:
#> • X-Api-Key: '<REDACTED>'
#> Body: empty

request("https://api.open.fec.gov/v1") |> 
  req_headers("X-Api-Key" = keyring::key_get("FEC_API_KEY"), .redact = "X-Api-Key")

#> <httr2_request>
#> GET https://api.open.fec.gov/v1
#> Headers:
#> • X-Api-Key: '<REDACTED>'
#> Body: empty

• Notice that the key never prints anywhere in the keyring version
• Sys.getenv() is also an option, but then the key is stored in plain text in your .Renviron file.

Authenticating with nectar

{nectar} 📦 translates APID to {httr2}

request("https://api.open.fec.gov/v1") |> 
  nectar::req_auth_api_key(
    location = "header", 
    parameter_name = "X-Api-Key", 
    api_key = Sys.getenv("FEC_API_KEY")
  )

#> <httr2_request>
#> GET https://api.open.fec.gov/v1
#> Headers:
#> • X-Api-Key: '<REDACTED>'
#> Body: empty

• Technically only PART of the purpose of {nectar}
• Aimed primarily at package authors, but I’m finding more and more use cases
• I’ll recommend it for book helper functions

How can I authenticate a request using OAuth?

Oauth terminology: Participants

term(s)	meaning
user	the person who you’re acting as
application, app	your R code
client, oauth application	you create this at oauth host
oauth host	the API

• Oauth is complicated! Don’t feel bad if you’re confused!
• We’ll go over the “dance” on an upcoming slide
• The point of the multiple steps is to make it hard for attacker to intercept, and minimize what they can do if they do
• Also to allow third parties to act on your behalf; can be confusing, because we’re both the user and the app!
• Bold = what I’ll call it on other pages
• App could be package, could be website, could be a separate API (“login with Google”)
• Client has an id and a secret, that’s how your app logs in
  • We’ll talk more about these in a minute
• Client also has settings like valid redirect URLs (next slide)

Oauth terminology: Things

term(s)	meaning
scope	string(s) describing specific capabilities
authorization code	very temporary key
oauth token, token	the real key, often with extra info

• If app only hits read endpoints, only needs to request read scopes
  • Or if this user has only asked to read
• Auth code can only be sent to specified URLs
• The “real key” here is a bearer token.
• Token often includes a “refresh token”, longer-lived shortcut around the auth process.
  • “This user already gave me permission, I just need an updated keycard.”

Oauth terminology: Places

term(s)	meaning
authorization url	where to send initial request
redirect url(s)	where host can send auth codes
token url	where app can exchange auth codes

• User is sent to auth url in browser, sees authorization request directly from host
• Auth code can only be sent to redirect URLs
  • Often will be localhost, where httr2 sets up a web host to listens for a response
  • Isn’t default, but you often need to specify a port for host to accept this as a valid redirect
• Request from app to token url is sent server-to-server, no user/browser in-between. Auth code in header in this case, generally.

The OAuth “dance”

• user to app: Hit this API for me!
• <app sends user to host @ auth url in browser>
• host to user: Can client act as you with these scopes?
• user to host: Yes
• <host sends user to app @ redirect url with auth code>
• app directly to host @ token url: Here’s my client + user’s auth code
• host directly to app: Here’s an oauth token for that user (with the requested permissions)

TODO: Image of OAuth dance

• This is sometimes called 3-legged auth
  • Took me forever to differentiate “app” from “user” to make the 3 legs make sense
  • Host is at multiple urls but it just counts as one leg
• Step 2: App sends client id and permissions as part of that introduction
• Not covered: Then (not much) later app sends bearer token to host with API requests, and host checks that token can do those things.

OAuth credential dangers

• 🟢 Client id: Like knowing package name
• 🟡 Authorization code: Unlikely to be an issue
  • Only sent to provided redirect url
  • Extremely short lived (often minutes or less)
• 🟡 Refresh token:
  • This + client secret for access token
  • Usually revoked if you auth from scratch

• Yellows here are light green, really
• Auth code: Need client secret for it to do anything
• Refresh token: Longer lived, but otherwise similar to auth code

OAuth credential dangers (cont)

• 🟡 Client secret: Iffy
  • Can pretend to be you (user still needs to say ok)
  • Can your client do anything special?
    • Installed (e.g. Slack app)?
    • API usage limits (e.g. YouTube)?
• 🟠 Access token: The thing we’re protecting
  • Can do whatever it’s authorized to do
  • Usually easy to revoke
• 🔴 Username + password: We don’t want to know these

• Hadley says client secrets aren’t a big deal, but I disagree in a lot of cases.
• Your auth page at host can be copied, so that’s not a real concern, but…
• If localhost is valid redirect, they can trick user to auth as you
• Only matters if client itself can do something
• YouTube has client limits in addition to user limits

How can I use OAuth in R?

Configure your client

Construct one client object for your code

httr2::oauth_client(
  id, 
  token_url, 
  secret = NULL, 
  key = NULL, 
  auth = c("body", "header", "jwt_sig"), 
  auth_params = list(), 
  name = hash(id)
)

• id = Client ID
• token_url = URL where clients exchange authorization codes for tokens
• secret = Client secret
• Often auth = "header"
• If this client has multiple uses: name = unique for this use case
  • Defines where tokens are cached
  • Eg, Log in once for personal, another time for work (but see cache_key in next function)
• I don’t know yet why we don’t specify auth_url here (it’s in a later step)

OAuth client demo

yt_client <- oauth_client(
  id = Sys.getenv("YOUTUBE_CLIENT_ID"), 
  token_url = "https://oauth2.googleapis.com/token",
  secret = Sys.getenv("YOUTUBE_CLIENT_SECRET")
)

httr2::req_oauth_auth_code()

httr2::req_oauth_auth_code(
  req,
  client,
  auth_url,
  scope = NULL,
  pkce = TRUE,
  auth_params = list(),
  token_params = list(),
  redirect_uri = oauth_redirect_uri(),
  cache_disk = FALSE,
  cache_key = NULL
)

• I’m trying to figure out why auth_url isn’t wrapped in client
  • Best guess is there are use cases of same client at different auth urls
  • Maybe YouTube does this?
  • Dev vs production?
• pkce = “Proof key for code exchange”. Good if API supports it, but I often have to turn it off
• I can’t think of examples of auth_params or token_params
  • would be ~headers required by host
  • auth_params sent to auth_url, token_params sent to token_url
• cache_disk = Set this TRUE if you can
  • cache_key = unique name if you’ll use this client for multiple tokens

Oauth request demo

playlists <- request("https://youtube.googleapis.com/youtube/v3") |> 
  req_url_path_append("playlists") |> 
  req_url_query(part = "snippet", mine = TRUE, maxResults = 50) |> 
  req_oauth_auth_code(
    client = yt_client, 
    auth_url = "https://accounts.google.com/o/oauth2/v2/auth",
    scope = "https://www.googleapis.com/auth/youtube",
    redirect_uri = "http://127.0.0.1:8888"
  ) |> 
  req_perform()

Leftovers

Automating OAuth

• If you can, use httr2 cache: easiest, but
  • auto-deletes when 30 days old
  • fills logs w/ “Caching httr2 token in …” messages
• httr2::req_oauth_bearer_jwt() if you have JSON web token (service account)
• httr2::req_oauth_refresh() if you have a refresh token
  • httr2::oauth_flow_auth_code() once to get refresh

Browser cookies

This will feel hacky because it is hacky.

• Install EditThisCookie browser extension
• Use API in browser
• Open EditThisCookie extension
• Options > “Choose the preferred export format for cookies” > Netscape HTTP Cookie File
• Open EditThisCookie extension
• Export
• Paste into a file at path
• httr2::req_cookie_preserve(req, path)

• Or cookie can be used in header

Meeting Videos

Cohort 1

Meeting chat log

LOG