Find APIs

️✅ Learning objectives

• Search the web for APIs.
• Search for API-wrapping packages.
• Use browser developer tools to find undocumented APIs.

library(dplyr)
library(stringr)
library(tidyr)
library(tools)
library(httr2)
library(anyapi)
library(apisniffer)

Search the web for APIs

• apis.guru
• Public APIs
  • GitHub repo
• {site} + api
• developer(s).{site}
• GitHub/{organization}

• apis.guru is in the {anyapi} package (on github)
• I plan to add Public APIs (in addition)
  • More active dev
  • There’s an open issue to add OpenAPI specs when available
• Google search is a GET API,
  • But google.com/robots.txt Disallow’s hitting /search.
  • And probably need LiveHTML to get something useful out of it
• developers.google.com
• Sometimes api(s).{site} but I don’t have an example right now
• WaPo GitHub doesn’t have an API, but it has some of what you might hope to use an API to find.

Search for API-wrapping packages

General tips for text filtering

• tolower(FIELD) to find “API”, “api”, “Api”, etc
• \\b in regex pattern for “word boundary”
  • "\\bapi\\b" = “api surrounded by spaces, (), newline, etc”

• These are generally useful, but we’ll use them specifically for packages

Searching CRAN packages

api_pkgs <- tools::CRAN_package_db() |> 
  dplyr::as_tibble() |> 
  dplyr::filter(
    stringr::str_detect(tolower(Description), "\\bapi\\b") |
      stringr::str_detect(tolower(Title), "\\bapi\\b")
  ) |> 
  dplyr::select(Package, Title)
api_pkgs

#> # A tibble: 900 × 2
#>    Package           Title                                                      
#>    <chr>             <chr>                                                      
#>  1 academictwitteR   "Access the Twitter Academic Research Product Track V2 API…
#>  2 ACEsearch         "'ACE' Search Engine API"                                  
#>  3 acled.api         "Automated Retrieval of ACLED Conflict Event Data"         
#>  4 adbcdrivermanager "'Arrow' Database Connectivity ('ADBC') Driver Manager"    
#>  5 adbcpostgresql    "'Arrow' Database Connectivity ('ADBC') 'PostgreSQL' Drive…
#>  6 adbcsqlite        "'Arrow' Database Connectivity ('ADBC') 'SQLite' Driver"   
#>  7 adformR           "Get Adform Ads Data via the 'Windsor.ai' API"             
#>  8 adobeanalyticsr   "R Client for 'Adobe Analytics' API 2.0"                   
#>  9 adsDataHubR       "Google Ads Data Hub API Client"                           
#> 10 adwordsR          "Access the 'Google Adwords' API"                          
#> # ℹ 890 more rows

• CRAN_package_db() returns 69 columns of info (everything that can be in the DESCRIPTION file)
• Title = short description, generally sentence case.
• Description = paragraph or so about the package
• Could also check Author and/or Authors@R for credit to API owner.

Searching the R Universe

• rOpenSci project for package discovery
• Web interface at r-universe.dev
• API (under development) at r-universe.dev/api/search

• Users and organizations can set up their own universes
• Likely to be different by the time I do next iteration of these slides

R Universe API: Request

r_universe_apis_req <- httr2::request("https://r-universe.dev/api/search") |> 
  httr2::req_url_query(
    q = "api",
    all = TRUE,
    limit = 100
  )

r_universe_apis_resps <- r_universe_apis_req |> 
  httr2::req_perform_iterative(
    httr2::iterate_with_offset(
      "skip",
      start = 0,
      offset = 100,
      resp_pages = \(resp) ceiling(httr2::resp_body_json(resp)$total/100)
    )
  )

• API isn’t done.
  • No documented pagination, but…
  • Skip parameter will work!

R Universe Results

r_universe_apis_resps |> 
  httr2::resps_data(
    \(resp) {
      httr2::resp_body_json(resp)$results |> 
        tibble::enframe(name = NULL) |> 
        tidyr::unnest_wider(value)
    }
  ) |> 
  dplyr::select(Package, Title)

#> # A tibble: 1,418 × 2
#>    Package         Title                                                        
#>    <chr>           <chr>                                                        
#>  1 hoopR           "Access Men's Basketball Play by Play Data"                  
#>  2 wehoop          "Access Women's Basketball Play by Play Data"                
#>  3 gptstudio       "Use Large Language Models Directly in your Development\nEnv…
#>  4 geosapi         "GeoServer REST API R Interface"                             
#>  5 googleLanguageR "Call Google's 'Natural Language' API, 'Cloud Translation' A…
#>  6 RobinHood       "Interface for the RobinHood.com No Commission Investing Pla…
#>  7 hereR           "'sf'-Based Interface to the 'HERE' REST APIs"               
#>  8 nhlapi          "A Minimum-Dependency 'R' Interface to the 'NHL' API"        
#>  9 salesforcer     "An Implementation of 'Salesforce' APIs Using Tidy Principle…
#> 10 oaii            "'OpenAI' API R Interface"                                   
#> # ℹ 1,408 more rows

Sniff API requests in browser

Browser developer tools

• Differs browser-to-browser, showing Chrome
• ctrl-shift-I (cmd-shift-I) or right click > Inspect
• Network tab
• Filter to Fetch/XHR
• Right click > Header Options > Path
• Demo: Amazon suggestions

• Microsoft Edge is also a Chromium-based browser, so should be same there
• Fetch & XHR are two JavaScript APIs for making requests.
  • XHR = XmlHttpRequest, but not used just for XML.
  • Fetch is more modern version, but both are used.
• Importantly: They’re how web pages often make API requests on your behalf.
• Load Amazon, then ctrl-shift-i (make sure it’s empty)
• Show clearing with the circle/line icon
• Click the search box
• Type “Web APIs with R”, pausing to see requests & results
• Point out “Path” column
• Filter to “suggestions”
• Single click last one
• Walk through Headers, Payload, Response
• Right click > Copy > Copy as cURL (bash)
  • Can paste this and use httr2::curl_translate()

Sniff API requests with {apisniffer}

• {apisniffer}
• Goal:
  • Load page
  • (optional) Interact
  • Returns tibble of API info
  • (maybe) Also returns functions to access detected APIs

• Currently doesn’t allow interaction
• Returns raw data, not tibble
• Can use httr2::url_parse() to break url into pieces
• request objects have sections by {httr2} function