Quickly parse API responses

TODO

• Merge back into “Get started with APIs”?

️✅ Learning objectives

• Parse nested lists with {tibblify}.
• Use an API’s OpenAPI description to determine the expected format of responses.
• Parse API responses with {tibblify} and the response description.

library(jsonlite)
library(tidyr)
library(dplyr)
library(tibblify)
library(yaml)
library(waldo)

Aside: Why rectangle?

• Apps usually “think” in objects
  • Preferred data: Objects-with-properties (example json)
• Data scientists usually compare many objects at once
  • Preferred data: data frames
    • Columns of variables (of same class)
    • Rows of observations (~objects)
• APIs are usually designed for the first model
• Even data APIs tend to “think” in objects
  • Because most programmers do

• This really belongs in the previous chapter or maybe intro, but I’m putting it here for this cohort.
• Apps and thus most programmers.
  • Even when you get a list of things, like on Amazon, each of those product blocks in the list is an html div object. It’s rarely really columns of similar data.
• Maybe this is obvious to some, but this felt like an epiphany to me.
• Fundamental difference between R and everything else.
  • Things like pandas/polars are attempts to make Python think like R.
• Really pause to discuss.

The tibblify package

Typical JSON data

demo_json.json

url <- "https://dslc-io.github.io/club-wapir/slides/schemas/demo_json.json"
demo_json <- jsonlite::fromJSON(url)

• When you click through, point out that the object-per-api is a perfectly reasonable standard, but that this data makes way more sense in R with a row per api, but a standardized column for each variable, like we saw in the last chapter & last slide
• Note that sometimes jsonlite::fromJSON() mangles data, jsonlite::read_json() is same thing without auto-parsing.

Rectangling demo_json manually

demo_json |> 
  tibble::enframe(name = "api_id") |>
  tidyr::unnest_longer(value, indices_include = FALSE) |> 
  tidyr::unnest_longer(value, indices_to = "version") |>
  tidyr::unnest_wider(value)

#> # A tibble: 3 × 8
#>   api_id        added updated swaggerUrl swaggerYamlUrl openapiVer link  version
#>   <chr>         <chr> <chr>   <chr>      <chr>          <chr>      <chr> <chr>  
#> 1 apis.guru     2015… 2023-0… https://a… https://api.a… 3.0.0      http… 2.2.0  
#> 2 fec.gov       2018… 2023-0… https://a… https://api.a… 3.0.0      http… 1.0    
#> 3 googleapis.c… 2020… 2023-0… https://a… https://api.a… 3.0.0      http… v3

• Wouldn’t it be nice if we could do all of that automatically?

Introducing tibblify

• {tibblify} 📦 to auto-convert hierarchical data to tibbles
• Super-charged tidyr::unnest_auto()
• Tibbles all the way down
• Experimental functionality for APIs

• (bullet 4): This probably belongs in a separate package, but for now it’s in a dev branch of tibblify.

Rectangling demo_json with tibblify

dj_tibblified <- tibblify::tibblify(demo_json)
dj_tibblified

#> # A tibble: 3 × 2
#>   .names                           versions
#>   <chr>                  <list<tibble[,7]>>
#> 1 apis.guru                         [1 × 7]
#> 2 fec.gov                           [1 × 7]
#> 3 googleapis.com:youtube            [1 × 7]

dj_tibblified |> 
  dplyr::rename(api_id = ".names") |> 
  tidyr::unnest(versions)

#> # A tibble: 3 × 8
#>   api_id         .names added updated swaggerUrl swaggerYamlUrl openapiVer link 
#>   <chr>          <chr>  <chr> <chr>   <chr>      <chr>          <chr>      <chr>
#> 1 apis.guru      2.2.0  2015… 2023-0… https://a… https://api.a… 3.0.0      http…
#> 2 fec.gov        1.0    2018… 2023-0… https://a… https://api.a… 3.0.0      http…
#> 3 googleapis.co… v3     2020… 2023-0… https://a… https://api.a… 3.0.0      http…

• At first tibblified looks similar to result of enframe().
• By default anything it has to name gets .names
  • We’ll see how to clean that up later.
• But the nested things are tibbles, so we can use tidyr::unnest()

Rectangling manually vs tibblify

dj_tidyr <- 
  demo_json |> 
  tibble::enframe(name = "api_id") |>
  tidyr::unnest_longer(
    value, indices_include = FALSE
  ) |> 
  tidyr::unnest_longer(
    value, indices_to = "version"
  ) |>
  tidyr::unnest_wider(value)

dj_tibblify <- 
  demo_json |>
  tibblify::tibblify() |> 
  dplyr::rename(api_id = ".names") |> 
  tidyr::unnest(versions) |> 
  dplyr::rename(version = ".names")

waldo::compare(
  dj_tibblify, dj_tidyr, 
  list_as_map = TRUE # Ignore column order
)

#> ✔ No differences

• {waldo} is a package for finding differences.
• list_as_map = TRUE means “ignore column order” here.
• What did tibblify do to figure out how to rectangle the data?

tspec_guess()

• tibblify::tibblify() spec argument
  • “What should this look like?”
• Guessed by default with tibblify::guess_tspec()

tibblify::guess_tspec(demo_json)

#> tspec_df(
#>   .names_to = ".names",
#>   tib_df(
#>     "versions",
#>     .names_to = ".names",
#>     tib_chr("added"),
#>     tib_chr("updated"),
#>     tib_chr("swaggerUrl"),
#>     tib_chr("swaggerYamlUrl"),
#>     tib_chr("openapiVer"),
#>     tib_chr("link"),
#>   ),
#> )

• Point out .names_to.
• Point out date columns (“added” and “updated”).
• It would be nice if we had a way to tell it what to expect.

The OpenAPI Specification

Multiple Standards

• Swagger 2.0 ➡️ OpenAPI 2.0
• OpenAPI 3.x
• OpenAPI 4.0 (in development)
• Postman Collection
• API Blueprint
• Web Application Description Language (WADL)

• Swagger was created by wordnik.com, free online dictionary that compiles a bunch of sources (and wanted a way to store API info consistently).
• Swagger specification & tools bought by SmartBear, spec donated to OpenAPI Initiative to make it officially open forever.
• Swagger now technically means the tools, OpenAPI is the spec.
  • But people talk about “swagger specs” all the time.
• We’ll focus on OpenAPI 3
• We might talk about tools to convert others to OpenAPI 3.

YAML

JSON
(jsonlite::read_json())

{
  "info": {
    "contact": {
      "email": "mike.ralphson@gmail.com",
      "name": "APIs.guru",
      "url": "https://APIs.guru"
    },
    "title": "APIs.guru",
    "version": "2.2.0",
    "x-apisguru-categories": [
      "open_data",
      "developer_tools"
    ]
  },
    "security": []
}

YAML
(yaml::read_yaml())

info:
  contact:
    email: mike.ralphson@gmail.com
    name: APIs.guru
    url: https://APIs.guru
  title: APIs.guru
  version: 2.2.0
  x-apisguru-categories:
    - open_data
    - developer_tools
security: [] # No security needed

• Originally “Yet Another Markup Language”
  • Meaning of name changed because they wanted to stress it’s for data, not documents per se.
  • “YAML Ain’t Markup Language”
• YAML adds comments, readability
  • Quoting optional for strings
  • Indentation for children, rather than brackets
• All JSON strings are valid YAML, not all YAML is valid JSON.
• .yaml or .yml
• You might have seen it:
  • RMarkdown/Quarto headers & config
  • Package configuration files (_pkgdown.yml, codecov.yml)
  • GitHub Actions

Exploring an API Description

• APIs.guru

openapi: 3.0.0 # Specification version number
info: # title, version, description, contact, license
servers: # One or more URLs + optional descriptions
tags: # Optional name, description, externalDocs of endpoint categories
externalDocs: # Optional url & description of additional documentation
security: # Optional list of named default security schemes
paths: # Endpoints of the API
webhooks: # Description of endpoints YOU can specify for API to SEND to
jsonSchemaDialect: # URI to extend components/schemas
x-whatever: # Extend with additional properties
components: # Reusable schemas, securitySchemes, reusable pieces of everything above

• Technically the OpenAPI specification is the standard, and each API has a “description” that’s defined in a “document” (which can be a composite of multiple files).
• People can and will do things wrong/weird.
• openapi = Spec version number, not API
• Info, servers, tags, externalDocs = Info about the API.
• security = default schemes to use, schemes are described in components/securitySchemes
• paths = the things the API can do, detailed in later chapter(s)
• webhooks, jsonSchemaDialect, extensions = we won’t talk about these much if at all
• Components is for storing reusable pieces of info about the API.

apis.guru APIs Schema

components:
  schemas:
    APIs:
      additionalProperties:
        $ref: "#/components/schemas/API"
      description: |
        List of API details.
        It is a JSON object with API IDs(`<provider>[:<service>]`) as keys.
      minProperties: 1
      type: object

tspec_apis <- tspec_df(
  .names_to = "api_id",
  tspec_api
)

• The thing we’re loading is described in APIs schema
• I’ve simplified this slightly from the full openapi object.
• JSON doesn’t require a name for the names, so we need to pick something (“api_id”)
• additionalProperties –> tspec_, properties –> tib_

apis.guru API Schema

components:
  schemas:
    API:
      additionalProperties: false
      description: Meta information about API
      properties:
        versions:
          additionalProperties:
            $ref: "#/components/schemas/ApiVersion"
          description: List of supported versions of the API
          minProperties: 1
          type: object
      required:
        - versions
      type: object

tspec_api <- tspec_row(
  tib_df(
    "versions", 
    .names_to = "version", 
    tspec_api_version
  )
)

• additionalProperties: false means tspec_row()
• versions property needs a tib_ function.
• versions is an object, so tib_df (or tib_row)
• versions has additionalProperties, so defined in tspec_df().

apis.guru ApiVersion Schema

components:
  schemas:
    ApiVersion:
      additionalProperties: false
      properties:
        added:
          description: Timestamp when the version was added
          format: date-time
          type: string
        link:
          description: Link to the individual API entry for this API
          format: url
          type: string
        openapiVer:
          description: The value of the `openapi` or `swagger` property of the source definition
          type: string
        swaggerUrl:
          description: URL to OpenAPI definition in JSON format
          format: url
          type: string
        swaggerYamlUrl:
          description: URL to OpenAPI definition in YAML format
          format: url
          type: string
        updated:
          description: Timestamp when the version was updated
          format: date-time
          type: string
      required:
        - added
        - updated
        - swaggerUrl
        - swaggerYamlUrl
        - openapiVer
      type: object

apis.guru ApiVersion tspec

tib_chr_datetime <- function(key, ..., required = TRUE) {
  tibblify::tib_scalar(
    key = key,
    ptype = vctrs::new_datetime(tzone = "UTC"),
    required = required,
    ptype_inner = character(),
    transform = \(x) as.POSIXct(x, format = "%Y-%m-%dT%H:%M:%OSZ", tz = "UTC"),
    ...
  )
}
tspec_api_version <- tspec_row(
  tib_chr_datetime("added"),
  tib_chr_datetime("updated"),
  tib_chr("openapiVer"),
  tib_chr("swaggerUrl"),
  tib_chr("swaggerYamlUrl"),
  tib_chr("link", required = FALSE)
)

Using tspecs: version

tibblify(demo_json$apis.guru$versions$`2.2.0`, tspec_api_version)

#> # A tibble: 1 × 6
#>   added               updated             openapiVer swaggerUrl   swaggerYamlUrl
#>   <dttm>              <dttm>              <chr>      <chr>        <chr>         
#> 1 2015-11-26 17:52:26 2023-04-05 13:10:14 3.0.0      https://api… https://api.a…
#> # ℹ 1 more variable: link <chr>

Using tspecs: api

tibblify(demo_json$apis.guru, tspec_api) |> tidyr::unnest(versions)

#> # A tibble: 1 × 7
#>   version added               updated             openapiVer swaggerUrl         
#>   <chr>   <dttm>              <dttm>              <chr>      <chr>              
#> 1 2.2.0   2015-11-26 17:52:26 2023-04-05 13:10:14 3.0.0      https://api.apis.g…
#> # ℹ 2 more variables: swaggerYamlUrl <chr>, link <chr>

Using tspecs: apis

tibblify(demo_json, tspec_apis) |> tidyr::unnest(versions)

#> # A tibble: 3 × 8
#>   api_id   version added               updated             openapiVer swaggerUrl
#>   <chr>    <chr>   <dttm>              <dttm>              <chr>      <chr>     
#> 1 apis.gu… 2.2.0   2015-11-26 17:52:26 2023-04-05 13:10:14 3.0.0      https://a…
#> 2 fec.gov  1.0     2018-11-20 00:04:28 2023-03-06 07:12:59 3.0.0      https://a…
#> 3 googlea… v3      2020-11-02 10:32:34 2023-04-21 23:09:23 3.0.0      https://a…
#> # ℹ 2 more variables: swaggerYamlUrl <chr>, link <chr>