Developer Notes

This page contains documentation relevant for those wishing to contribute to the package and specific instructions for how to add support for a new geocoding service.

Introduction

The two core functions to focus on in the package are geo() and reverse_geo(). These functions have very similar layouts, but geo() is for forward geocoding while reverse_geo() is for reverse geocoding. The geocode() and reverse_geocode() functions only extract input data from a dataframe and pass it to the geo() and reverse_geo() functions respectively for geocoding.

Both the geo() and reverse_geo() functions take inputs (either addresses or coordinates) and call other functions as needed to deduplicate the inputs, pause to comply with API usage rate policies, and execute queries. Key parameters and settings for geocoding are stored for easy access and display in built-in datasets.

Consider this query:

library(dplyr)
library(tidygeocoder)

df <- tibble(
  id = c(1, 2, 1),
  locations = c('tokyo', 'madrid', 'tokyo')
  )

df %>%
  geocode(address = locations, method = 'osm', full_results = TRUE, verbose = TRUE)
#> 
#> Number of Unique Addresses: 2
#> Passing 2 addresses to the Nominatim single address geocoder
#> 
#> Number of Unique Addresses: 1
#> Querying API URL: https://nominatim.openstreetmap.org/search
#> Passing the following parameters to the API:
#> q : "tokyo"
#> limit : "1"
#> format : "json"
#> HTTP Status Code: 200
#> Query completed in: 0.5 seconds
#> Total query time (including sleep): 1 seconds
#> 
#> 
#> Number of Unique Addresses: 1
#> Querying API URL: https://nominatim.openstreetmap.org/search
#> Passing the following parameters to the API:
#> q : "madrid"
#> limit : "1"
#> format : "json"
#> HTTP Status Code: 200
#> Query completed in: 0.2 seconds
#> Total query time (including sleep): 1 seconds
#> 
#> Query completed in: 2 seconds
#> # A tibble: 3 × 14
#>      id locations   lat   long  place_id licence  osm_t…¹ osm_id bound…² displ…³
#>   <dbl> <chr>     <dbl>  <dbl>     <int> <chr>    <chr>    <int> <list>  <chr>  
#> 1     1 tokyo      35.7 140.   298316729 Data © … relati… 1.54e6 <chr>   東京都…
#> 2     2 madrid     40.4  -3.70 298773150 Data © … relati… 5.33e6 <chr>   Madrid…
#> 3     1 tokyo      35.7 140.   298316729 Data © … relati… 1.54e6 <chr>   東京都…
#> # … with 4 more variables: class <chr>, type <chr>, importance <dbl>,
#> #   icon <chr>, and abbreviated variable names ¹​osm_type, ²​boundingbox,
#> #   ³​display_name

Here is what is going on behind the scenes:

• The geocode() function extracts the address data from the input dataframe and passes it to the geo() function.
• The geo() function looks for unique inputs and prepares them for geocoding. In this case, there is one duplicate input so we only have two unique inputs.
• The geo() function must figure out whether to use single address geocoding (1 address per query) or batch geocoding (multiple addresses per query). In this case the specified Nominatim (“osm”) geocoding service does not have a batch geocoding function so single address geocoding is used.
• Because single address geocoding is used, the geo() function is called once for each input to geocode all addresses (twice in this case) and the results are combined. If batch geocoding was used then the appropriate batch geocoding function would be called based on the geocoding service specified.
• Because the specified geocoding service has a usage limit, the rate of querying is limited accordingly. By default this is based on the min_time_reference dataset. This behavior can be modified with the min_time argument.
• Since the input data was deduplicated, the results must be aligned to the original inputs (which contained duplicates) so that the original data structure is preserved. Alternatively, if you only want to return unique results, you can specify unique_only = TRUE.
• This combined data is returned by geo() to the geocode() function. The geocode() function then combines the returned data with the original dataset.

Refer to the notes below on adding a geocoding service for more specific documentation on the code structure.

Adding a New Geocoding Service

This section documents how to add support for a new geocoding service to the package. Required changes are organized by file. If anything isn’t clear, feel free to file an issue.

Base all changes on the main branch.

Files to Update

• R/api_url.R
  • Add a standalone function for obtaining the API URL and update the get_api_url() function accordingly. If arguments need to be added to the get_api_url() function, make sure to adjust the calls to this function in the geo() and reverse_geo() functions accordingly.
• data-raw/api_parameter_reference.R
  • Add rows to the api_parameter_reference dataset to include the geocoding service. Each service is referred to by a short name in the method column (which is how the service is specified in the geo() and geocode() functions). The generic_name column has the universal parameter name that is used across geocoding services (ie. “address”, “limit”, etc.) while the api_name column stores the parameter names that are specific to the geocoding service.
  • Note that there is no need to include parameters that are only used for reverse geocoding or parameters that have no equivalent in other geocoding services (ie. there is no generic_name) unless the parameters are required. Parameters can always be passed to services directly with the custom_query argument in geo() or reverse_geo().
• data-raw/api_references.R
  • Add a row to min_time_reference with the minimum time each query should take (in seconds) according to the geocoding service’s free tier usage restrictions.
  • Add a row to api_key_reference if the service requires an API key.
  • If the service you are adding has batch geocoding capabilities, add the maximum batch size (as a row) to batch_limit_reference.
  • Add a row to api_info_reference with links to the service’s website, documentation, and usage policy.
• R/geo.R
  • If the service supports batch geocoding then add a new function in R/batch_geocoding.R and add it to the batch_func_map named list.
• R/reverse_geo.R
  • Update the get_coord_parameters() function based on how the service passed latitude and longitude coordinates for reverse geocoding.
  • If the service supports reverse batch geocoding then add a new function in R/reverse_batch_geocoding.R and add it to the reverse_batch_func_map named list.
• R/results_processing.R
  • Update the extract_results() function which is used for parsing single addresses (ie. not batch geocoding). You can see examples of how I’ve tested out parsing the results of geocoding services here.
  • In a similar fashion, update the extract_reverse_results() function for reverse geocoding.
  • Update the extract_errors_from_results() function to extract error messages for invalid queries.
• If applicable, add new tests to the scripts in the tests directory for the method. Note that tests should avoid making a HTTP query (ie. use no_query = TRUE in the geo() and geocode() functions).
• R/global_variables.R
  • If applicable, add your service to one of the global variables.

Other Files

These files don’t necessarily need to be updated. However, you might need to make changes to these files if the service you are implementing requires some non-standard workarounds.

• R/query_factory.R
  • Houses the functions used to create and execute API queries.
• R/documentation.R
  • Functions for producing rmarkdown package documentation.
• R/data.R
  • Documentation for in-built datasets.
• R/utils.R
  • Common utility functions.
• R/input_handling.R
  • Handles the deduplication of input data.
• external/create_logo.Rmd
  • Creates the package logo.
• vignettes/tidygeocoder.Rmd.orig
  • This file produces the vignette. See the knit command and comments at the top of this file.

Testing

• Test out the new service to make sure it behaves as expected. You can reference tests and example code in the ‘sandbox’ folder.
• Run devtools::check() to make sure the package still passes all tests and checks, but note that these tests are designed to work offline so they do not make queries to geocoding services.
• As a final check, run external/online_tests.R to test making queries to the geocoding services. These tests are not included in the internal package tests (devtools::test()) because they require API keys which would not exist on all systems and are dependent on the geocoding services being online at that the time of the test.
• Run the commands detailed in cran-comments.md to test the package on other environments. Note that these tests should also be included in the automated GitHub actions tests for pull requests.

Releasing a New Version

To release a new version of tidygeocoder:

• Run the tests detailed above.
• Update the package version in DESCRIPTION.
• Double check the package version that appears on the citation page. It is automatically set to the latest official release in inst/CITATION.
• Run the precomputed vignette to update the output. Inspect the vignette output for any problems (ie. are there missing results?). It can be helpful to build the site and view the HTML output for this.
• Reserve a DOI for a new package version with Zenodo.
• Update the Zenodo DOI in README.Rmd and reknit to update README.md.
• Rebuild the site: pkgdown::build_site() (make sure to do this again if you need to make any more updates that show up on the site)
• Check URLs: urlchecker::url_check()
• Check spelling: devtools::spell_check()

Last, run devtools::release() to release the new version once everything looks good.