Skip to contents

Testing EJAM App with shinytest2

Source: vignettes/dev-run-shinytests.Rmd
dev-run-shinytests.Rmd

This document covers the UI-related automated tests.

NOTE on work in progress / how to run the webapp tests

This package can use shinytest2::test_app().

EJAM:::test_ejam() instead uses an approach that first sources testthat/setup.R which does setup needed and also sources testthat/setup-shinytest2.R which defines shinytest2_webapp_functionality(), a function that contains the steps tested in the web app, and then directly uses that function from within test files like /tests/testthat/test-webapp-latlon-functionality.R etc.

Dev Environment

If you are successfully running the app, you should have all the necessary packages, or at least those in Imports or Suggests of the DESCRIPTION file. Some dev-related packages and tools to note:

  • shinytest2 - shinytest2 is the key R package helping to test shiny web app functionality

  • Pandoc and knitr – The Pandoc software comes bundled with RStudio, and is a “swiss-army knife” for document conversion. There also is a function called [knitr::pandoc()] that is a wrapper that calls Pandoc to convert docs to HTML, PDF, etc.

  • PhantomJS maybe – This used to be needed for downloads (and was installed with webshot::install_phantomjs()). But note this newer info:

NOTE FROM https://rstudio.github.io/shinytest2/articles/z-migration.html: - shinytest2 is the successor to shinytest. shinytest was implemented using webdriver which uses PhantomJS. PhantomJS has been unsupported since 2017 and does not support displaying bslib’s Bootstrap v5. shinytest2 uses chromote to connect to your locally installed Chrome or Chromium application, allowing shinytest2 to display bslib’s Bootstrap v5.

How shinytest2 Works

shinytest2 (here referred to henceforth as “shinytest” not to be confused with the older R package that was named shinytest!) automates shiny web app functionality testing, so we can determine if code updates break or unexpectedly modify parts of an application.

It runs the app in a headless Chrome or Chromium browser and simulates user interactions. The EJAM shinytest2 tests no longer compare full saved snapshots of app state, generated HTML, maps, tables, or downloaded files. Instead, they assert stable, intentional facts about app behavior. This avoids brittle failures from minor changes in generated HTML, dates, package versions, table rendering details, or map markup.

Key Features:

  • Runs scripted app interactions for each supported web app test category
  • Checks that the expected site-selection path is used for each category
  • Checks uploaded filenames or FIPS picker selections where relevant
  • Checks that analysis completion is exported by the app
  • Checks selected inputs such as radius, title, plot controls, and picker values
  • Checks downloaded reports and spreadsheets by file type, approximate size, and basic structure
  • Checks that key tables, plots, and report outputs render in the latlon full-path test
  • Avoids checking exact full HTML reports, map markup, complete table contents, exact dates, or package version text

EJAM’s shinytest2 Folder Structure 


R/test_ejam.R

tests/
  └── testthat/
      ├── setup.R
      ├── setup-shinytest2.R
      └── test-webapp-[DATA TYPE]-functionality.R (e.g. test-webapp-FIPS-functionality.R)

File Descriptions

  • testthat/setup.R – Does setup for the test environment (loads global_*.R and app scripts into the testing environment, etc.). This file is auto-sourced by testthat.

  • setup-shinytest2.R – Is also auto-sourced by testthat because its filename starts with setup-, and it contains the source code for shinytest2_webapp_functionality(), which does a series of tests of web app UI interactions using the app to upload or select, and analyze, multiple data types (FIPS, shapefile, latlon, NAICS, etc.).

  • testthat/test-webapp-[DATA TYPE]-functionality.R – Each of these calls shinytest2_webapp_functionality(), specifying the data type to test with, after that helper has been made available by testthat’s automatic sourcing of the setup files.

Updating Tests

You may wish to modify the shinytest scripts, either to add new interactions with the application or to modify existing ones, such as in the case of an app component that can no longer be interacted with. Here are some methods and tips for updating the shinytest script accordingly.

Direct Updates

Modify source code of shinytest2_webapp_functionality() to add new interactions with the app for the shinytest to test. You would update that file by coding new UI interactions directly.

Using shinytest2::record_test() to generate testing code

If you’re not sure how to code new UI interactions directly, run shinytest2::record_test() to test the app interactively and record your actions, which can then be copied into test scripts.

Using shiny::exportTestValues(name = value)

Throughout the app code, shiny::exportTestValues() can be used to expose values from reactive expressions or other items that are not inputs or outputs. The shinytest2 code can then read those exported values with app$get_values(export = TRUE) or wait for them with app$wait_for_value(export = "name"). EJAM uses this pattern for stable checks such as analysis_complete and multisite_report_download_ready. See details in shinytest2_webapp_functionality() as defined in the /tests/testthat/ folder.

Running Tests Locally

One way to run the shinytests is via the GitHub Actions.

Another way is this:

x = EJAM:::test_ejam(ask=F, run_these="webapp")

You can also run a test directly in an interactive R session. This is useful for debugging one category at a time.

remotes::install_local() # once
library(EJAM) # once
source(testthat::test_path("setup.R")) # once. gets done automatically though, by things like testthat::test_file()

# run a single test:
shinytest2_webapp_functionality("latlon")

It is recommended during development to use remotes::install_local() to ensure your development code is the one tested. This is because shinytest2 automatically references the installed version of a package.

Another useful way was this (but this might be deprecated by shinytest2)

# first, source `setup.R`, from the tests/testthat/ folder.
source(testthat::test_path("setup.R"))

# then for one subset of tests, like just the latlon analysis features:
shinytest2::test_app(".", filter="latlon-functionality", check_setup = FALSE)

# for all the webapp functionality tests
shinytest2::test_app(".", filter="-functionality", check_setup = FALSE)

GitHub Actions Integration

Using GitHub Actions (GHA) we can have GitHub run our shiny web app UI tests prior to merging a Pull Request, to give us peace of mind that the app will still work with the merged code.

Workflow

  • The GHA sets up R, installs dependencies, and runs scripted shinytest2 tests.
  • The workflow is stored in .github/workflows/test-webapp-functionality.yaml.
  • PRs to a specified branch such as development, main can trigger GHA workflows (as specified in the workflow yml file).

Speed Optimization

  • If GHA takes too long, cache dependencies by temporarily disabling steps after setup.
  • Keep slow checks centralized. For example, the latlon shinytest2 test runs the broad report download, spreadsheet download, details table, and plot checks; other categories should focus on category-specific selection and analysis checks.

Updating Expected Behavior Checks

Do not use testthat::snapshot_accept() for these shinytest2 tests. If a test fails, inspect whether the app behavior changed in a meaningful way. If the behavior is still correct, update the explicit assertion in setup-shinytest2.R so it checks a stable fact instead of exact generated output.

Examples of stable checks include:

  • an uploaded file input contains the expected filename
  • a FIPS picker contains the expected selected FIPS code
  • analysis_complete is TRUE
  • a downloaded report exists, has an .html extension, is above a minimum size, and contains a few stable markers
  • a downloaded spreadsheet exists, has an .xlsx extension, has the expected ZIP signature, and contains required sheet names
  • a plot output has rendered an image with nonzero dimensions

Debugging Tests & GitHub Actions

Debugging shinytest2

  • Use save_log() to inspect logs.
  • Add print(), message(), or warning() statements in shinytest2_webapp_functionality().
  • Run, line-by line or in chunks, the main shinytest code in shinytest2_webapp_functionality()

Then, after running lines or chunks, run app$get_log() to view the log.

Debugging GHA

Generally, if the shinytest2 tests pass locally using the same branch and dependencies, GHA should pass. However, the tests can still fail due to OS differences, browser availability, R version differences, package differences, or timeouts. Here are some tips for debugging these issues:

  • Inspect the log in the GitHub repo, under the Actions tab or the Checks tab of the PR.
  • Inspect artifacts or logs after a failed run to identify whether the failure was an app behavior regression, browser startup problem, timeout, missing dependency, or assertion that is too brittle.

Current State of Tests

  • If the shinytests are failing, do not accept new snapshots. Inspect the failing assertion and decide whether app behavior regressed or the test should check a more stable fact.
  • The versions of R and shiny and shinytest2 used for testing also should be noted as potentially affecting tests.

Resources

RStudio shinytest2 Documentation