reactiveFileReader in Shiny for .RData

Question

My current workflow in a shiny application is to run a R script as a cron job periodically to pull various tables from multiple databases as well as download data from some APIs. These are then saved as a .Rdata file in a folder called data .

In my global.R file I load the data by using load("data/workingdata.Rdata") . This results in all the dataframes (about 30) loading into the environment. I know I can use the reactiveFileReader() function to refresh the data, but obviously it would have to be used in the server.R file because of an associated session with the function. Also, I am not sure if load is accepted as a readFunc in reactiveFileReader() . What should be the best strategy for the scenario here?

Answer 1

A few thoughts on your workflow:

In the end with your RData-approach you are setting up another data source in parallel to your databases / APIs.

When working with files there always is some housekeeping-overhead (eg is your .RData file completed when reading it?). In my eyes this (partly) is what DBMS are made for – taking care about the housekeeping. Most of them have sophisticated solutions to ensure that you get what you query very fast; so why reinvent the wheel?

Instead of continuously creating your .RData files and polling data with the reactiveFileReader() function you could directly query the DB for changes using reactivePoll (see this for an example using sqlite). If your queries are long running (which I guess is the cause for your workflow) you can wrap them in a future and run them asynchronously (see this post to get some inspiration). Alternatively many DBMS provide something like materialized views to avoid long waiting times (according user privileges presumed).

Of course, all of this is based on assumptions, due to the fact, that your eco-system isn't known to me, but in my experience reducing interfaces means reducing sources of error.

Answer 2

This example uses a reactiveVal object with observe and invalidateLater . The data is loaded into a new environment and assigned to the reactiveVal every 2 seconds.

library(shiny)

ui <- fluidPage(
  actionButton("generate", "Click to generate an Rdata file"),
  tableOutput("table")
)

server <- shinyServer(function(input, output, session) {

  ## Use reactiveVal with observe/invalidateLater to load Rdata
  data <- reactiveVal(value = NULL)
  observe({
    invalidateLater(2000, session)
    n <- new.env()
    print("load data")
    env <- load("workingdata.Rdata", envir = n)
    data(n[[names(n)]])
  })


  ## Click the button to generate a new random data frame and write to file
  observeEvent(input$generate, {
    sample_dataframe <- iris[sample(1:nrow(iris), 10, F),]
    save(sample_dataframe, file="workingdata.Rdata")
    rm(sample_dataframe)
  })

  ## Table output
  output$table <- renderTable({
    req(data())
    data()
  })
})


shinyApp(ui = ui, server = server)

Answer 3

You could use load("data/workingdata.Rdata") at the top of server.R. Then, anytime anyone starts a new Shiny session, the data would be the most recent. The possible downsides are that:

• there could be a hiccup if the data is being written at the same time a new Shiny session is loading data.
• data will be stale if a session is open just before and then after new data is available.

I imagine the first possible problem wouldn't arise enough to be a problem. The second possible problem is more likely to occur, but unless you are in a super critical situation, I can't see it being a substantial enough problem to worry about.

Does that work for you?