RODBC::sqlQuery detecting hidden characters

Question

I am struggling with hidden characters when querying an Oracle database using RODBC. First my code:

library(RODBC)
con <- odbcConect('dsn', uid = 'user', pwd = 'pass')
query <- read_file('Query.sql')
query <- gsub('\n', ' ',gsub('\t', ' ',gsub('\r' ,' ', query)))

I'm using gsub here to manually remove the three hidden characters I had identified in my sql file.

df <- sqlQuery(con, query = query)

This is returning a list of two errors.

[1] "HY000 911 [Oracle][ODBC][Ora]ORA-00911: invalid character\n" 

and

[2] "[RODBC] ERROR: Could not SQLExecDirect...

Initially I was copy and pasting this query from outlook into a text file. Then I retyped the entire thing in hopes that would get rid of the hidden characters. Now I'm using a string of gsubs to manually remove the hidden characters and I'm still receiving the error. Looking through the 'query' vector I don't see any hidden characters so I'm not sure where the issue is coming from.

I've read that RODBC can struggle with aggregating in SQL queries but this query only uses LEFT JOIN, CASE and WHERE for higher level keywords.

Any help is appreciated.

Answer 1

Perhaps convert everything to ASCII, using base:iconv() .

If the query itself has bad characters, I would start by hand with a simple SELECT query with one column; start with a fresh text file where you know the encoding is well-behaved. Make sure the query works in Oracle Developer (or something that doesn't depend on R). Then make sure the the RODBC connection works with that simple query.

Assuming you must use sql files that you didn't write from scratch, you probably want to work your way in this direction.

content   <- readr::read_file("Query.sql")
cleaned   <- base::iconv(x=content, from="latin1", to="ASCII//TRANSLIT", sub="&")

if( grepl("&", cleaned) ) {
  cat(cleaned)
  stop("The query might contain non-ASCII characters with no good non-ASCII equivalent.  Check the console for the '&' substitution character.")
}

con <- odbcConect('dsn', uid = 'user', pwd = 'pass')
df  <- sqlQuery(con, query = returned_value)

I'm throwing an error above when there's likely a bad character, since it might not be clear what should replace it.

Even if that's not your ultimate desired solution, collapsing everything to ASCII could help confirm your suspicion. Also, consider using Notepad++ or Atom to show nonprinting characters, like in https://stackoverflow.com/a/8523118/1082435 .