RODBC loses time values of datetime when result set is large

Question

So this is VERY strange. RODBC seems to drop the time portion of DateTime SQL columns if the result set is large enough. (The queries are running against an SQL S

Answer 1

sqlQuery() has an option called as.is. Setting this to TRUE will pull everything as seen in for example Microsoft SQL Management Studio.

Answer 2

I had the same issue and concluded that it is due to DST:

This fails: as.POSIXct(c("2015-03-29 01:59:22", "2015-03-29 02:00:04"))

This works: as.POSIXct(c("2015-03-29 01:59:22", "2015-03-29 02:00:04"), tz="UTC")

I could not find how to force tz="UTC" in default RODBC behavior, however using as.is = TRUE and converting columns myself does the job.

Note: At first I had the impression that it was due to huge result, but in fact it was due to the fact that in huge results I have more chances to cross DST updates.

Answer 3

sqlQuery(ch, getSQL(sqlquerypath)) 

stripped the times off my datetime column.

sqlQuery(ch, getSQL(sqlquerypath), as.is = TRUE) 

fixed the issue.

Answer 4

I think this is a case of times being stripped from the dates where the date range includes shifts to/from daylight savings time. If you are selecting periods that don't include daylight savings shifts, the times will be retained (e.g., from 1/1/2007 to 3/1/2007. This could possibly be avoided by changing the system time on your computer to follow a time zone (e.g., Arizona) where there are no daylight savings shifts (sounds bizarre, but it has worked for me).

To overcome this issue, import the DateTimes as characters (using "as.is") and then convert them to POSIXct. You could alternatively use "strptime" which converts to POSIXlt and allows you to specify the format. Here's an example of a SQL query where DateTimes are imported as.is (TRUE), but asociated DataValues are not (FALSE) and then the DateTime is converted to an R date format:

data <- sqlQuery(channel, paste("SELECT LocalDateTime, DataValue FROM DataValues WHERE  LocalDateTime >= '1/1/2007 0:00' AND LocalDateTime < '1/1/2008 0:00' ORDER BY LocalDateTime ASC"),as.is=c(TRUE,FALSE))

data$LocalDateTime <- as.POSIXct(totalP$LocalDateTime,tz="MST")

Answer 5

I cope with the same problem as well. Even stranger, on a large dataset one column would import both date and time the other column only imported the date.

My advise would be to split the data/time in SQL

 myconn <- odbcConnect(dsnName, uid, pwd)
 results <- sqlQuery(myconn, "SELECT TOP 100 MyID, format(MyDateTimeColumn,"HH:mm:ss") as MyTimeColumn,format(MyDateTimeColumn,"yyyy-MM-dd") as MyDateColumn  from MyTable ORDER BY MyDateTimeColumn DESC")
 close(myconn)

Then combine them in R afterwards. Hope it helps.

Answer 6

It may be a daylight saving issue. If there is a time that doesnt exist in your timezone (because of daylight saving) it may cause something like this.