Convert Double to Date using Spark in R

Question

I have an R data frame as below

Date @AD.CC_CC @AD.CC_CC.1 @CL.CC_CC @CL.CC_CC.1
2018-02-05      -380        -380            


        

Answer 1

This looks like a sparklyr bug. The simplest workaround is to cast dates to character, before calling copy_to:

df <- tibble::tibble(Date=as.Date(c("2018-02-05", "2018-02-06")))
sdf <- df %>% mutate(Date = as.character(Date)) %>% copy_to(sc, .)

sdf

# Source:   table<sparklyr_11ae23aa677e> [?? x 1]
# Database: spark_connection
  Date      
  <chr>     
1 2018-02-05
2 2018-02-06

and casting it later:

sdf %>% mutate(Date = to_date(Date))

# Source:   lazy query [?? x 1]
# Database: spark_connection
  Date      
  <date>    
1 2018-02-05
2 2018-02-06

You can also try using the numeric value as an offset since beginning of the Unix epoch:

sdf <- df  %>% copy_to(sc, .)

sdf

# Source:   table<sparklyr_13ab19ec6f53> [?? x 1]
# Database: spark_connection
   Date
  <dbl>
1 17567
2 17568

sdf %>% mutate(Date = date_add(to_date("1970-01-01"), Date))

# Source:   lazy query [?? x 1]
# Database: spark_connection
  Date      
 <date>    
1 2018-02-05
2 2018-02-06

Alternatively, you can skip copy_to completely (it has very limited applications anyway, and is seldom useful in production) and use one of built-in input formats (spark_read_*).