Change language encoding in existing Df (not on import)

问题

looking to "correct" the encoding of a DF with a mix of english and french. I'm not loading it from a .csv but from an API, so won't be able to change encoding on import.

df <- tibble(ID  = 1:4, 
             text = c("engish", "pour la mise en Å“uvre dâ€™une ville", "SÃ©curitÃ© de l'information - Ouverture des donnÃ©es", "Directeur GÃ©nÃ©ral"))

Encoding(df$text)

[1] "unknown" "latin1"  "latin1"  "latin1"

Using this function from the proustr package changes the encoding, but not the characters:


pattern_quote <- paste0(intToUtf8(8243),"|",intToUtf8(8246),"|",intToUtf8(171)," | ",intToUtf8(187),"|",
                        intToUtf8(8220),"|",intToUtf8(8221),"|",intToUtf8(96),"|",intToUtf8(180),"|",
                        intToUtf8(8222),"|",intToUtf8(8220))

pattern_apo <- paste0(intToUtf8(1370),"|",intToUtf8(65040),"|",intToUtf8(8217))

clean_punc <- function(vec){
  vec <- str_replace_all(vec, pattern = pattern_quote, 
                         replacement = '"')
  vec <- str_replace_all(vec, pattern = pattern_apo, 
                         replacement ="'")
  str_replace_all(vec, pattern = intToUtf8(8230), replacement = "\\.\\.\\.")
}

Encoding(clean_punc(df$text))

[1] "unknown" "UTF-8"   "UTF-8"   "UTF-8"

But the result is the same:

df %>% map_dfr(., clean_punc)

# A tibble: 4 x 2
  ID    text                                           
  <chr> <chr>                                          
1 1     engish                                         
2 2     "pour la mise en Å\"uvre dâ€™une ville"        
3 3     SÃ©curitÃ© de l'information - Ouverture des do~
4 4     Directeur GÃ©nÃ©ral

Maybe changing encoding isn't what I want, but rather need to str_replace with the various patterns?

回答1:

You can try iconv:

df <- tibble::tibble(ID  = 1:4, 
             text = c("engish", "pour la mise en Å“uvre dâ€™une ville", "SÃ©curitÃ© de l'information - Ouverture des donnÃ©es", "Directeur GÃ©nÃ©ral"))

df
#> # A tibble: 4 x 2
#>      ID text                                                
#>   <int> <chr>                                               
#> 1     1 engish                                              
#> 2     2 pour la mise en Å“uvre dâ€™une ville                
#> 3     3 SÃ©curitÃ© de l'information - Ouverture des donnÃ©es
#> 4     4 Directeur GÃ©nÃ©ral

df$text <- iconv(df$text, to="Windows-1252")
df
#> # A tibble: 4 x 2
#>      ID text                                             
#>   <int> <chr>                                            
#> 1     1 engish                                           
#> 2     2 pour la mise en œuvre d’une ville                
#> 3     3 Sécurité de l'information - Ouverture des données
#> 4     4 Directeur Général

^{Created on 2019-06-27 by the reprex package (v0.3.0)}

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.0 (2019-04-26)
#>  os       macOS Mojave 10.14.3        
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Paris                
#>  date     2019-06-27                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.6.0)
#>  backports     1.1.4   2019-04-10 [1] CRAN (R 3.6.0)
#>  callr         3.2.0   2019-03-15 [1] CRAN (R 3.6.0)
#>  cli           1.1.0   2019-03-19 [1] CRAN (R 3.6.0)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 3.6.0)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 3.6.0)
#>  devtools      2.0.2   2019-04-08 [1] CRAN (R 3.6.0)
#>  digest        0.6.19  2019-05-20 [1] CRAN (R 3.6.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 3.6.0)
#>  fansi         0.4.0   2018-10-05 [1] CRAN (R 3.6.0)
#>  fs            1.3.1   2019-05-06 [1] CRAN (R 3.6.0)
#>  glue          1.3.1   2019-03-12 [1] CRAN (R 3.6.0)
#>  highr         0.8     2019-03-20 [1] CRAN (R 3.6.0)
#>  htmltools     0.3.6   2017-04-28 [1] CRAN (R 3.6.0)
#>  knitr         1.23    2019-05-18 [1] CRAN (R 3.6.0)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 3.6.0)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 3.6.0)
#>  pillar        1.4.1   2019-05-28 [1] CRAN (R 3.6.0)
#>  pkgbuild      1.0.3   2019-03-20 [1] CRAN (R 3.6.0)
#>  pkgconfig     2.0.2   2018-08-16 [1] CRAN (R 3.6.0)
#>  pkgload       1.0.2   2018-10-29 [1] CRAN (R 3.6.0)
#>  prettyunits   1.0.2   2015-07-13 [1] CRAN (R 3.6.0)
#>  processx      3.3.1   2019-05-08 [1] CRAN (R 3.6.0)
#>  ps            1.3.0   2018-12-21 [1] CRAN (R 3.6.0)
#>  R6            2.4.0   2019-02-14 [1] CRAN (R 3.6.0)
#>  Rcpp          1.0.1   2019-03-17 [1] CRAN (R 3.6.0)
#>  remotes       2.0.4   2019-04-10 [1] CRAN (R 3.6.0)
#>  rlang         0.4.0   2019-06-25 [1] CRAN (R 3.6.0)
#>  rmarkdown     1.13    2019-05-22 [1] CRAN (R 3.6.0)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 3.6.0)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 3.6.0)
#>  stringi       1.4.3   2019-03-12 [1] CRAN (R 3.6.0)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 3.6.0)
#>  testthat      2.1.1   2019-04-23 [1] CRAN (R 3.6.0)
#>  tibble        2.1.3   2019-06-06 [1] CRAN (R 3.6.0)
#>  usethis       1.5.0   2019-04-07 [1] CRAN (R 3.6.0)
#>  utf8          1.1.4   2018-05-24 [1] CRAN (R 3.6.0)
#>  vctrs         0.1.0   2018-11-29 [1] CRAN (R 3.6.0)
#>  withr         2.1.2   2018-03-15 [1] CRAN (R 3.6.0)
#>  xfun          0.7     2019-05-14 [1] CRAN (R 3.6.0)
#>  yaml          2.2.0   2018-07-25 [1] CRAN (R 3.6.0)
#>  zeallot       0.1.0   2018-01-28 [1] CRAN (R 3.6.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

来源：https://stackoverflow.com/questions/56794274/change-language-encoding-in-existing-df-not-on-import

标签

stringr