Remove duplicates in string

Question

I have the following data set

df <- data.frame(
    path = c(\"a,b,a\", 
        \"(direct) / (none),   (direct) / (none), google / cpc,    google / cpc\"         


        

Answer 1

Basic logic behind below code :

i)split each row on "," , (ii) remove whitespace (iii) take unique values
(iv) collapse back on "," and paste

t = apply(df, 1, function(x) paste0(unique(trimws(unlist(strsplit(x,",")))), collapse = ","))
df=data.frame(t)
# df
#                               t
#1                            a,b
#2 (direct) / (none),google / cpc
#3                            f,d
#4                            a,c

Answer 2

You were almost there. The only thing is that you need to split with ",\\s*" instead of just ",". In the latter case, calling unique won't produce the wanted output, since some string may differ for the number of blank spaces. If you remove them when you split, you solve this issue.

On another note, since you used setDT(df), I guess you are using data.table. If so, you need to use proper data.table grammar to avoid copies:

df[,path:=sapply(
   strsplit(as.character(df$path ), split=",\\s*"), 
    function(x) {paste(unique(x), collapse = ', ')})]

will modify the path column by reference.

Answer 3

It looks like your problem is the initial white space in the second strings. Are you trying to preserve that, or are you willing to lose it? If you're willing to lose it, then

df$path <- sapply(strsplit(as.character(df$path), split=","), function(x) {
    paste(unique(trimws(x)), collapse = ', ') } )

is what you want:

> df$path <- sapply(strsplit(as.character(df$path), split=","), function(x) {
+     paste(unique(trimws(x)), collapse = ', ') } )
> df$path
[1] "a, b"                            "(direct) / (none), google / cpc"
[3] "f, d"                            "a, c"
>