How to initialize second glove model with solution from first?

一世执手 提交于 2020-05-30 03:38:38

问题


I am trying to implement one of the solutions to the question about How to align two GloVe models in text2vec?. I don't understand what are the proper values for input at GlobalVectors$new(..., init = list(w_i, w_j). How do I ensure the values for w_i and w_j are correct?

Here's a minimal reproducible example. First, prepare some corpora to compare, taken from the quanteda tutorial. I am using dfm_match(all_words) to try and ensure all words are present in each set, but this doesn't seem to have the desired effect.

library(quanteda)

# from https://quanteda.io/articles/pkgdown/replication/text2vec.html

# get a list of all words in all documents
all_words <-
  data_corpus_inaugural %>% 
  tokens(remove_punct = TRUE,
         remove_symbols = TRUE,
         remove_numbers = TRUE) %>% 
  types()

# should expect this mean features in each set
length(all_words)

# these are our three sets that we want to compare, we want to project the
# change in a few key words on a fixed background of other words
corpus_1 <- data_corpus_inaugural[1:19]
corpus_2 <- data_corpus_inaugural[20:39]
corpus_3 <- data_corpus_inaugural[40:58]

my_tokens1 <- texts(corpus_1) %>%
  char_tolower() %>%
  tokens(remove_punct = TRUE,
         remove_symbols = TRUE,
         remove_numbers = TRUE) 

my_tokens2 <- texts(corpus_2) %>%
  char_tolower() %>%
  tokens(remove_punct = TRUE,
         remove_symbols = TRUE,
         remove_numbers = TRUE) 

my_tokens3 <- texts(corpus_3) %>%
  char_tolower() %>%
  tokens(remove_punct = TRUE,
         remove_symbols = TRUE,
         remove_numbers = TRUE) 

my_feats1 <- 
  dfm(my_tokens1, verbose = TRUE) %>%
  dfm_trim(min_termfreq = 5) %>% 
  dfm_match(all_words) %>% 
  featnames()

my_feats2 <- 
  dfm(my_tokens2, verbose = TRUE) %>%
  dfm_trim(min_termfreq = 5) %>%
  dfm_match(all_words) %>% 
  featnames()

my_feats3 <- 
  dfm(my_tokens3, verbose = TRUE) %>%
  dfm_trim(min_termfreq = 5) %>%
  dfm_match(all_words) %>% 
  featnames()

# leave the pads so that non-adjacent words will not become adjacent
my_toks1_2 <- tokens_select(my_tokens1, my_feats1, padding = TRUE)
my_toks2_2 <- tokens_select(my_tokens2, my_feats2, padding = TRUE)
my_toks3_2 <- tokens_select(my_tokens3, my_feats3, padding = TRUE)

# Construct the feature co-occurrence matrix
my_fcm1 <- fcm(my_toks1_2, context = "window", tri = TRUE)
my_fcm2 <- fcm(my_toks2_2, context = "window", tri = TRUE)
my_fcm3 <- fcm(my_toks3_2, context = "window", tri = TRUE)

Somewhere in the above steps I believe I need to ensure that the fcm for each set has all the words of all sets to get the matrix dimensions the same, but I'm not sure how to accomplish that.

Now fit the word embedding model for the first set:


library("text2vec")

glove1 <- GlobalVectors$new(rank = 50, 
                            x_max = 10)

my_main1 <- glove1$fit_transform(my_fcm1, 
                               n_iter = 10,
                               convergence_tol = 0.01, 
                               n_threads = 8)

my_context1 <- glove1$components
word_vectors1 <- my_main1 + t(my_context1)

And here is where I get stuck, I want to initialise the second model with the first, so that the coordinate system will be comparable between the first and second models. I read that w_i and w_j are main and context words, and b_i and b_j are biases. I've found output for those in my first model object, but I get an error:

glove2 <- GlobalVectors$new(rank = 50, 
                            x_max = 10,
                            init = list(w_i = glove1$.__enclos_env__$private$w_i, 
                                        b_i = glove1$.__enclos_env__$private$b_i, 
                                        w_j = glove1$.__enclos_env__$private$w_j, 
                                        b_j = glove1$.__enclos_env__$private$b_j))

my_main2 <- glove2$fit_transform(my_fcm2, 
                                 n_iter = 10,
                                 convergence_tol = 0.01, 
                                 n_threads = 8)

The error is Error in glove2$fit_transform(my_fcm2, n_iter = 10, convergence_tol = 0.01, : init values provided in the constructor don't match expected dimensions from the input matrix

Assuming I have identified w_i, etc., correctly in the first model, how can I get ensure they are the correct size?

Here's my session info:

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.15.2

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] text2vec_0.6   quanteda_2.0.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4            pillar_1.4.3          compiler_3.6.0        tools_3.6.0           stopwords_1.0        
 [6] digest_0.6.25         packrat_0.5.0         lifecycle_0.2.0       tibble_3.0.0          gtable_0.3.0         
[11] lattice_0.20-40       pkgconfig_2.0.3       rlang_0.4.5           Matrix_1.2-18         fastmatch_1.1-0      
[16] cli_2.0.2             rstudioapi_0.11       mlapi_0.1.0           parallel_3.6.0        RhpcBLASctl_0.20-17  
[21] dplyr_0.8.5           vctrs_0.2.4           grid_3.6.0            tidyselect_1.0.0.9000 glue_1.3.2           
[26] data.table_1.12.8     R6_2.4.1              fansi_0.4.1           lgr_0.3.4             ggplot2_3.3.0        
[31] purrr_0.3.3           magrittr_1.5          scales_1.1.0          ellipsis_0.3.0        assertthat_0.2.1     
[36] float_0.2-3           rsparse_0.4.0         colorspace_1.4-1      stringi_1.4.6         RcppParallel_5.0.0   
[41] munsell_0.5.0         crayon_1.3.4.9000 


回答1:


Here is a working example. See ?rsparse::GloVe documentation for details.

library(rsparse)
data("movielens100k")
x = crossprod(sign(movielens100k))

model = GloVe$new(rank = 10, x_max = 5)

w_i = model$fit_transform(x = x, n_iter = 5, n_threads = 1)
w_j = model$components
init = list(w_i = t(w_i), model$bias_i, w_j = w_j, b_j = model$bias_j)

model2 = GloVe$new(rank = 10, x_max = 10, init = init)
w_i2 = model2$fit_transform(x)


来源:https://stackoverflow.com/questions/61146392/how-to-initialize-second-glove-model-with-solution-from-first

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!