Explain with example: how embedding layers in keras works

Question

I don\'t understand the Embedding layer of Keras. Although there are lots of articles explaining it, I am still confused. For example, the code below isfrom imdb sentiment analy

Answer 1

I agree with the previous detailed answer, but I would like to try and give a more intuitive explanation.

To understand how Embedding layer works, it is better to just take a step back and understand why we need Embedding in the first place.

Usually ML models take vectors (array of numbers) as input and, when dealing with text, we convert the strings into numbers. One of the easiest way to do this is one-hot encoding where, you treat each strings as categorical variable. But the first issue is that if you use a dictionary (vocabulary) of 10000 words, then one-hot encoding is pretty much waste of space (memory).

Also as discrete entities are mapped to either 0 or 1 signaling a specific category, one-hot encoding cannot capture any relation between words. Thus if you're familiar with IMDB movie data-set, one-hot encoding is nothing but useless for sentiment analysis. Because, if you measure the similarity using the cosine distance, then similarity is always zero for every comparison between different indices.

This should guide us to find a method where --

• Similar words can have a similar encoding,
• To represent the categorical variables, we will have fewer numbers than the number of unique categories.

Enters Embedding..

Embedding is a dense vector of floating point values and, these numbers are generated randomly and during training these values are updated via backprop just as the weights in a dense layer get updated during training.
As defined in TensorFlow docs

The Embedding layer can be understood as a lookup table that maps from integer indices (which stand for specific words) to dense vectors (their embeddings).

Before building the model with sequential you have already used Keras Tokenizer API and input data is already integer coded. Now once you mention the number of embedding dimensions (e.g. 16, 32, 64, etc.), the number of columns of the lookup table will be determined by that.

Output of the embedding layer is always a 2D array, that's why it is usually flattened before connecting to a dense layer. In the previous answer also, you can see a 2D array of weights for the 0th layer and the number of columns = embedding vector length.

That's how I think of Embedding layer in Keras. Hopefully this shed little more light and I thought this could be a good accompaniment of the answer posted by @Vaasha.

Reference: TensorFlow Word Embedding Tutorial.