Can R read html-encoded emoji characters?

Question

Question

My question, explained below, is:

How can R be used to read a string that includes HTML emoji codes like ��

Answer 1

tl;dr: the emoji aren't valid HTML entities; UTF-16 numbers have been used to build them instead of Unicode code points. I describe an algorithm at the bottom of the answer to convert them so that they are valid XML.

---

Identifying the Problem

R definitely handles emoji:

In fact, a few packages exist for handling emoji in R. For example, the emojifont and emo packages both let you retrieve emoji based on Slack-style keywords. It's just a question of getting your source characters through from the HTML-escaped format so that you can convert them.

xml2::read_xml seems to do fine with other HTML entities, like an ampersand or double quotes. I looked at this SO answer to see whether there were any XML-specific constraints on HTML entities, and it seemed like they were storing emoji fine. So I tried changing the emoji codes in your reprex to the ones in that answer:

body="Hug emoji: 😀😃"

And, sure enough, they were preserved (though they're obviously not the hug emoji anymore):

> test8 = read_html('Desktop/test.xml')
> test8 %>% xml_child() %>% xml_child() %>% xml_child() %>% xml_attr('body')
[1] "Hug emoji: \U0001f600\U0001f603"

I looked up the hug emoji on this page, and the decimal HTML entity given there is not ��. It looks like the UTF-16 decimal codes for the emoji have been wrapped in &# and ;.

In conclusion, I think the answer is that your emoji are, in fact, not valid HTML entities. If you can't control the source, you might need to do some pre-processing to account for these errors.

So, why does the browser convert them properly? I'm wondering if the browser is a little more flexible with these things and is making some guesses about what those codes could be. I'm just speculating, though.

---

Converting UTF-16 to Unicode code points

After some more investigation, it looks like valid emoji HTML entities use the Unicode code point (in decimal, if it's &#...;, or hex, if it's &#x...;). The Unicode code point is different from the UTF-8 or UTF-16 code. (That link explains a lot about how emoji and other characters are variously encoded, BTW! Good read.)

So we need to convert the UTF-16 codes used in your source data to Unicode code points. Referring to this Wikipedia article on UTF-16, I've verified how it's done. Each Unicode code point (our target) is a 20-bit number, or five hex digits. When going from Unicode to UTF-16, you split it up into two 10-bit numbers (the middle hex digit gets cut in half, with two of its bits going to each block), do some maths on them and get your result).

Going backwards, as you want to, it's done like this:

• Your decimal UTF-16 number (which is in two separate blocks for now) is 55358 56599
• Converting those blocks to hex (separately) gives 0x0d83e 0x0dd17
• You subtract 0xd800 from the first block and 0xdc00 from the second to give 0x3e 0x117
• Converting them to binary, padding them out to 10 bits and concatenating them, it's 0b0000 1111 1001 0001 0111
• Then we convert that back to hex, which is 0x0f917
• Finally, we add 0x10000, giving 0x1f917
• Therefore, our (hex) HTML entity is 🤗. Or, in decimal, &#129303

So, to preprocess this dataset, you'll need to extract the existing numbers, use the algorithm above, then put the result back in (with one &#...;, not two).

---

Displaying emoji in R

As far as I'm aware, there's no solution to printing emoji in the R console: they always come out as "U0001f600" (or what have you). However, the packages I described above can help you plot emoji in some circumstances (I'm hoping to expand ggflags to display arbitrary full-colour emoji at some point). They can also help you search for emoji to get their codes, but they can't get names given the codes AFAIK. But maybe you could try importing the emoji list from emojilib into R and doing a join with your data frame, if you've extracted the emoji codes into a column, to get the English names.

Answer 2

Translated Chad's JavaScript answer to Go since I too had the same issue, but needed a solution in Go.

https://play.golang.org/p/h9JBFzqcd90

package main

import (
    "fmt"
    "html"
    "regexp"
    "strconv"
    "strings"
)

func main() {
    emoji := "������������"

    regexp := regexp.MustCompile(`(&#\d+;){2}`)
    matches := regexp.FindAllString(emoji, -1)

    var builder strings.Builder

    for _, match := range matches {
        s := strings.Replace(match, "&#", "", -1)

        parts := strings.Split(s, ";")
        a := parts[0]
        b := parts[1]

        c, err := strconv.Atoi(a)
        if err != nil {
            panic(err)
        }

        d, err := strconv.Atoi(b)
        if err != nil {
            panic(err)
        }

        c = c - 0xd800
        d = d - 0xdc00

        e := strconv.FormatInt(int64(c), 2)
        f := strconv.FormatInt(int64(d), 2)

        g := "0000000000"[2:len(e)] + e
        h := "0000000000"[10:len(f)] + f

        j, err := strconv.ParseInt(g + h, 2, 64)
        if err != nil {
            panic(err)
        }

        k := j + 0x10000

        _, err = builder.WriteString("&#x" + strconv.FormatInt(k, 16) + ";")
        if err != nil {
            panic(err)
        }
    }

    fmt.Println(html.UnescapeString(emoji))
    emoji = html.UnescapeString(builder.String())
    fmt.Println(emoji)
}

Answer 3

JavaScript Solution

I had this exact same problem, but needed the solution in JavaScript, not R. Using rensa's comment above (hugely helpful!), I created the following code to solve this issue, and I just wanted to share it in case anyone else happens across this thread as I did, but needed it in JavaScript.

str.replace(/(&#\d+;){2}/g, function(match) {
    match = match.replace(/&#/g,'').split(';');
    var binFirst = (parseInt('0x' + parseInt(match[0]).toString(16)) - 0xd800).toString(2);
    var binSecond = (parseInt('0x' + parseInt(match[1]).toString(16)) - 0xdc00).toString(2);
    binFirst = '0000000000'.substr(binFirst.length) + binFirst;
    binSecond = '0000000000'.substr(binSecond.length) + binSecond;
    return '&#x' + (('0x' + (parseInt(binFirst + binSecond, 2).toString(16))) - (-0x10000)).toString(16) + ';';
});

---

And, here's a full snippet of it working if you'd like to run it:

var str = '������������'

str = str.replace(/(&#\d+;){2}/g, function(match) {
	match = match.replace(/&#/g,'').split(';');
	var binFirst = (parseInt('0x' + parseInt(match[0]).toString(16)) - 0xd800).toString(2);
	var binSecond = (parseInt('0x' + parseInt(match[1]).toString(16)) - 0xdc00).toString(2);
	binFirst = '0000000000'.substr(binFirst.length) + binFirst;
	binSecond = '0000000000'.substr(binSecond.length) + binSecond;
	return '&#x' + (('0x' + (parseInt(binFirst + binSecond, 2).toString(16))) - (-0x10000)).toString(16) + ';';
});

document.getElementById('result').innerHTML = str;

//  ������������
//  is turned into
//  😊😘😀😆😂😁
//  which is rendered by the browser as the emojis

Original:<br>&#55357;&#56842;&#55357;&#56856;&#55357;&#56832;&#55357;&#56838;&#55357;&#56834;&#55357;&#56833;<br><br>
Result:<br>
<div id='result'></div>

My SMS XML Parser application is working great now, but it stalls out on large XML files so, I'm thinking about rewriting it in PHP. If/when I do, I'll post that code as well.

Answer 4

I've implemented the algorithm described by rensa above in R, and am sharing it here. I am happy to release the code snippet below under a CC0 dedication (i.e., putting this implementation into the public domain for free reuse).

This is a quick and unpolished implementation of rensa's algorithm, but it works!

utf16_double_dec_code_to_utf8 <- function(utf16_decimal_code){
  string_elements <- str_match_all(utf16_decimal_code, "&#(.*?);")[[1]][,2]

  string3a <- string_elements[1]
  string3b <- string_elements[2]

  string4a <- sprintf("0x0%x", as.numeric(string3a))
  string4b <- sprintf("0x0%x", as.numeric(string3b))

  string5a <- paste0(
    # "0x", 
    as.hexmode(string4a) - 0xd800
  )
  string5b <- paste0(
    # "0x",
    as.hexmode(string4b) - 0xdc00
  )

  string6 <- paste0(
    stringi::stri_pad(
      paste0(BMS::hex2bin(string5a), collapse = ""),
      10,
      pad = "0"
    ) %>%
      stringr::str_trunc(10, side = "left", ellipsis = ""),
    stringi::stri_pad(
      paste0(BMS::hex2bin(string5b), collapse = ""),
      10,
      pad = "0"
    ) %>%
      stringr::str_trunc(10, side = "left", ellipsis = "")
  )

  string7 <- BMS::bin2hex(as.numeric(strsplit(string6, split = "")[[1]]))

  string8 <- as.hexmode(string7) + 0x10000

  unicode_pattern <- string8
  unicode_pattern
}

make_unicode_entity <- function(x) {
  paste0("\\U000", utf16_double_dec_code_to_utf8(x))
}
make_html_entity <- function(x) {
  paste0("&#x", utf16_double_dec_code_to_utf8(x), ";")
}

# An example string, using the "hug" emoji:
example_string <- "test &#55358;&#56599; test"

output_string <- stringr::str_replace_all(
  example_string,
  "(&#[0-9]*?;){2}",  # Find all two-character "&#...;&#...;" codes.
  make_unicode_entity
  # make_html_entity
)

cat(output_string)

# To print Unicode string (doesn't display in R console, but can be copied and
# pasted elsewhere:
# (This assumes you've used 'make_unicode_entity' above in the str_replace_all
# call):
stringi::stri_unescape_unicode(output_string)