is getting converted as “\u0092” by nokogiri in ruby on rails

问题

I have html page which has following line with some html entities like "".

#Here I am not pasting whole html page content. just putting issue line only
html_file = "<html>....<body><p>they&#146;re originally intended to describe the spread of of viral diseases, but they&amp;#146;re nice analogies for how web/SN apps grow.<p> ...</body></html>"


doc   = Nokogiri::HTML(html)
body  = doc.xpath('//body')
body_content = body[0].inner_html

puts body_content

Result:

These terms come from the fields of medicine and biology  they\u0092re originally intended to describe the spread of of viral diseases, but they\u0092re nice analogies for how web/SN apps grow.

I want to leave these entities as it is instead of changing it to unicode. Any thing, Am I missing?

Thanks

回答1:

they&#146;re

is wrong and should be avoided. If you want to use a close-single-quote there, to reproduce the typographical practice of rendering apostrophes as a slanted quote, then the correct character is U+2019 RIGHT SINGLE QUOTATION MARK, which can be written as ’ or ’. Or, if you're using UTF-8, just included verbatim as ’.

 should refer to character U+0092, a little-used and pointless control character that typically renders as blank or a missing-glyph box. And indeed in XML, it does.

But in HTML (other than XHTML, which uses XML rules), it's a long-standing browser quirk that character references in the range  to  are misinterpreted to mean the characters associated with bytes 128 to 159 in the Windows Western code page (cp1252) instead of the Unicode characters with those code points. The HTML5 standard finally documents this behaviour.

The problem is that Nokogiri doesn't know about this quirk, and takes character reference 146 at its word, ending up with the character 146 (\u0092) that you don't really want. I think Nokogiri is using libxml2 to parse HTML, so ultimately the proper fix would be to libxml2's htmlParseCharRef function, to substitute characters 128–159.

In the meantime you could perhaps try ‘fixing up’ character references manually with crude string substitution like ->’ before parsing. It's a bit wrong, but at least in HTML the only other place you can have the markup sequence  without it being a character reference would be in a comment, so hopefully it wouldn't matter if you changed the content there accidentally too.

回答2:

Have you tried changing

&amp;#146;

into

&#146;

i think the parser parses the ampersand first then concats it with "#146" and then parse them both. it's just an opinion though..I want this to be just a comment IDK how..lol

Well I got the idea from focos in his answer post here, and the unicode from here.

来源：https://stackoverflow.com/questions/7031633/146-is-getting-converted-as-u0092-by-nokogiri-in-ruby-on-rails

标签

ruby-on-rails

ruby

nokogiri

&#146; is getting converted as “\u0092” by nokogiri in ruby on rails

问题

回答1:

回答2:

is getting converted as “\u0092” by nokogiri in ruby on rails