character showing up in files. How to remove them?

问题

I am doing compressing of JavaScript files and the compressor is complaining that my files have ï»¿ character in them.

How can I search for these characters and remove them?

回答1:

perl -pi~ -CSD -e 's/^\x{fffe}//' file1.js path/to/file2.js

I would assume the tool will break if you have other utf-8 in your files, but if not, perhaps this workaround can help you. (Untested ...)

Edit: added the -CSD option, as per tchrist's comment.

回答2:

You can easily remove them using vim, here are the steps:

1) In your terminal, open the file using vim:

vim file_name

2) Remove all BOM characters:

:set nobomb

3) Save the file:

:wq

回答3:

Another method to remove those characters - using Vim:

vim -b fileName

Now those "hidden" characters are visible (<feff>) and can be removed.

回答4:

Thanks for the previous answers, here's a sed(1) variant just in case:

sed '1s/^\xEF\xBB\xBF//'

回答5:

On Unix/Linux:

sed 's/\xEF\xBB\xBF//' < inputfile > outputfile

On MacOSX

sed $'s/\xEF\xBB\xBF//' < inputfile > outputfile

Notice the $ after sed for mac.

On Windows

There is Super Sed an enhanced version of sed. For Windows this is a standalone .exe, intended for running from the command line.

回答6:

Using tail might be easier:

tail --bytes=+4 filename > new_filename

回答7:

@tripleee's solution didn't work for me. But changing the file encoding to ASCII and again to UTF-8 did the trick :-)

回答8:

I've used vimgrep for this

:vim "[\uFEFF]" *

also normal vim search command

/[\uFEFF]

回答9:

In windows you could use backported recode utility from UnxUtils.

回答10:

In Sublime Text you can install the Highlighter package and then customize the regular expression in your user settings.

Here I added \uFEFF to the end of the highlighter_regex property.

{
    "highlighter_enabled": true,
    "highlighter_regex": "(\t+ +)|( +\t+)|[\u2026\u2018\u2019\u201c\u201d\u2013\u2014\uFEFF]|[\t ]+$",
    "highlighter_scope_name": "invalid",
    "highlighter_max_file_size": 1048576,
    "highlighter_delay": 3000
}

To overwrite the default package settings place the file here:

~/.config/sublime-text-3/Packages/User/highlighter.sublime-settings

回答11:

The 'file' command shows if the BOM is present:

For example: 'file myfile.xml' displays: "XML 1.0 document, UTF-8 Unicode (with BOM) text, with very long lines, with CRLF line terminators"

dos2unix will remove the BOM.

回答12:

Save the file without code signature.

回答13:

I'm suggest the use of "dos2unix" tool, please test to run dos2unix ./thefile.js.

If necessary try to use something like this for multiple files:

for x in $(find . -type f -exec echo {} +); do dos2unix $x ; done

My Regards.

来源：https://stackoverflow.com/questions/7297888/0xef-0xbb-0xbf-character-showing-up-in-files-how-to-remove-them

标签

file

unicode

utf-8

utf

<0xEF,0xBB,0xBF> character showing up in files. How to remove them?

问题