Why does PowerShell redirection >> change the formatting of the text content?

Question

I want to use the redirect append >> or write > to write to a txt file, but when I do, I receive a weird format "\\x00a\\x00p..

Answer 1

Note: The problem is ultimately that in Windows PowerShell different cmdlets / operators use different default encodings. This problem has been resolved in PowerShell Core(v6+), where BOM-less UTF-8 is consistently used.

---

• >> blindly applies Out-File's default encoding when appending to an existing file (in effect, > behaves like Out-File and >> like Out-File -Append), which in Windows PowerShell is the encoding named Unicode, i.e., UTF-16LE, where most characters are encoded as 2-byte sequences, even those in the ASCII range; the latter have a 0x0 (NUL) as the high byte.

  • Therefore, unless the target file's existing contents use the same encoding, you'll end up with a mix of different encodings, which is what happened in your case.^[1]

• While Add-Content, by contrast, does try to detect a file's existing encoding^{Thanks again, js2010.}, you used it on an empty file, in which case Set-Content's default encoding is applied, which in Windows PowerShell is the encoding named Default, which refers to your system's active ANSI code page.

• Therefore, to match the single-byte ANSI encoding initially created by your Add-Content call when appending further content, use Out-File -Append -Encoding Default instead of >>, or simply keep using Add-Content.

  • Alternatively, pick a different encoding with Add-Content -Encoding ... and match it in the Out-File -Append call; UTF-8 is generally the best choice, though note that when you create a UTF-8 file in Windows PowerShell, it will start with a BOM (a pseudo byte-order mark identifying the file as UTF-8, which Unix-like platforms typically do not expect).

  • In PowerShell v5.1+ you may also change the default encoding globally, including for > and >> (which isn't possible in earlier versions). To change to UTF-8, for instance, use:
    $PSDefaultParameterValues['*:Encoding']='UTF8'

---

Aside from different default encodings (in Windows PowerShell), it is important to note that Set-Content / Add-Content on the one hand and > / >> / Out-File [-Append] on the other behave fundamentally differently with non-string input:

In short: the former apply simple .ToString()-formatting to the input objects, whereas the latter perform the same output formatting you would see in the console - see this answer for details.

---

^{[1] Due to the initial content set by Add-Content, Windows PowerShell interprets the file as ANSI-encoded (the default in the absence of a BOM), where each byte is its own character. The UTF-16 content appended later is therefore also interpreted as if it were ANSI, so the 0x0 bytes are treated like characters in their own right, which print to the console like spaces.}