Problem using os.system() with sed command

Question

I\'m writing a small method to replace some text in a file. The only argument I need is the new text, as it is always the same file and text to be replaced.

I\'m hav

Answer 1

So from previous answers we now know that id is a Unicode string, which makes cmd1 a Unicode string, which os.system() is converting to a byte string for execution in the default encoding.

a) I suggest using subprocess rather than os.system()

b) I suggest not using the name of a built-in function as a variable (id).

c) I suggest explicitly encoding the string to a byte string before executing:

if isinstance(cmd,unicode):
    cmd = cmd.encode("UTF-8")

d) For Lennart Regebro's suggestion add:

assert type(cmd1) == type(cmd2)

after

print cmd1 == cmd2

Answer 2

Obligatory: don't use os.system - use the subprocess module:

import subprocess

def updateExportConfigId(m_id, source='path/file.old', 
                             destination='path/file.new'):
    if isinstance(m_id, unicode):
        m_id = m_id.encode('utf-8')
    cmd= [
          "sed",
          ",$s/MANAGER_ID=[0-9]*/MANAGER_ID=%s/g" % m_id,  
          source,
         ]
    subprocess.call(cmd, stdout=open(destination, 'w'))

with this code you can pass the manager id, it can have spaces, quote chars, etc. The file names can also be passed to the function, and can also contain spaces and some other special chars. That's because your shell is not unnecessarly invoked, so one less process is started on your OS, and you don't have to worry on escaping special shell characters.

Another option: Don't launch sed. Use python's re module.

import re
def updateExportConfigID(m_id, source, destination):
    if isinstance(m_id, unicode):
        m_id = m_id.encode('utf-8')
    for line in source:
        new_line = re.sub(r'MANAGER_ID=\d*', 
                          r'MANAGER_ID=' + re.escape(m_id), 
                          line)
        destination.write(new_line)

and call it like this:

updateExportConfigID('GRRRR', open('path/file.old'), open('path/file.new', 'w'))

No new processes needed.

Answer 3

What is wrong is that there is some difference. Yeah, I know that's not helpful, but you need to figure out the difference.

Try running this:

import os
def updateExportConfigId(id):
    stringId = "%s" % id
    cmd1 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
    stringId = "GRRRRRRRRR"
    cmd2 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"

    print "cmd1:" , cmd1
    print "cmd2:" , cmd2
    print cmd1 == cmd2

updateExportConfigId("GRRRRRRRRR")

The code should print:

sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
True

Thereby showing that they are exactly the same. If the last line is "False" then they are not the same, and you should be able to see the difference.

Answer 4

To help you debug it, try adding:

print repr(cmd)

It might be that some special characters slipped into the command that normal print is hiding when you copy and paste it.

Answer 5

Finally, I found a way to run the os.system(cmd)!

Simple trick, to "clean" the cmd string:

os.system(str(cmd))

Now, I'm able to build the cmd with all arguments I need and at the end I just "clean" it with str() call before run it with os.system() call.

Thanks a lot for your answers!

swon

Answer 6

Maybe it helps to use only raw strings.