Write contents of URL request to file

Question

I am trying to fetch a list from a php file using python and save it to a file:

import urllib.request

page = urllib.request.urlopen('http://crypto-bot.hopto.org/server/list.php')

f = open("test.txt", "w")
f.write(str(page))
f.close()

print(page.read())

Output on screen (divided onto four lines for readability):

ALF\nAMC\nANC\nARG\nBQC\nBTB\nBTE\nBTG\nBUK\nCAP\nCGB\nCLR\nCMC\nCRC\nCSC\nDGC\n
DMD\nELC\nEMD\nFRC\nFRK\nFST\nFTC\nGDC\nGLC\nGLD\nGLX\nHBN\nIXC\nKGC\nLBW\nLKY\n
LTC\nMEC\nMNC\nNBL\nNEC\nNMC\nNRB\nNVC\nPHS\nPPC\nPXC\nPYC\nQRK\nSBC\nSPT\nSRC\n
STR\nTRC\nWDC\nXPM\nYAC\nYBC\nZET\n

Output in file:

<http.client.HTTPResponse object at 0x00000000031DAEF0>

Can you tell me what I am doing wrong?

Answer 1

Use urllib.urlretrieve (urllib.request.urlretrieve in Python 3).

In the console:

>>> import urllib
>>> urllib.urlretrieve('http://crypto-bot.hopto.org/server/list.php','test.txt') 
('test.txt', <httplib.HTTPMessage instance at 0x101338050>)

This results in a file, test.txt, being saving in the current working directory with the contents

ALF
AMC
ANC
ARG
...etc...

Answer 2

You need to read from the file object before writing to the file. Also you should the same object to both file and screen.

Do this:

import urllib.request

page = urllib.request.urlopen('http://crypto-bot.hopto.org/server/list.php')

f = open("test.txt", "w")
content = page.read()
f.write(content)
f.close()

print(content)

Answer 3

You're not reading the content from the urlopen file-like when you write to the file.

Also, shutil.copyfileobj().