python, lxml and xpath - html table parsing

Question

I \'am new to lxml, quite new to python and could not find a solution to the following:

I need to import a few tables with 3 columns and an undefined number of rows star

Answer 1

You need to use a loop to access the row's data, like this:

for row in data:  
    for col in row:
        print col

Calling next() once as you did will access only the first item, which is why you see one column.

Note that due to the nature of generators, you can only access them once. If you changed the call process_row(row) into list(process_row(row)), the generator would be converted to a list which can be reused.

Update: If you need just the 3rd row and on, use data[2:]

Answer 2

This is a generator:

def process_row(row):  
     for cell in row.xpath('./td'):  
         print cell.text_content()  
         yield cell.text_content() 

You're calling it as though you thought it returns a list. It doesn't. There are contexts in which it behaves like a list:

print [r for r in process_row(row)]

but that's only because a generator and a list both expose the same interface to for loops. Using it in a context where it gets evaluated just one time, e.g.:

return [process_row(row) for row in table.xpath('./tr')]

just calls a new instance of the generator once for each new value of row, returning the first result yielded.

So that's your first problem. Your second one is that you're expecting:

tbl = doc.xpath("//body/table[2]//tr[position()>2]")[0]

to give you the third and all subsequent rows, and it's only setting tbl to the third row. Well, the call to xpath is returning the third and all subsequent rows. It's the [0] at the end that's messing you up.