发表新帖
发表新帖

BeautifulSoup, a dictionary from an HTML table

后端 未结
关注
4 1674
既然无缘
既然无缘 2020-12-05 07:38

I am trying to scrape table data from a website.

Here is a simple example table:

t = \'\' +\\
    \'


        
                      

              相关标签:
       

      
      
      

        

          4条回答        


        
    
                         				            
  • 
            
           
            
    
                              
                 
              
              
    
                
                  南旧        
                
              
    
                            
    
                2020-12-05 08:29
              
    
            
    
            
    
                                                                       

    You can follow the same approach as mvillaress, but improve it a bit, using List Comprehensions:
    


    from BeautifulSoup import BeautifulSoup

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

bs = BeautifulSoup(t)
tds = [row.findAll('td') for row in bs.findAll('tr')]
results = { td[0].string: td[1].string for td in tds }
print results

    
                                                                        
    
                                                        
    
            
    
            
    
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
    
                                                            
    
            
    
                      
          
    • 
          	          
  • 
            
           
            
    
                              
                 
              
              
    
                
                  挽巷        
                
              
    
                            
    
                2020-12-05 08:30
              
    
            
    
            
    
                                                                       

    If you're scraping a table has an explicit "thead" and "tbody" such as:
    


    <table>
    <thead>
        <tr>
            <th>Total</th>
            <th>Finished</th>
            <th>Unfinished</th>
        </tr>
    </thead>
    <tbody>
        <tr> <td>63</td> <td>33</td> <td>2</td> </tr>
        <tr> <td>69</td> <td>29</td> <td>3</td> </tr>
        <tr> <td>57</td> <td>28</td> <td>1</td> </tr>
    </tbody>
</table>

    


    You can use the following:
    


    headers = [header.text_content() for header in table.cssselect("thead tr th")]
results = [{headers[i]: cell.text_content() for i, cell in enumerate(row.cssselect("td"))} for row in table.cssselect("tbody tr")]

    


    This will produce:
    


    [
  {"Total": "63", "Finished": "33", "Unfinished": "2"},
  {"Total": "69", "Finished": "29", "Unfinished": "3"},
  {"Total": "57", "Finished": "28", "Unfinished": "1"}
]

    


    P.S. This is using lxml.html. If you are using BeautifulSoup replace ".text_content()" with ".string" and ".cssselect" with ".findAll".
    
                                                                        
    
                                                        
    
            
    
            
    
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
    
                                                            
    
            
    
                      
          
    • 
          	          
  • 
            
           
            
    
                              
                 
              
              
    
                
                  梦毁少年i        
                
              
    
                            
    
                2020-12-05 08:34
              
    
            
    
            
    
                                                                       

    Try this:
    


    from BeautifulSoup import BeautifulSoup, Comment

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

bs = BeautifulSoup(t)

results = {}
for row in bs.findAll('tr'):
    aux = row.findAll('td')
    results[aux[0].string] = aux[1].string

print results

    
                                                                        
    
                                                        
    
            
    
            
    
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
    
                                                            
    
            
    
                      
          
    • 
          	          
  • 
            
           
            
    
                              
                 
              
              
    
                
                  借酒劲吻你        
                
              
    
                            
    
                2020-12-05 08:34
              
    
            
    
            
    
                                                                       

    BeautifulSoup and Python have evolved, so if someone comes here with newer versions:
    


    Python>=3.7
BeautifulSoup>=4.7

    


    Here's updated code that works:
    


    # import bs4 and create your 'soup' object

table = soup.find('table')

headers = [header.text for header in table.find_all('th')]
results = [{headers[i]: cell for i, cell in enumerate(row.find_all('td'))}
           for row in table.find_all('tr')]

    
                                                                        
    
                                                        
    
            
    
            
    
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
    
                                                            
    
            
    
                      
          
    • 
          	          
                             

        
        

          

            
            

              
              

            
    


                                 
              

            

                          
    

        
         

                
                

                  
                

                

                   看不清?
                

              

                                  

                    
   
                 
             
              
提交回复

            

          

        

      

       
      
    
    

          
 
     
 

        
热议问题