Scraping large pdf tables which span across multiple pages

Question

I am trying to scrape PDF tables which span across multiple pages. I tried many things but the best seems to be pdftotext -layout as advised here. The problem is t

Answer 1

In the past I have used pdftohtml which can be used to generate xml, described here. The columns are generally fairly well separated so you could use the positioning to extract columns.

I wrote a large part of pdftables, apologies for the opaqueness! It works OK for some pages of the document you show, for example page 2 gives me the output at the bottom this reply. For other pages it falls over, on page 33, for example. The problem here is that there are two numbers under one column heading and they get stuck together by pdftables. The "COMARCA, CODI i, NOM EMA" columns don't get separated in either case. You can submit issues for pdftables on GitHub, I'm not working on it actively at the moment. It is available by pip install.

If you wanted to go the commercial route then Abbyy FineReader is very good, they produce a cloud SDK which will give you 30 or so pages free. They have example code in multiple languages but their support isn't great.

     14 columns, 39 rows
                                      0    1    2    3    4    5    6    7    8    9   10   11   12   13
    -----------------------------------------------------------------------------------------------------
  0 |             COMARCACODI i NOM EMA| GEN| FEB| MAR| ABR| MAI| JUN| JUL| AGO| SET| OCT| NOV| DES| ANY|
  1 |                  VYNullesAlt Camp| 7,5| 5,5|10,9|12,3|16,7|21,6|22,3|24,4|20,1|15,9|11,0| 8,5|14,8|
  2 |             DQVila-rodonaAlt Camp| 7,9| 5,6|11,0|12,0|16,6|21,6|22,0|24,3|19,9|15,8|11,0| 8,6|14,7|
  3 |              Alt EmpordàU1Cabanes| 8,2| 6,5|11,7|12,6|17,5|22,0|23,1|24,4|20,4|16,6|11,8| 8,3|15,3|
  4 |  Alt EmpordàW1Castelló d'Empúries| 8,1| 6,4|11,6|12,9|17,0|21,1|22,0|23,4|20,1|16,4|12,1| 8,5|15,0|
  5 |              Alt EmpordàVZEspolla| 9,0| 6,7|12,4|12,7|17,8|22,0|23,3|24,8|20,9|16,7|12,0| 8,9|15,6|
  6 |              D6PortbouAlt Empordà| 9,6| 5,5|12,7|12,5|17,4|21,5|22,9|24,4|19,8|17,0|12,3|10,1|15,5|
  7 |                D4RosesAlt Empordà| 9,3| 7,2|13,0|13,6|18,2|22,6|23,9|25,7|21,3|17,5|13,2| 9,9|16,3|
  8 |   Alt EmpordàU2Sant Pere Pescador| 7,8| 6,3|11,5|12,9|16,8|21,2|22,2|23,6|20,2|16,5|12,3| 8,5|15,0|
  9 |  Alt EmpordàW2Torroella de Fluvià| 7,4| 6,0|11,2|12,6|16,4|21,2|22,3|23,7|19,9|16,1|11,7| 8,0|14,7|
 10 |             Alt EmpordàW3Ventalló| 7,3| 6,2|11,4|12,8|16,9|21,8|22,8|24,3|20,4|16,5|12,0| 8,1|15,1|
 11 |            Alt PenedèsWPCanaletes| 7,0| 5,2|11,3|11,9|16,7|21,5|22,0|24,2|19,7|15,6|10,7| 8,1|14,5|
 12 |            Alt PenedèsDIFont-rubí| 8,1| 6,2|12,0|11,9|16,9|21,8|22,0|24,4|20,0|15,9|11,4| 8,9|15,0|
 13 |           Alt PenedèsW4la Granada| 7,0| 5,5|11,2|12,6|17,2|21,9|22,4|24,3|20,0|16,0|11,1| 8,3|14,8|
 14 |   Alt PenedèsU3Sant Martí Sarroca| 6,4| 5,1|10,9|12,4|17,0|21,8|22,3|24,3|19,9|15,7|10,8| 8,0|14,6|
 15 | Alt PenedèsWYSant Sadurní d'Anoia| 6,4| 5,1|11,0|12,8|17,6|22,6|23,2|25,0|20,5|16,2|10,9| 7,8|15,0|
 16 |       CDla Seu d'UrgellAlt Urgell| 3,6| 2,5| 8,5| 8,4|14,6|20,3|21,0|23,4|16,9|12,2| 7,0| 3,2|11,8|
 17 |                W5OlianaAlt Urgell| 2,0| 2,7| 9,8|10,2|16,8|23,0|22,9|25,6|19,1|13,9| 8,6| 3,1|13,2|
 18 |               Alt UrgellCJOrganyà| 2,6| 3,5| 9,8| 9,9|16,1|22,0|22,6|25,3|18,8|13,5| 8,2| 2,9|13,0|
 19 |     Alta RibagorçaZ2Boí (2.535 m)|-2,4|-7,5|-1,3|-3,4| 3,8| 8,6| 9,4|12,0| 6,3| 2,7|-1,1|-3,2| 2,0|
 20 |  Alta RibagorçaCTel Pont de Suert| 0,5| 1,6| 6,9| 7,9|14,1|18,0|19,1|20,4|15,7|10,7| 6,1| 1,3|10,2|
 21 |   CEels Hostalets de PierolaAnoia| 7,3| 5,5|11,7|12,1|17,4|22,4|22,9|25,2|20,3|16,2|11,1| 8,3|15,1|
 22 |                 XBla LlacunaAnoia| 5,4| 3,3| 9,3|10,3|15,6|20,8|20,9|23,3|18,0|14,1| 9,1| 6,9|13,1|
 23 |               AnoiaXAla Panadella| 3,6| 1,7| 9,2| 8,7|14,9|20,5|20,4|23,2|17,2|13,3| 7,9| 5,1|12,2|
 24 |                      H1Ã’denaAnoia| 5,1| 3,3| 9,4|11,5|16,3|21,7|22,5|24,6|19,4|15,2| 9,3| 6,0|13,7|
 25 |                      WWArtésBages| 3,5| 2,8| 9,2|11,2|16,6|22,4|23,2|25,1|19,3|15,0| 9,1| 4,3|13,5|
 26 |        U4Castellnou de BagesBages| 4,8| 3,8|10,5|10,9|16,3|22,0|22,5|25,0|19,3|15,0| 9,6| 5,9|13,9|
 27 |        R1el Pont de VilomaraBages| 3,8| 3,1| 9,9|12,3|17,4|22,9|23,5|25,4|20,0|15,7| 9,7| 5,0|14,1|
 28 |    BagesWNMontserrat - Sant Dimes| 6,2| 3,3| 9,7| 8,6|14,8|19,5|19,5|22,4|16,9|13,5| 9,0| 7,1|12,6|
 29 | CLSant Salvador de GuardiolaBages| 3,3| 2,8| 9,1|11,5|16,4|22,0|22,4|24,6|19,2|14,9| 9,1| 4,8|13,4|
 30 |   U5Prades - los HortalsBaix Camp| 2,8| 0,0| 6,4| 7,4|13,0|18,4|18,0|21,3|15,0|11,3| 6,5| 4,1|10,4|
 31 |                W6RiudomsBaix Camp| 9,7| 7,1|12,0|13,4|17,6|22,4|23,1|25,2|21,2|17,1|12,3|10,1|16,0|
 32 |     U6Vinyols i els ArcsBaix Camp|10,2| 7,6|12,0|13,8|17,6|22,5|24,0|25,9|22,3|18,2|13,2|11,1|16,6|
 33 |                Baix EbreU7Aldover|10,0| 8,5|13,2|14,8|19,7|24,6|25,2|27,1|22,7|18,3|12,9|11,1|17,4|
 34 |             DBel PerellóBaix Ebre| 8,7| 7,0|12,0|13,3|17,9|22,6|23,3|25,3|21,4|17,2|11,9|10,3|15,9|
 35 |                U9l'AldeaBaix Ebre| 9,9| 8,1|12,5|14,3|18,5|23,3|24,1|26,0|22,1|17,9|13,1|10,7|16,8|
 36 |       UAl'Ametlla de MarBaix Ebre| 9,6| 7,8|12,3|13,8|18,0|22,9|23,9|25,8|22,0|17,6|12,5|10,6|16,4|
 37 |          Baix EbreX5PN dels Ports| 3,4|-0,2| 6,5| 6,8|13,4|18,7|17,8|21,2|15,2|11,3| 6,1| 4,9|10,5|
 38 |       Baix EmpordàDOCastell d'Aro| 6,7| 5,1|10,6|12,0|16,2|20,9|21,8|23,8|20,1|16,3|12,2| 8,1|14,5|
    -----------------------------------------------------------------------------------------------------  

The unicode problems are down to my dev environment (Spyder).