build SQL dynamic query with psycopg2 python library and using good conversion type tools

Question

I have some problem to design a good algorithm which use specification of psycopg2 library described here

I want to build a dynamic query equal to this string :

Answer 1

You are trying to pass a table name as a parameter. You probably could've seen this immediately if you'd just looked at the PostgreSQL error log.

The table name you're trying to pass through psycopg2 as a parameter is being escaped, producing a query like:

INSERT INTO E'my_table'(name, url, id, point_geom, poly_geom) VALUES (E'ST_GeomFromText(''POLYGON(( 52.146542 19.050557, 52.148430 19.045527, 52.149525 19.045831, 52.147400 19.050780, 52.147400 19.050780, 52.146542 19.050557))'',4326)');'

This isn't what you intended and won't work; you can't escape a table name like a literal. You must use normal Python string interpolation to construct dynamic SQL, you can only use parameterized statement placeholders for actual literal values.

params = ('POLYGON(( 52.146542 19.050557, 52.148430 19.045527, 52.149525 19.045831, 52.147400 19.050780, 52.147400 19.050780, 52.146542 19.050557))',4326)
escaped_name = name.replace('"",'""')
curs.execute('INSERT INTO "%s"(name, url, id, point_geom, poly_geom) VALUES (ST_GeomFromText(%%s,%%s));' % escaped_name, params)

See how I've interpolated the name directly to produce the query string:

INSERT INTO my_table(name, url, id, point_geom, poly_geom) VALUES (ST_GeomFromText(%s,%s));

(%% gets converted to plain % by % substitution). Then I'm using that query with the string defining the POLYGON and the other argument to ST_GeomFromText as query parameters.

I haven't tested this, but it should give you the right idea and help explain what's wrong.

BE EXTEMELY CAREFUL when doing string interpolation like this, it's an easy avenue for SQL injection. I've done very crude quoting in the code shown above, but I'd want to use a proper identifier quoting function if your client library offers one.

Answer 2

Now that 2.7 is on PyPi here is is an example for a dynamic query.

In this example I'll assume the polygon as dictionary from your csv file. Keys could be name, url, id, point_geom, poly_geom as mentioned above but they won't matter really as long as the table structure contains the same keys.

There's probably a way to shorten this but I hope this clarifies the use of the sql functions, namely sql.SQL, sql.Identifier, and sql.Placeholder and how to concatenate a list of strings sql.SQL('..').join(list()).

from psycopg2 import sql
table = 'my_table'
polygon = Polyogon.from_file()  # or something
column_list = list()
value_list = list()

# Convert the dictionary to lists
for column, value in polygon.items():
    column_list.append(sql.Identifier(column))  # Convert to identifiers
    value_list.append(value)

# Build the query, values will be inserted later
query = sql.SQL("INSERT INTO {} ({}) VALUES ({}) ON CONFLICT DO NOTHING").format(
                sql.Identifier(table),
                sql.SQL(', ').join(column_list),  # already sql.Identifier
                sql.SQL(', ').join([sql.Placeholder()] * len(value_list)))

# Execute the cursor
with postgres.cursor() as p_cursor:
    # execute requires tuples and not a list
    p_cursor.execute(insert_query, tuple(value_list))  

Reference: http://initd.org/psycopg/docs/sql.html

Answer 3

The proper way is to use psycopg2 2.7's new sql module which includes an Identifier object. This allows you to dynamically specify SQL identifiers in a safe way.

Unfortunately 2.7 is not on PyPi yet (2.6.2 as of writing).

Until then, psycopg2 cover this under the heading "How can I pass field/table names to a query?" http://initd.org/psycopg/docs/faq.html#problems-with-type-conversions

You can pass SQL identifiers in along with data values to the execute function by using the AsIs function.

Note: this provides NO security. It is as good as using a format string, which is not recommended. The only real advantage of this is you encourage future code to follow the execute + data style. You can also easily search for AsIs in future.

from psycopg2.extensions import AsIs
<snip>
with transaction() as cur:
    # WARNING: not secure
    cur.execute('SELECT * from %(table)s', {'table': AsIs('mytable')})