sql-insert

How to improve SQLite insert performance in Python 3.6?

99封情书 提交于 2020-07-17 06:48:18
问题 Background I would like to insert 1-million records to SQLite using Python. I tried a number of ways to improve it but it is still not so satisfied. The database load file to memory using 0.23 second (search pass below) but SQLite 1.77 second to load and insert to file. Environment Intel Core i7-7700 @ 3.6GHz 16GB RAM Micron 1100 256GB SSD, Windows 10 x64 Python 3.6.5 Minconda sqlite3.version 2.6.0 GenerateData.py I generate the 1 million test input data with the same format as my real data.

C# & SQL Server : how to Insert DBNull.Value in command parameter if string value is empty?

流过昼夜 提交于 2020-07-09 15:47:25
问题 I've been searching for a couple of hours now and cannot find a solution. I am inserting some strings into SQL, however on on occasions, the method I use to do this may contain strings that are empty, i.e "" I therefore want to insert a null value into SQL Server instead. Firstly I test my method to make sure I am able to insert a DBNull.Value manually by using the following: cmd.Parameters.AddWithValue("@surname", DBNull.Value); // THIS WORKS OK HOWEVER when I try the following below, the

C# & SQL Server : how to Insert DBNull.Value in command parameter if string value is empty?

怎甘沉沦 提交于 2020-07-09 15:47:07
问题 I've been searching for a couple of hours now and cannot find a solution. I am inserting some strings into SQL, however on on occasions, the method I use to do this may contain strings that are empty, i.e "" I therefore want to insert a null value into SQL Server instead. Firstly I test my method to make sure I am able to insert a DBNull.Value manually by using the following: cmd.Parameters.AddWithValue("@surname", DBNull.Value); // THIS WORKS OK HOWEVER when I try the following below, the

how avoid dublicate data to mysql

天涯浪子 提交于 2020-06-23 14:02:21
问题 I write the code to scrap car info(title, make, model, transmission, year, price) data from ebay.com and save in the mysql, I want if all row's(title, make, model, ...) item's is be similar to another row then avoid to insert this data to the mysql, *only when all row's item be similar(because some title is simialr or some model or...) code : import requests from bs4 import BeautifulSoup import re import mysql.connector conn = mysql.connector.connect(user='root', password='******', host='127

Does sqlsrv_query limit the number of statements that can be exectuted in one query?

∥☆過路亽.° 提交于 2020-06-16 17:25:52
问题 I am generating a SQL insert statement within a PHP for loop. The SQL string generated is a large number of individual SQL statements like this: INSERT INTO tbl VALUES(1,2,3); INSERT INTO tbl VALUES(4,5,6); INSERT INTO tbl VALUES(7,8,9); etc... Then I execute with: $InsertResult = sqlsrv_query($conn, $InsertSQL); The problem is that only the first 312 statements get executed instead of the full 2082 lines (only 312 rows are inserted into the table). When I output the $InsertSQL variable to

Does sqlsrv_query limit the number of statements that can be exectuted in one query?

有些话、适合烂在心里 提交于 2020-06-16 17:25:14
问题 I am generating a SQL insert statement within a PHP for loop. The SQL string generated is a large number of individual SQL statements like this: INSERT INTO tbl VALUES(1,2,3); INSERT INTO tbl VALUES(4,5,6); INSERT INTO tbl VALUES(7,8,9); etc... Then I execute with: $InsertResult = sqlsrv_query($conn, $InsertSQL); The problem is that only the first 312 statements get executed instead of the full 2082 lines (only 312 rows are inserted into the table). When I output the $InsertSQL variable to

Postgres error: null value in column “id” - during insert operation

纵然是瞬间 提交于 2020-06-11 16:19:58
问题 I use postgresql and yii2 framework. Well I got a very interesting error message: SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "id" violates not-null constraint DETAIL: Failing row contains (null, 1, null, null, null, null, 1, Demo, , , , 1998-01-01, , , , 345345435453453, , , , , 1, , , f, f, f, f, 10, f, 1, f, f, f, null, null, null, 1470477479, 1470477479, null). But I checked my Insert command, and there is not "id" column there! INSERT INTO "advertiser" ("languages"

Postgres error: null value in column “id” - during insert operation

99封情书 提交于 2020-06-11 16:18:55
问题 I use postgresql and yii2 framework. Well I got a very interesting error message: SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "id" violates not-null constraint DETAIL: Failing row contains (null, 1, null, null, null, null, 1, Demo, , , , 1998-01-01, , , , 345345435453453, , , , , 1, , , f, f, f, f, 10, f, 1, f, f, f, null, null, null, 1470477479, 1470477479, null). But I checked my Insert command, and there is not "id" column there! INSERT INTO "advertiser" ("languages"

how to have postgres ignore inserts with a duplicate key but keep going

早过忘川 提交于 2020-05-27 02:43:12
问题 I am inserting record data in a collection in memory into postgres and want the database to ignore any record that already exists in the database (by virtue of having the same primary key) but keep going with the rest of my inserts. I'm using clojure and hugsql, btw, but I'm guessing the answer might be language agnostic. As I'm essentially treating the database as a set in this way I may be engaging in an antipattern. 回答1: If you're using Postgres 9.5 or newer (which I assume you are, since

how to have postgres ignore inserts with a duplicate key but keep going

本秂侑毒 提交于 2020-05-27 02:41:06
问题 I am inserting record data in a collection in memory into postgres and want the database to ignore any record that already exists in the database (by virtue of having the same primary key) but keep going with the rest of my inserts. I'm using clojure and hugsql, btw, but I'm guessing the answer might be language agnostic. As I'm essentially treating the database as a set in this way I may be engaging in an antipattern. 回答1: If you're using Postgres 9.5 or newer (which I assume you are, since