postgresql

INSERT or UPDATE bulk data from dataframe/CSV to PostgreSQL database

帅比萌擦擦* 提交于 2021-02-19 08:57:05
问题 Requirement: Insert new data and update existing data in bulk (row count > 1000) from a dataframe/CSV (which ever suites) and save it in PostgreSQL database. Table: TEST_TABLE CREATE TABLE TEST_TABLE ( itemid varchar(100) NOT NULL PRIMARY KEY, title varchar(255), street varchar(10), pincode VARCHAR(100)); INSERT: ['756252', 'tom title', 'APC Road', '598733' ], ['75623', 'dick title', 'Bush Road', '598787' ], ['756211', 'harry title', 'Obama Street', '598733' ] dataframe content: data = [[

Spring Batch and Postgres database slow write

柔情痞子 提交于 2021-02-19 08:22:07
问题 I've developed a Spring Batch App with the Postgres and particularly facing slow write to database issue for 3 Steps out of 20 steps. I'd like to share example which is taking most time among three. Below is the table where I'm writing data and data count is 2,33,382 . I'm using Spring JDBC and in particular NamedParameterJdbcTemplate to perform the Batch update. CREATE TABLE test.country ( last_update_date date NULL, src_system varchar(20) NULL, country_id varchar(255) NULL, app_id varchar

C# enum to postgres enum

社会主义新天地 提交于 2021-02-19 07:53:45
问题 I am currently using postgres enum CREATE TYPE http_action_enum AS ENUM ('GET', 'HEAD', 'POST', 'PUT', 'DELETE', 'CONNECT', 'OPTIONS', 'TRACE', 'PATCH'); CREATE TABLE IF NOT EXISTS response ( id UUID PRIMARY KEY, http_action http_action_enum NOT NULL ); But when I using the ef framework to insert to postgres database I keep hit the below error: Exception data: Severity: ERROR SqlState: 42804 MessageText: column "destination" is of type source_dest_enum but expression is of type integer Hint:

SQLAlchemy, reflection, different backends and table/column case insensitivity

▼魔方 西西 提交于 2021-02-19 07:53:09
问题 Intro: I'm writing web interface with SQLAlchemy reflection that supports multiple databases. It turns out that authors of application defined postgresql with lowercase tables/columns, eg. job.jobstatus while sqlite has mixed case, eg Job.JobStatus . I'm using DeclarativeReflectedBase from examples to combine reflection and declarative style. The issue: Configure SQLAlchemy to work with tables/columns case insensitive with reflection I have done so far: I have changed DeclarativeReflectedBase

SQLAlchemy, reflection, different backends and table/column case insensitivity

别说谁变了你拦得住时间么 提交于 2021-02-19 07:53:03
问题 Intro: I'm writing web interface with SQLAlchemy reflection that supports multiple databases. It turns out that authors of application defined postgresql with lowercase tables/columns, eg. job.jobstatus while sqlite has mixed case, eg Job.JobStatus . I'm using DeclarativeReflectedBase from examples to combine reflection and declarative style. The issue: Configure SQLAlchemy to work with tables/columns case insensitive with reflection I have done so far: I have changed DeclarativeReflectedBase

PostgreSQL Primary Key sequence lost after migration using AWS DMS

放肆的年华 提交于 2021-02-19 07:45:12
问题 I recently migrated a self hosted Postgres database to AWS RDS using the AWS Migration Service. Postgres Version: 10.6 I have noticed that all of my primary keys are no longer set to "sequence", and when I attempt to manually add a sequence it starts at 1 instead of continuing the count that is already set. I use Rails with the database, so my sql skills are pretty low. I can generally find my way around inserts and updates, but this is not a realm I have much experience in. My question has 2

Calculating distance between a gps location and postgis geography value using a function?

烈酒焚心 提交于 2021-02-19 07:44:08
问题 I have lat and long values for a user. I have addresses of various locations saved in db. I want to find the nearest offers based on a radius value. I am using postgis. Currently my location column is of type geography. What is the format in which I should enter the data into "location" column geography type when I have only latitude and longitude? I cannot use postgis functions since I am using graphql to do data insert. I have been tearing my hair apart trying to write a function for this

Indexing Foreign Keys in Postgresql

点点圈 提交于 2021-02-19 05:59:52
问题 Like many Postgres n00bs we have a lot of tables with foreign key constraints that are not indexed. I some cases this should not be a big performance hit - but this would be subject for further analysis. I have read the following article: https://www.cybertec-postgresql.com/en/index-your-foreign-key/ And used the following query to find all foreign keys without an index: SELECT c.conrelid::regclass AS "table", /* list of key column names in order */ string_agg(a.attname, ',' ORDER BY x.n) AS

How to access postgresql.conf file of postgresql deployment on Heroku

浪尽此生 提交于 2021-02-19 05:59:46
问题 I have a Django app hosted on Heroku, with a postgresql DB, on a Standard 0 plan. I want to access and read the postgresql.conf file for this particular app (to ascertain/edit the settings). Can anyone guide me how to do that? Note: I can't do that via the Heroku dashboard, and I don't know how to access the app's file structure over the Ubuntu commandline. 回答1: You can list all current settings without an access to postgres.conf : SELECT name, current_setting(name) FROM pg_settings; name |

How to flatten a PostgreSQL result

Deadly 提交于 2021-02-19 05:04:28
问题 I have experiments, features, and feature_values. Features have values in different experiments. So I have something like: Experiments: experiment_id, experiment_name Features: feature_id, feature_name Feature_values: experiment_id, feature_id, value Lets say, I have three experiments (exp1, exp2, exp3) and three features (feat1, feat2, feat3). I would like to have a SQL-result that looks like: feature_name | exp1 | exp2 | exp3 -------------+------+------+----- feat1 | 100 | 150 | 110 feat2 |