sqlalchemy

How to efficiently load mixed-type pandas DataFrame into an Oracle DB

由 删除回忆录丶 提交于 2021-02-10 18:18:18
问题 Happy new year everyone! I'm currently struggling with ETL performance issues as I'm trying to write larger Pandas DataFrames (1-2 mio rows, 150 columns) into an Oracle data base . Even for just 1000 rows, Panda's default to_sql() method runs well over 2 minutes (see code snippet below). My strong hypothesis is that these performance issues are in some way related to the underlying data types (mostly strings). I ran the same job on 1000 rows of random strings (benchmark: 3 min) and 1000 rows

pandas/sqlalchemy/pyodbc: Result object does not return rows from stored proc when UPDATE statement appears before SELECT

由 人盡茶涼 提交于 2021-02-10 17:44:43
问题 I'm using SQL Server 2014, pandas 0.23.4, sqlalchemy 1.2.11, pyodbc 4.0.24, and Python 3.7.0. I have a very simple stored procedure that performs an UPDATE on a table and then a SELECT on it: CREATE PROCEDURE my_proc_1 @v2 INT AS BEGIN UPDATE my_table_1 SET v2 = @v2 ; SELECT * from my_table_1 ; END GO This runs fine in MS SQL Server Management Studio. However, when I try to invoke it via Python using this code: import pandas as pd from sqlalchemy import create_engine if __name__ == "__main__"

带你认识 flask 错误处理

由 喜欢而已 提交于 2021-02-10 16:56:51
点击上方蓝字关注我们 欢迎关注我的公众号，志学Python 01 flask 中错误处理机制 在Flask应用中爆发错误时会发生什么？得到答案的最好的方法就是亲身体验一下。启动应用，并确保至少有两个用户注册，以其中一个用户身份登录，打开个人主页并单击“编辑”链接。在个人资料编辑器中，尝试将用户名更改为已经注册的另一个用户的用户名，boom！（爆炸声） 这将带来一个可怕的“Internal Server Error”页面： 如果你查看运行应用的终端会话，将看到 stack trace （堆栈跟踪）。堆栈跟踪在调试错误时非常有用，因为它们显示堆栈中调用的顺序，一直到产生错误的行： (venv) $ flask run * Serving Flask app "microblog" * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) [2017-09-14 22:40:02,027] ERROR in app: Exception on /edit_profile [POST] Traceback (most recent call last): File "/home/miguel/microblog/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py",

SQLAlchemy one-to-one, store foreign key in each table?

由 邮差的信 提交于 2021-02-10 14:14:56
问题 I have a relationship that is one to one between cage codes and duns numbers. I have set up my relationship that looks like, where I store a ForeignKey on each of the respective tables. class Cage(Base): __tablename__ = 'DimCage' id = Column(Integer, primary_key=True) cage = Column(String(8), unique=True, nullable=False) duns_id = Column(Integer, ForeignKey('DimDuns.id')) duns = relationship('Duns', uselist=False, back_populates='cage') class Duns(Base): __tablename__ = 'DimDuns' id = Column

Python: How to mock SQLAlchemy event handlers (using mock.unittest)

由 谁都会走 提交于 2021-02-10 12:13:55
问题 So I have a SQLAlchemy model that has an event listener: class User(db.Model): id = db.Column(db.Integer, primary_key=True) @event.listens_for(User, "after_insert") @event.listens_for(User, "after_update") def do_something(mapper, connection, self): foo = SomeClass(self) foo.do_something_to_database() And I have a unit test that needs to update/insert the Model @patch('my_package.user.do_something') def test_user(mock_do_something): user = User() # This insert will invoke 'do_something' via

How to tell if a InstrumentedAttribute is a relation in sqlalchemy?

由 泄露秘密 提交于 2021-02-10 12:06:22
问题 Given a sqlalchemy model class class Table(Base): __table__name = 'table' col1 = Column(...) rel = relationship(...) If you look at Table , both col1 and rel are of type InstrumentedAttribute . How can I distinguish these two? I tried to look into their attributes. But there are too many...and there's no document about this matter. 回答1: Check the type of Table.<column>.property which can commonly be either instance of ColumnProperty or RelationshipProperty: >>> type(Table.col1.property)

Python: How to mock SQLAlchemy event handlers (using mock.unittest)

由 时间秒杀一切 提交于 2021-02-10 12:06:18
问题 So I have a SQLAlchemy model that has an event listener: class User(db.Model): id = db.Column(db.Integer, primary_key=True) @event.listens_for(User, "after_insert") @event.listens_for(User, "after_update") def do_something(mapper, connection, self): foo = SomeClass(self) foo.do_something_to_database() And I have a unit test that needs to update/insert the Model @patch('my_package.user.do_something') def test_user(mock_do_something): user = User() # This insert will invoke 'do_something' via

How to tell if a InstrumentedAttribute is a relation in sqlalchemy?

由 眉间皱痕 提交于 2021-02-10 12:04:23
问题 Given a sqlalchemy model class class Table(Base): __table__name = 'table' col1 = Column(...) rel = relationship(...) If you look at Table , both col1 and rel are of type InstrumentedAttribute . How can I distinguish these two? I tried to look into their attributes. But there are too many...and there's no document about this matter. 回答1: Check the type of Table.<column>.property which can commonly be either instance of ColumnProperty or RelationshipProperty: >>> type(Table.col1.property)

Unable to connect to CloudSQL from Kubernetes Engine (Can't connect to MySQL server on 'localhost')

由 南笙酒味 提交于 2021-02-10 06:30:52
问题 I tried to follow the steps in: https://cloud.google.com/sql/docs/mysql/connect-kubernetes-engine. I have the application container and the cloudsql proxy container running in the same pod. After creating the cluster, logs for the proxy container seems correct: $kubectl logs users-app-HASH1-HASH2 cloudsql-proxy 2018/08/03 18:58:45 using credential file for authentication; email=it-test@tutorial-bookshelf-xxxxxx.iam.gserviceaccount.com 2018/08/03 18:58:45 Listening on 127.0.0.1:3306 for

Ignoring a model when using alembic autogenerate

由 浪子不回头ぞ 提交于 2021-02-10 05:21:46
问题 I am trying to autogenerate revisions for my DB using alembic . While doing so, I want to ignore some models (they have data types that are not supported by current version of MySQL). Here is what I tried and it seems to work fine, but I am not sure that's the most idiomatic way of doing it inside alembic/env.py def include_object(object, type_, name, reflected, compare_to): if type_ == 'table' and name == 'model_to_be_ignored': return False return True and then inside run_migrations_online
订阅 sqlalchemy