Do I need a ETL?

Question

We currently use Datastage ETL to - Export a CSV/text file with data from 15 tables(3 different schemas) on a daily basis.

I am wondering If there is a simpler way t

Answer 1

You do not need a ETL tool for that purpose. You can perform all the tasks using python, right from extracting data from CSVs/XMLs/text files, transforming data (identifying data types, null value transformation) and loading into tables. https://towardsdatascience.com/python-etl-tools-best-8-options-5ef731e70b49

Answer 2

ETL can definitely be performed without the help of ETL Tools. for eg: we can develop python scripts or there is open sources like Drift to work with it.

Answer 3

Why not use a free and easy to use ETL tool such as expressor Studio. You can download it at http://www.expressorstudio.com.

Answer 4

My 2 cents.

Datastage is an awful tool, and expensive to license.

SSIS is much simpler, or cloverETL is good.

ETL tool vs code is a good question.

ETL tools often have better performance as can queue data up ready to be used where programming is is going to do this one at a time, and datastage can do this in parallel (but again i think it blows). PLus ETL tools can get data from multiple heterogeneous sources, where as you cant do this (easily) with code.

However if any data transformations etc are all to be done with data on the same server, I generally end up doing as much in SQL/TSQL(or PL/SQL) as possible, as it is just tonnes easier to debug/maintain. Primary Keys/Foreign Keys are your friend, and any missed lookups can be checked through checking counts later on to ensure data integrity is in order.

Answer 5

We use Python. Every programming language -- every single one ever invented -- is an alternative to an ETL.

You never need an ETL.

The questions is these:

• Which is cheaper to build? Custom software or a configuration of an ETL?

• Which is cheaper to maintain an operate?

• Which is easier to adapt to changing requirements?