connect SQL to apache nifi

Question

I\'m new to nifi and i want to connect SQL server database to nifi and create a data flow with the processors. how can I do this, can any one Help me with this clearly.

Answer 1

Here are two great articles on getting information in and out of databases with NiFi:

1. http://www.batchiq.com/database-injest-with-nifi.html

2. http://www.batchiq.com/database-extract-with-nifi.html

They describe/illustrate how to configure a DBCPConnectionPool service to provide connection(s) to an RDBMS, and example flows to extract data and ingest data.

Answer 2

Expanding on TamusJRoyce's answer

If you are running nifi via a docker image like apache/nifi or the aforementioned Hortonworks sandbox, the following should help you get the required driver on the image so that you don't need to exec into the container to do it manually.

See the comments below the docker file

FROM apache/nifi

USER root
RUN mkdir /lib/jdbc
WORKDIR /lib/jdbc
RUN wget https://download.microsoft.com/download/3/F/7/3F74A9B9-C5F0-43EA-A721-07DA590FD186/sqljdbc_6.2.2.0_enu.tar.gz
RUN tar xvzf sqljdbc_6.2.2.0_enu.tar.gz
RUN cp ./sqljdbc_6.2/enu/mssql-jdbc-6.2.2.jre8.jar ./

USER nifi

EXPOSE 8080 8443 10000 8000

WORKDIR ${NIFI_HOME}
ENTRYPOINT ["../scripts/start.sh"]

1. The above image uses apache/nifi as the base image. You can use any nifi docker image has a base if you would like.
2. You can specify any location for lib/jdbc, just remember that you need to use this as the reference for the file location so that it is referenced as file:///lib/jdbc/mssql-jdbc-6.2.2.jre8.jar
3. Lastly, switch back to the nifi user and finish off with the standard nifi image details. This will allow the image to run correctly.

Answer 3

Expanding on mattyb answer

If you are using the latest Hortonworks sandbox, or other setup that uses docker containers, read below.

You have to install the JDBC jar file inside the docker. For SQL Server, it should be 6.2 or above.

docker ps
docker exec -it <mycontainer uuid> bash

How do I get into a Docker container's shell? will help you log into the container.

cd file:///usr/lib/jvm/jre/lib/
mkdir jdbc
cd ./jdbc
wget https://download.microsoft.com/download/3/F/7/3F74A9B9-C5F0-43EA-A721-07DA590FD186/sqljdbc_6.2.2.0_enu.tar.gz
tar xvzf sqljdbc_6.2.2.0_enu.tar.gz
cp ./sqljdbc_6.2/enu/mssql-jdbc-6.2.2.jre8.jar ./

jdbc:sqlserver://192.168.1.201:1433;databaseName=[your database]
com.microsoft.sqlserver.jdbc.SQLServerDriver

You might need to replace the ip address with IPv4 address of your host found under ipconfig in Windows or ifconfig in Mac/Linux.

You may change file:///usr/lib/jvm/jre/lib/ to any path you desire.