SSIS/SQL Automation

Question

My company is a direct mail company, and we are looking for ways to improve workflow as we continue to grow via automation with SSIS, SQL and Excel.

Basically, what

Answer 1

You have two general approaches: an event driven process or a polling process. The former runs when an event happens (file lands in a folder) while the latter runs on a periodic basis (every 5 minutes it looks to see if a file exists).

It is my experience that the event driven model sounds really good in practice but is horrible in implementation. We used a variety of off the shelf software and homegrown "file watcher" tasks to process data. Inevitably, something would happen and it wasn't registering the event so no data was being processed. The resolution was usually simple, restart the process and then move the files out and back again. Except we worked in a regulated environment where we didn't have access to move the files so that would take a different person to reset the queue versus who could move files about.

A polling process is much easier. If there's work to be done, it does so. Otherwise it goes back to sleep. Windows Task Scheduler can do this just fine. SQL Server has it's own job scheduling system called SQL Agent. It too can run packages at various intervals.

It might be tempting to have SSIS run in a infinite loop checking for files or using WMI events to try to catch a file being placed but don't do either of those. SSIS is not designed to be running all the time. Instead, use proven products to handle the stuff they were meant for.