How can I loop through multiple files and process them using SSIS package?

Question

We have a situation where we are importing a number of files with the same format, using the same SSIS package. The package is being loaded and executed from a .net assembly.

Answer 1

I understand that you have found the answer to your question by using Foreach Loop Container. I think that is the best approach in my opinion if the files are of the same format. I am posting the following example so other users can find it useful.

One possible way of achieving this is by using ForEach Loop container in the Control Flow tab and then placing a Data Flow task within the Control Flow task. I have explained the process in the below example. This example uses SQL Server back-end as the destination and Excel 97-2003 format .xls as the source files. Please note that the Excel files should be of same format.

Step-by-step process:

1. Create a table named dbo.Location as shown in screenshot #1. This example will populate this table by reading three different Excel files having the same layout. The screenshot shows an empty table before the package execution.

2. Create two Excel files in path c:\temp\ as shown in screenshots #2 - #4. Notice that both the Excel files have the same layout but different content.

3. On the SSIS package, create three variables as shown in screenshot #5. Variable FolderPath will contain the path where the Excel files are located; FileExtension will contain the Excel file extension (here in this case it is *.xls) and FilePath should be configured to point to one valid Excel file (this is required only during the initial configuration of the Excel connection manager).

4. Create an Excel connection in the connection manager pointing to one valid Excel file as shown in screenshot #6.

5. Create an OLE DB Connection in the connection manager pointing to the SQL Server.

6. On the SSIS package, place a ForEach Loop container and a Data Flow task within the ForEach loop container as shown in screenshot #7.

7. Configure ForEach loop container as shown in screenshots #8 and #9. By doing this, variable User::FilePath will contain the full path Excel files located in the folder c:\temp\ with the help of variables FolderPath and FileExtension configured on the Collection section.

8. Inside the data flow task, place an Excel source to read Excel file data and OLE DB destination to insert data into SQL Server table dbo.Location. Data flow task should look like as shown in screenshot #10.

9. Configure the Excel source as shown in screenshots #11 and #12 to read the data using Excel connection.

10. Configure the OLE DB destination as shown in screenshots #13 and #14 to insert the data into SQL Server database table.

11. On the Excel connection in the connection manager, configure the Expressions ExcelFilePath and ServerName as shown in screenshot #15.

12. Sample execution of the data flow task is shown in screenshot #16.

13. Screenshot #17 displays the data in the table dbo.Location after package execution. Please note that it contains all the rows present in Excel files shown in screenshots #3 and #4.

14. On the Data Flow task properties, Set the DelayValidation to True so that the SSIS doesn't throw errors when you open the package.

Hope that helps.

Screenshot #1:

Screenshot #2:

Screenshot #3:

Screenshot #4:

Screenshot #5:

Screenshot #6:

Screenshot #7:

Screenshot #8:

Screenshot #9:

Screenshot #10:

Screenshot #11:

Screenshot #12:

Screenshot #13:

Screenshot #14:

Screenshot #15:

Screenshot #16:

Screenshot #17:

Screenshot #18:

Answer 2

In the end, I used a foreach loop in the package to process each file in term. This was very fast! I still have a lot to learn about SSIS!

Answer 3

It seems like it would be safe to me. But I would run some controlled tests on the process, perhaps with a very simple SSIS package first, then your more complex one.

I can't say that I have done this exact scenario myself.