How can I set an expression to the FileSpec property on Foreach File enumerator?

Question

I\'m trying to create an SSIS package to process files from a directory that contains many years worth of files. The files are all named numerically, so to save processing ever

Answer 1

Here is one way you can achieve this. You could use Expression Task combined with Foreach Loop Container to match the numerical values of the file names. Here is an example that illustrates how to do this. The sample uses SSIS 2012.

This may not be very efficient but it is one way of doing this.

Let's assume there is a folder with bunch of files named in the format YYYYMMDD. The folder contains files for the first day of every month since 1921 like 19210101, 19210201, 19210301 .... all the upto current month 20121101. That adds upto 1,103 files.

Let's say the requirement is only to loop through the files that were created since June 1948. That would mean the SSIS package has to loop through only the files greater than 19480601.

On the SSIS package, create the following three parameters. It is better to configure parameters for these because these values are configurable across environment.

• ExtensionToMatch - This parameter of String data type will contain the extension that the package has to loop through. This will supplement the value to FileSpec variable that will be used on the Foreach Loop container.

• FolderToEnumerate - This parameter of String data type will store the folder path that contains the files to loop through.

• MinIndexId - this parameter of Int32 data type will contain the minimum numerical value above which the files should match the pattern.

Create the following four parameters that will help us loop through the files.

• ActiveFilePath - This variable of String data type will hold the file name as the Foreach Loop container loops through each file in the folder. This variable is used in the expression of another variable. To avoid error, set it to a non-empty value, say 1.

• FileCount - This is a dummy variable of Int32 data type will be used for this sample to illustrate the number of files that the Foreach Loop container will loop through.

• FileSpec - This variable of String data type will hold the file pattern to loop through. Set the expression of this variable to below mentioned value. This expression will use the extension specified on the parameters. If there are no extensions, it will *.* to loop through all files.

"*" + (@[$Package::ExtensionToMatch] == "" ? ".*" : @[$Package::ExtensionToMatch])

• ProcessThisFile - This variable of Boolean data type will evaluate whether a particular file matches the criteria or not.

Configure the package as shown below. Foreach loop container will loop through all the files matching the pattern specified on the FileSpec variable. An expression specified on the Expression Task will evaluate during runtime and will populate the variable ProcessThisFile. The variable will then be used on the Precedence constraint to determine whether to process the file or not.

The script task within the Foreach loop container will increment the counter of variable FileCount by 1 for each file that successfully matches the expression.

The script task outside the Foreach loop will simply display how many files were looped through by the Foreach loop container.

Configure the Foreach loop container to loop through the folder using the parameter and the files using the variable.

Store the file name in variable ActiveFilePath as the loop passes through each file.

On the Expression task, set the expression to the following value. The expression will convert the file name without the extension to a number and then will check if it evaluates to greater than the given number in the parameter MinIndexId

@[User::ProcessThisFile] = (DT_BOOL)((DT_I4)(REPLACE(@[User::ActiveFilePath], @[User::FileSpec] ,"")) > @[$Package::MinIndexId] ? 1: 0)

Right-click on the Precedence constraint and configure it to use the variable ProcessThisFile on the expression. This tells the package to process the file only if it matches the condition set on the expression task.

@[User::ProcessThisFile]

On the first script task, I have the variable User::FileCount set to the ReadWriteVariables and the following C# code within the script task. This increments the counter for file that successfully matches the condition.

public void Main()
{
    Dts.Variables["User::FileCount"].Value = Convert.ToInt32(Dts.Variables["User::FileCount"].Value) + 1;
    Dts.TaskResult = (int)ScriptResults.Success;
}

On the second script task, I have the variable User::FileCount set to the ReadOnlyVariables and the following C# code within the script task. This simply outputs the total number of files that were processed.

public void Main()
{
    MessageBox.Show(String.Format("Total files looped through: {0}", Dts.Variables["User::FileCount"].Value));
    Dts.TaskResult = (int)ScriptResults.Success;
}

When the package is executed with MinIndexId set to 1948061 (excluding this), it outputs the value 773.

When the package is executed with MinIndexId set to 20111201 (excluding this), it outputs the value 11.

Hope that helps.