SSIS failing to save packages and reboots Visual Studio

Question

This is my first experience with SSIS so bear with me... I am using SSIS to migrate tables from Oracle to SSMS, there are some very large tables I am trying to transfer (50

Answer 1

Are you running your packages in parallel ? If yes, change to serie.

You can also try to divide this big table into subsets using an operation like modulo. See that example :

http://henkvandervalk.com/reading-as-fast-as-possible-from-a-table-with-ssis-part-ii

(in the example, he is running in parallel, but you can put this in serie)

Also, if you are running the SSIS package on a computer that is running an instance of SQL Server, when you run the package, set the Maximum server memory option for the SQL Server instance to a smaller value. That will increases available memory.

Answer 2

I suggest reading data in chunks:

Instead of loading the whole table, try to split the data into chunks and import them to SQL Server. From a while, I answered a similar answer related to SQLite, i will try to reproduce it to fit the Oracle syntax:

---

Step by Step guide

In this example each chunk contains 10000 rows.

1. Declare 2 Variables of type Int32 (@[User::RowCount] and @[User::IncrementValue])
2. Add an Execute SQL Task that execute a select Count(*) command and store the Result Set into the variable @[User::RowCount]

3. Add a For Loop with the following preferences:

4. Inside the for loop container add a Data flow task
5. Inside the dataflow task add an ODBC Source and OLEDB Destination
6. In the ODBC Source select SQL Command option and write a SELECT * FROM TABLE query *(to retrieve metadata only`
7. Map the columns between source and destination
8. Go back to the Control flow and click on the Data flow task and hit F4 to view the properties window

9. In the properties window go to expression and Assign the following expression to [ODBC Source].[SQLCommand] property: (for more info refer to How to pass SSIS variables in ODBC SQLCommand expression?)

   "SELECT * FROM MYTABLE ORDER BY ID_COLUMN
   OFFSET " + (DT_WSTR,50)@[User::IncrementValue] + "FETCH NEXT 10000 ROWS ONLY;"
   

Where MYTABLE is the source table name, and IDCOLUMN is your primary key or identity column.

Control Flow Screenshot

References

• ODBC Source - SQL Server
• How to pass SSIS variables in ODBC SQLCommand expression?
• HOW TO USE SSIS ODBC SOURCE AND DIFFERENCE BETWEEN OLE DB AND ODBC?
• How do I limit the number of rows returned by an Oracle query after ordering?
• Getting top n to n rows from db2

---

Update 1 - Other possible workarounds

While searching for similar issues i found some additional workarounds that you can try:

(1) Change the SQL Server max memory

• SSIS: The Buffer Manager Failed a Memory Allocation Call

  sp_configure 'show advanced options', 1;
  GO
  RECONFIGURE;
  GO
  sp_configure 'max server memory', 4096;
  GO
  RECONFIGURE;
  GO
  

(2) Enable Named pipes

• [Fixed] The buffer manager detected that the system was low on virtual memory, but was unable to swap out any buffers

  1. Go to Control Panel – > Administrative Tools -> Computer Management
  2. On Protocol for SQL Instance -> Set Named Pipes = Enabled
  3. Restart the SQL instance Service
  4. After that try to import the data and it will fetch the data in chunks now instead of fetch all at once. Hope that will work for you guys and save your time.

(3) If using SQL Server 2008 install hotfixes

• The SSIS 2008 runtime process crashes when you run the SSIS 2008 package under a low-memory condition

---

Update 2 - Understanding the error

In the following MSDN link, the error cause was described as following:

Virtual memory is a superset of physical memory. Processes in Windows typically do not specify which they are to use, as that would (greatly) inhibit how Windows can multitask. SSIS allocates virtual memory. If Windows is able to, all of these allocations are held in physical memory, where access is faster. However, if SSIS requests more memory than is physically available, then that virtual memory spills to disk, making the package operate orders of magnitude slower. And in worst cases, if there is not enough virtual memory in the system, then the package will fail.