SqlDataReader does not return all records (3rd attempt)

Question

I\'ve tried to find solution for this problem twice before, but unfortunately those answers haven\'t provided permanent fix, so here I am, giving it another try.

I have

Answer 1

Basically, as we've flogged out in the comments(*), the problem isn't with SqlDataRead, the stored procedure, or SQL at al. Rather, your List.Add is failing because it cannot allocate the additional memory for 2^(n+1) items to extend the List and copy your existing 2^n items into. Most of the time your n=19 (so 524289 items), but sometimes it could be higher.

There are three basic things that you could do about this:

1. Pre-Allocate: As you've discovered, by pre-allocating you should be able to gwet anywhere from 1.5 to 3 times as many items. This works best if you know ahead of time how many items you'll have, so I'd recommend either excuting a SELECT COUNT(*).. ahead of time, or adding a COUNT(*) OVER(PARTITION BY 1) column and picking it out of the first row returned to pre-allocate the List. The problem with this approach is that you're still pretty close to your limit and could easily run out of memory in the near future...

2. Re-Configure: Right now you are only getting at most 2^22 bytes of memory for this, when in theory you shoud be able to get around 2^29-2^30. That means that something on your machine is preventing you from extending your writeable Virtual Memory limit that high. Likely causes include the size of your pagefile and competition from other processes (but there are other possibilities). Fix that and you should have more than enough headroom for this.

3. Streaming: Do you really need all 1.5 million items in memory at the same time? If not and you can determine which you don't need (or extract the info that you do need) on the fly, then you can solve this problem the same way that SqlDataReader does, with streaming. Just read a row, use it, then lose it and go on to the next row.

Hopefully this helps.

(* -- Thanks, obviously, to @granadaCoder and @MartinSmith)

---

If you really think that the problem rests solely with the List data structure (and not that you are just running out of memory), then there are some other ways to work around the List structure's allocation behavior. One way would be to implement an alternative List class (as IList(of Integer)).

Through the interface it would appear the same as List but internally it would have a different allocation scheme, by storing the data in a nested List(of List(of Integer)). Every 1000 items, it would create a new List(of Integer), add it to the parent nested list and then use it to add in the next 1000 items.

The reason that I didn't suggest this before is because, like pre-allocation, this may allow you to get closer to your memory limit, but, if that's the problem, you are still going to run out eventually (just as with pre-allocating) because this limit is too close to the actual number of items that you need (1.5 million).

Answer 2

Basically you read all record in SqlDataReader with select query I suggest you to add order by in your query and it sort all records in Acceding order and they also read in acceding order in SqlDataReader.

I also face this problem in my last project I have read more than 2 million records from database with unique id serialNo but this records are not come in sequence after 1000 records it jumps to 21, 00, 263th record and all records are come in wrong sequence.

Then I use (order by serialNo) this query and my problem is solved you not need to do anything extra only put order by in your select query and it will work for you

I hope this helps for you.