Fastest way to convert datatable to generic list

Question

I have a data tier select method that returns a datatable. It\'s called from a business tier method that should then return a strongly typed generic List.

What I wa

Answer 1

Just to provide some more user - friendliness to Mark's answer in a simple console app :

class Program
{
    static void Main(string[] args)
    {
        //define a DataTable obj
        DataTable table = new DataTable
        {
            Columns = {
            {"Foo", typeof(int)},
            {"Bar", typeof(string)}
         }
        };
        //populate it the DataTable 
        for (int i = 0; i < 3; i++)
        {
            table.Rows.Add(i, "Row " + i);
        }

        List<MyType> listWithTypedObjects= new List<MyType>(table.Rows.Count);
        foreach (DataRow row in table.Rows)
        {
            listWithTypedObjects.Add(new MyType((int)row[0], (string)row[1]));
        }

        Console.WriteLine(" PRINTING THE POPULATED LIST ");
        foreach (MyType objMyType in listWithTypedObjects)
        {
            Console.Write(" I have object of the type " + objMyType.ToString() + " => " );
            Console.Write(" with Prop1OfTypeInt " + objMyType.Prop1OfTypeInt.ToString() + " , ");
            Console.WriteLine(" with Prop1OfTypeInt " + objMyType.Prop2OfTypeString.ToString() + "  "); 
        }

        Console.WriteLine(" \n \n \n HIT A KEY TO EXIT THE PROGRAM ");
        Console.ReadKey();
    }
}

class MyType {

    public int Prop1OfTypeInt { get; set; }
    public string Prop2OfTypeString { get; set; } 

    /// <summary>
    /// Note the order of the passed parameters is important !!!
    /// </summary>
    public MyType( int prop1OfTypeInt , string prop2OfTypeString)
    {
        this.Prop1OfTypeInt = prop1OfTypeInt;
        this.Prop2OfTypeString = prop2OfTypeString; 

    }
}

Answer 2

Do you know the structure of the DataTable and the typed object ahead of time? You could use a delegate to do the mapping. If you don't (i.e. all you know is a Type and properties) there are ways of accelerating dynamic member access (such as HyperDescriptor).

Either way, consider an iterator block; that way you don't have to buffer the objects an entire second time; of course, if you are only dealing with smallish rowcounts this isn't an issue.

Can you clarify any of those points? I can add a lot more detail...

At the simplest, what is wrong with:

DataTable table = new DataTable {
    Columns = {
        {"Foo", typeof(int)},
        {"Bar", typeof(string)}
     }
};
for (int i = 0; i < 5000; i++) {
    table.Rows.Add(i, "Row " + i);
}

List<MyType> data = new List<MyType>(table.Rows.Count);
foreach (DataRow row in table.Rows) {
    data.Add(new MyType((int)row[0], (string)row[1]));
}

(the problems in the above might steer the right approach...)

Answer 3

The problem with the sample above is that it is terribly slow. I have a DataTable with about 400 rows and this conversion takes a good 5 or 6 seconds!

It does seem like a pretty common task, so I'm surprised to not see someone here with a more performant solution.

* Update!! * Just for kicks, I thought I would try converting using LINQ instead of iterating through the DataTable and adding to my List. The following is what I did:

   List<MyObject> returnList = new List<MyObject>();

   MyDataTable dtMyData = new MyTableAdapter().GetMyData();

   returnLists = (from l in dtMyData
                 select new MyObject
                 {
                    Active = l.IsActive,
                    Email = l.Email,
                    //...
                    //About 40 more properties
                    //...
                    ZipCode = l.Zipcode
                  }).ToList();

The first method (iterating through each row) took 5.3 seconds and the method using LINQ took 1.8 seconds!

Answer 4

I know this answer is very late but i spent an hour and above to prepare this code because i needed this first then i thought some one in search of a solution for this question may get use of this..

Inorder for my answer to work correctly, you need to have same names for your list properties as in the DataBase table-columns(fields) or the DataTable column names.

Solution:

public List<dynamic> GetListFromDT(Type className, DataTable dataTable)
        {
            List<dynamic> list = new List<dynamic>();
            foreach (DataRow row in dataTable.Rows)
            {
                object objClass = Activator.CreateInstance(className);
                Type type = objClass.GetType();
                foreach (DataColumn column in row.Table.Columns)
                {
                    PropertyInfo prop = type.GetProperty(column.ColumnName);
                    prop.SetValue(objClass, row[column.ColumnName], null);   
                }
                list.Add(objClass);
            }
            return list;
        }

How to use ?

Lets say you have a values populated datatable named dtPersons

DataTable dtPersons; //populate this datatable

Then lets say you have a class with the following properties.

public class Persons{
    public string Name {get; set;}
    public int Age {get; set;}
}

Now pack and put the method in your model. call the method as below.

List<dynamic> dynamicListReturned = GetListFromDT(typeof(Persons), dataTable);
List<Persons> listPersons = dynamicListReturned.Cast<Persons>().ToList();

That's it now you got your list from the datatable in listPersons.

Remember: Both the names in the class properties and the DataTable/Database should be the same

Answer 5

public partial class issuereceive_manageroffice_bal
{
    public int issue_id{get;set;}
    public string process{get;set;}
    public DateTime issue_date{get;set;}
    public TimeSpan issue_time{get;set;}
    public string eg_no{get;set;}
    public string lotno{get;set;}
    public string clarity{get;set;}
    public string sieves{get;set;}
    public string shape{get;set;}
    public double issue_carat{get;set;}
    public int issue_pieces{get;set;}
    public int receive_pieces{get;set;}
    public double receive_carat{get;set;}
    public int kp_pieces{get;set;}
    public decimal kp_carat{get;set;}
    public double loss{get;set;}
    public string issue_manager{get;set;}
    public string issue_by{get;set;}
    public string receive_by{get;set;}
    public int status{get;set;}
    public DateTime receive_date{get;set;}
    public string receive_time{get;set;}
    public int factory_id{get;set;}

}


List<issuereceive_manageroffice_bal> issue_receive_list = new List<issuereceive_manageroffice_bal>();
issue_receive_list =
      (from DataRow dr in DataTable.Rows
      select new issuereceive_manageroffice_bal()
           {
               issue_id = 0,
               issue_time = TimeSpan.Parse("0"),
               receive_time = null,
               shape = null,
               process = dr["process"].ToString(),
               issue_date = Convert.ToDateTime(dr["issue_date"]),
               eg_no = dr["eg_no"].ToString(),
               lotno = dr["lotno"].ToString(),
               clarity = dr["clarity"].ToString(),
               sieves = dr["sieves"].ToString(),
               issue_carat = dr["issue_carat"].ToString() != "" ? double.Parse(dr["issue_carat"].ToString()) : 0,
               issue_pieces = dr["issue_pieces"].ToString() != "" ? int.Parse(dr["issue_pieces"].ToString()) : 0,
               receive_carat = dr["receive_carat"].ToString() != "" ? double.Parse(dr["receive_carat"].ToString()) : 0,
               kp_pieces = dr["kp_pieces"].ToString() != "" ? int.Parse(dr["kp_pieces"].ToString()) : 0,
               kp_carat = dr["kp_carat"].ToString() != "" ? decimal.Parse(dr["kp_carat"].ToString()) : 0,
               loss = dr["loss"].ToString() != "" ? double.Parse(dr["loss"].ToString()) : 0,
               issue_manager = dr["lotno"].ToString(),
               issue_by = dr["issue_by"].ToString(),
               receive_by = dr["receive_by"].ToString(),
               status = dr["status"].ToString() != "" ? int.Parse(dr["status"].ToString()) : 0,
               receive_date = Convert.ToDateTime(dr["receive_date"]),
               factory_id = dr["factory_id"].ToString() != "" ? int.Parse(dr["factory_id"].ToString()) : 0,

           }).ToList();

Answer 6

You will need to iterate through the datarows and convert them into your objects at some time anyway, but you could write a custom collection with a constructor that takes a collection of datarows and that converts each item into your object type when requested instead of all at once.

Say you define your custom collection that implements IList<T>. It could then have two internal fields - a reference to the collection of datarows and a List<T>. The List<T> would be initialized to the length of the collection of datarows but with null values.

Now in the indexer you could check if the List<T> contains a value at that index and if not you could create the object using whatever means would be useful and store it there before returning it.

This would postpone the creation of your objects until they were requested and you would only create the objects that were requested.

Your objects would probably need a constructor taking a DataRow or you would need some sort of factory to create them, but that is another topic.