Using DataTable in .NET Core

Question

I have a stored procedure in SQL Server that accepts a User-Defined Table Type. I\'m following the answer from this post Bulk insert from C# list into SQL Server into multip

Answer 1

There are 2 solutions to this problem. One is using DbDataReader as @meziantou suggested in his answer, and was nice to supply a generic method that will convert an IEnumerable<T> to a DbDataReader.

The other solution I found was using SqlDataRecord, so I'm writing it down here (use whatever you see fit to your needs):

SQL Server table:

CREATE TABLE [dbo].[Users](
    [UserId] [int] IDENTITY(1,1) NOT NULL,
    [FirstName] [nvarchar](50) NULL,
    [LastNAme] [nvarchar](50) NULL,
 CONSTRAINT [PK_USers] PRIMARY KEY CLUSTERED 
(
    [UserId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

User-Defined Table Type:

CREATE TYPE [dbo].[TblUser] AS TABLE(
    [FirstName] [nvarchar](50) NULL,
    [LastNAme] [nvarchar](50) NULL
)

.NET Core code:

var db = new SqlConnection("Server=localhost; Database=Test; User Id=test; Password=123456;");

List<SqlDataRecord> users = new List<SqlDataRecord>();

SqlMetaData mDataFirstName = new SqlMetaData("FirstName", SqlDbType.NVarChar, 50);
SqlMetaData mDataLastName = new SqlMetaData("LastName", SqlDbType.NVarChar, 50);

SqlDataRecord user1 = new SqlDataRecord(new []{ mDataFirstName, mDataLastName });
user1.SetString(0, "Ophir");
user1.SetString(1, "Oren");
users.Add(user1);

SqlParameter param = new SqlParameter("@Users", SqlDbType.Structured)
{
    TypeName = "TblUser",
    Value = users
};

Dictionary<string, object> values = new Dictionary<string, object>();
values.Add("@Users", param);


db.Open();
using (var command = db.CreateCommand())
{
    command.CommandType = System.Data.CommandType.StoredProcedure;
    command.CommandText = "stp_Users_Insert";

    var p1 = command.CreateParameter();
    command.Parameters.Add(p1);
    p1.ParameterName = "@Users";
    p1.SqlDbType = System.Data.SqlDbType.Structured;
    p1.Value = users;
    command.ExecuteNonQuery();
}

Answer 2

I had the same issue, that you can't create a DataTable and therefore just dump that into a sheet.

The lack of DataTable support in Core does force you to create strongly typed objects and then loop through and map those to the output of EPPlus.

So a very simple example is to:

// Get your data directly from EF,
// or from whatever other source into a list,
// or Enumerable of the type
List<MyEntity> data = _whateverService.GetData();

using (ExcelPackage pck = new ExcelPackage())
{
   // Create a new sheet
   var newSheet = pck.Workbook.Worksheets.Add("Sheet 1");
   // Set the header:
   newSheet.Cells["A1"].Value = "Column 1 - Erm ID?";
   newSheet.Cells["B1"].Value = "Column 2 - Some data";
   newSheet.Cells["C1"].Value = "Column 3 - Other data";

  row = 2;
  foreach (var datarow in data)
  {
      // Set the data:
      newSheet.Cells["A" + row].Value = datarow.Id;
      newSheet.Cells["B" + row].Value = datarow.Column2;
      newSheet.Cells["C" + row].Value = datarow.Cilumn3;
      row++;
  }
}

So, you're taking an enumerable source of and strongly typed object, which you can do directly from an EF query, or a view model or anything else and then looping through to map it.

I've used this and the performance appears - to an end user - on par with the DataTable method. I've not inspected the source, but it wouldn't surprise me if the DataTable method is just doing the same thing internally and looping through each row.

You could create an extension method to using generics to pass in the object list and use reflection to map it correctly... Maybe I'll look at the project and see if I can contribute.

Edit to add:

In .NET Core it appears, from the GitHub issues tracker that DataTable support is pretty low down on the priority list, and do not expect it anytime soon. I think it's also a philosophical point, as the concept is generally you try to use strongly typed objects. So, it used to be that you could run a SQL Query into a DataTable and run with that... Now, you should run that query into a Model either directly mapped to a table with Entity Framework via a DbSet, or with ModelBinding and passing a type into the query.

You then have an IQueryable<T> which serves as your strongly typed replacement to DataTables. To be fair to this approach, for 99% of cases it a valid and better approach... However there will always be times the lack of DataTables will cause issues and need to be worked around!

Further Edit

In ADO.NET you can convert a datareader to a strongly typed list of objects: How can I easily convert DataReader to List<T>? to name but one example. With this list you can make your mapping from there.

If you want to / have to use ASP.NET Core that targets ASP.NET Core framework then you'll have to do this. If you can target Net 4.5 using a Core project then you'll be able to use System.Data and have DataTables back - the only caveat being that you then have to use Windows servers and IIS.

Do you really need the full .Net Core framework? Do you need to host on linux? If no, and you really need DataTables, just target the older Framework.

Answer 3

Developer82,

I am in the same situation where i want to use .net core but the unavailability of datatable, dataset is a bummer. since you are referencing a post that uses a List i thought that maybe the goal is to get the C# List into the database the cleanest way possible. If that is the goal then this might help.

I have used Dapper located here in various projects. It is supported int .netcore. below is a small console app that takes a populated c# list and inserts it into the DB and then issues a Select on that table to writes out the results to the console.

using System;
using System.Data;
using Dapper;
using System.Data.Common;
using System.Data.SqlClient;
using System.Collections.Generic;

namespace TestConsoleApp
{
    class Program
    {
        static void Main(string[] args)
        {
            List<DataItemDTO> dataItems = GetDataItems();

            var _selectSql = @"SELECT CustomerId, Name, BalanceDue from [dbo].[CustomerAccount]";

            var _insertSql = @"INSERT INTO [dbo].[CustomerAccount]
                                       ([CustomerId]
                                       ,[Name]
                                       ,[BalanceDue])
                                 VALUES
                                       (@CustomerId
                                       ,@Name
                                       ,@BalanceDue)";



            using (IDbConnection cn = new SqlConnection(@"Server=localhost\xxxxxxx;Database=xxxxdb;Trusted_Connection=True;"))
            {
                var rows = cn.Execute(_insertSql, dataItems,null,null,null );

                dataItems.Clear();

                var results = cn.Query<DataItemDTO>(_selectSql);

                foreach (var item in results)
                {
                    Console.WriteLine("CustomerId: {0} Name {1} BalanceDue {2}", item.CustomerId.ToString(), item.Name, item.BalanceDue.ToString());
                }


            }

            Console.WriteLine("Press any Key");
            Console.ReadKey();

        }

        private static List<DataItemDTO> GetDataItems()
        {
            List<DataItemDTO> items = new List<DataItemDTO>();

            items.Add(new DataItemDTO() { CustomerId = 1, Name = "abc1", BalanceDue = 11.58 });
            items.Add(new DataItemDTO() { CustomerId = 2, Name = "abc2", BalanceDue = 21.57 });
            items.Add(new DataItemDTO() { CustomerId = 3, Name = "abc3", BalanceDue = 31.56 });
            items.Add(new DataItemDTO() { CustomerId = 4, Name = "abc4", BalanceDue = 41.55 });
            items.Add(new DataItemDTO() { CustomerId = 5, Name = "abc5", BalanceDue = 51.54 });
            items.Add(new DataItemDTO() { CustomerId = 6, Name = "abc6", BalanceDue = 61.53 });
            items.Add(new DataItemDTO() { CustomerId = 7, Name = "abc7", BalanceDue = 71.52 });
            items.Add(new DataItemDTO() { CustomerId = 8, Name = "abc8", BalanceDue = 81.51 });

            return items;
        }
    }
}

I hope this code sample helps.

thank you.

Answer 4

You can use a DbDataReader as the value of the SQL parameter. So, the idea is to convert an IEnumerable<T> to a DbDataReader.

public class ObjectDataReader<T> : DbDataReader
{
    private bool _iteratorOwned;
    private IEnumerator<T> _iterator;
    private IDictionary<string, int> _propertyNameToOrdinal = new Dictionary<string, int>();
    private IDictionary<int, string> _ordinalToPropertyName = new Dictionary<int, string>();
    private Func<T, object>[] _getPropertyValueFuncs;

    public ObjectDataReader(IEnumerable<T> enumerable)
    {
        if (enumerable == null) throw new ArgumentNullException(nameof(enumerable));

        _iteratorOwned = true;
        _iterator = enumerable.GetEnumerator();
        _iterator.MoveNext();
        Initialize();
    }

    public ObjectDataReader(IEnumerator<T> iterator)
    {
        if (iterator == null) throw new ArgumentNullException(nameof(iterator));

        _iterator = iterator;    
        Initialize();
    }

    protected override void Dispose(bool disposing)
    {
        if (disposing && _iteratorOwned)
        {
            if(_iterator != null)
                _iterator.Dispose();
        }

        base.Dispose(disposing);
    }

    private void Initialize()
    {
        int ordinal = 0;
        var properties = typeof(T).GetProperties();
        _getPropertyValueFuncs = new Func<T, object>[properties.Length];
        foreach (var property in properties)
        {
            string propertyName = property.Name;
            _propertyNameToOrdinal.Add(propertyName, ordinal);
            _ordinalToPropertyName.Add(ordinal, propertyName);

            var parameterExpression = Expression.Parameter(typeof(T), "x");
            var func = (Func<T, object>)Expression.Lambda(Expression.Convert(Expression.Property(parameterExpression, propertyName), typeof(object)), parameterExpression).Compile();
            _getPropertyValueFuncs[ordinal] = func;

            ordinal++;
        }
    }

    public override object this[int ordinal] 
    {
        get
        {
            return GetValue(ordinal);
        }
    }

    public override object this[string name]
    {
        get
        {
            return GetValue(GetOrdinal(name));
        }
    }

    public override int Depth => 1;

    public override int FieldCount => _ordinalToPropertyName.Count;

    public override bool HasRows => true;

    public override bool IsClosed
    {
        get
        {
            return _iterator != null;
        }
    }

    public override int RecordsAffected
    {
        get
        {
            throw new NotImplementedException();
        }
    }

    public override bool GetBoolean(int ordinal)
    {
        return (bool)GetValue(ordinal);
    }

    public override byte GetByte(int ordinal)
    {
        return (byte)GetValue(ordinal);
    }

    public override long GetBytes(int ordinal, long dataOffset, byte[] buffer, int bufferOffset, int length)
    {
        throw new NotImplementedException();
    }

    public override char GetChar(int ordinal)
    {
        return (char)GetValue(ordinal);
    }

    public override long GetChars(int ordinal, long dataOffset, char[] buffer, int bufferOffset, int length)
    {
        throw new NotImplementedException();
    }

    public override string GetDataTypeName(int ordinal)
    {
        throw new NotImplementedException();
    }

    public override DateTime GetDateTime(int ordinal)
    {
        return (DateTime)GetValue(ordinal);
    }

    public override decimal GetDecimal(int ordinal)
    {
        return (decimal)GetValue(ordinal);
    }

    public override double GetDouble(int ordinal)
    {
        return (double)GetValue(ordinal);
    }

    public override IEnumerator GetEnumerator()
    {
        throw new NotImplementedException();
    }

    public override Type GetFieldType(int ordinal)
    {
        var value = GetValue(ordinal);
        if (value == null)
            return typeof(object);

        return value.GetType();
    }

    public override float GetFloat(int ordinal)
    {
        return (float)GetValue(ordinal);
    }

    public override Guid GetGuid(int ordinal)
    {
        return (Guid)GetValue(ordinal);
    }

    public override short GetInt16(int ordinal)
    {
        return (short)GetValue(ordinal);
    }

    public override int GetInt32(int ordinal)
    {
        return (int)GetValue(ordinal);
    }

    public override long GetInt64(int ordinal)
    {
        return (long)GetValue(ordinal);
    }

    public override string GetName(int ordinal)
    {
        string name;
        if (_ordinalToPropertyName.TryGetValue(ordinal, out name))
            return name;

        return null;
    }

    public override int GetOrdinal(string name)
    {
        int ordinal;
        if (_propertyNameToOrdinal.TryGetValue(name, out ordinal))
            return ordinal;

        return -1;
    }

    public override string GetString(int ordinal)
    {
        return (string)GetValue(ordinal);
    }

    public override object GetValue(int ordinal)
    {
        var func = _getPropertyValueFuncs[ordinal];
        return func(_iterator.Current);
    }

    public override int GetValues(object[] values)
    {
        int max = Math.Min(values.Length, FieldCount);
        for (var i = 0; i < max; i++)
        {
            values[i] = IsDBNull(i) ? DBNull.Value : GetValue(i);
        }

        return max;
    }

    public override bool IsDBNull(int ordinal)
    {
        return GetValue(ordinal) == null;
    }

    public override bool NextResult()
    {
        return false;
    }

    public override bool Read()
    {
        return _iterator.MoveNext();
    }
}

Then, you can use this class:

static void Main(string[] args)
{
    Console.WriteLine("Hello World!");
    string connectionString = "Server=(local);Database=Sample;Trusted_Connection=True;";

    using (var connection = new SqlConnection(connectionString))
    {
        connection.Open();

        using (var command = connection.CreateCommand())
        {
            command.CommandType = System.Data.CommandType.StoredProcedure;
            command.CommandText = "procMergePageView";

            var p1 = command.CreateParameter();
            command.Parameters.Add(p1);    
            p1.ParameterName = "@Display";
            p1.SqlDbType = System.Data.SqlDbType.Structured;
            var items = PageViewTableType.Generate(100);
            using (DbDataReader dr = new ObjectDataReader<PageViewTableType>(items))
            {
                p1.Value = dr;
                command.ExecuteNonQuery();
            }    
        }
    }
}

class PageViewTableType
{
    // Must match the name of the column of the TVP
    public long PageViewID { get; set; }

    // Generate dummy data
    public static IEnumerable<PageViewTableType> Generate(int count)
    {
        for (int i = 0; i < count; i++)
        {
            yield return new PageViewTableType { PageViewID = i };
        }
    }
}

The SQL scripts:

CREATE TABLE dbo.PageView
(
    PageViewID BIGINT NOT NULL CONSTRAINT pkPageView PRIMARY KEY CLUSTERED,
    PageViewCount BIGINT NOT NULL
);
GO

CREATE TYPE dbo.PageViewTableType AS TABLE
(
    PageViewID BIGINT NOT NULL
);
GO

CREATE PROCEDURE dbo.procMergePageView
    @Display dbo.PageViewTableType READONLY
AS
BEGIN
    MERGE INTO dbo.PageView AS T
    USING @Display AS S
    ON T.PageViewID = S.PageViewID
    WHEN MATCHED THEN UPDATE SET T.PageViewCount = T.PageViewCount + 1
    WHEN NOT MATCHED THEN INSERT VALUES(S.PageViewID, 1);
END

By the way, I've written a blog post about the ObjectDataReader<T>

Answer 5

@meziantou. I love your answer, but there is a breaking bug in your implementation. The first problem was that MoveNext was called in the constructor, which would cause any iteration of the reader to always skip the first value. Once I removed that, I discovered why that was done in the first place. I changed GetFieldType to use the Type information, rather than reading the type from the value, which solved that problem. Again, really excellent answer from you. Thank you for posting. Here is my fixed up version of ObjectDataReader.

    public class ObjectDataReader<T> : DbDataReader
    {   
        private bool _iteratorOwned;
        private IEnumerator<T> _iterator;
        private IDictionary<string, int> _propertyNameToOrdinal = new Dictionary<string, int>();
        private IDictionary<int, string> _ordinalToPropertyName = new Dictionary<int, string>();
        private PropertyInfoContainer[] _propertyInfos;

        class PropertyInfoContainer
        {
            public Func<T, object> EvaluatePropertyFunction { get; set; }
            public Type PropertyType { get; set; }
            public string PropertyName { get; set; }
            public PropertyInfoContainer(string propertyName
                , Type propertyType
                , Func<T, object> evaluatePropertyFunction)
            {
                this.PropertyName = propertyName;
                this.PropertyType = propertyType;
                this.EvaluatePropertyFunction = evaluatePropertyFunction;
            }
        }

        public ObjectDataReader(IEnumerable<T> enumerable)
        {
            if (enumerable == null) throw new ArgumentNullException(nameof(enumerable));

            _iteratorOwned = true;
            _iterator = enumerable.GetEnumerator();
            //_iterator.MoveNext();
            Initialize();
        }

        public ObjectDataReader(IEnumerator<T> iterator)
        {
            if (iterator == null) throw new ArgumentNullException(nameof(iterator));

            _iterator = iterator;
            Initialize();
        }

        protected override void Dispose(bool disposing)
        {
            if (disposing && _iteratorOwned)
            {
                if (_iterator != null)
                    _iterator.Dispose();
            }

            base.Dispose(disposing);
        }

        private void Initialize()
        {
            int ordinal = 0;
            var properties = typeof(T).GetProperties();
            _propertyInfos = new PropertyInfoContainer[properties.Length];
            foreach (var property in properties)
            {
                string propertyName = property.Name;
                _propertyNameToOrdinal.Add(propertyName, ordinal);
                _ordinalToPropertyName.Add(ordinal, propertyName);

                var parameterExpression = Expression.Parameter(typeof(T), "x");
                var func = (Func<T, object>)Expression.Lambda(Expression.Convert(Expression.Property(parameterExpression, propertyName), typeof(object)), parameterExpression).Compile();
                _propertyInfos[ordinal] = new PropertyInfoContainer(property.Name
                    , property.PropertyType
                    , func);

                ordinal++;
            }
        }

        public override object this[int ordinal]
        {
            get
            {
                return GetValue(ordinal);
            }
        }

        public override object this[string name]
        {
            get
            {
                return GetValue(GetOrdinal(name));
            }
        }

        public override int Depth => 1;

        public override int FieldCount => _ordinalToPropertyName.Count;

        public override bool HasRows => true;

        public override bool IsClosed
        {
            get
            {
                return _iterator != null;
            }
        }

        public override int RecordsAffected
        {
            get
            {
                throw new NotImplementedException();
            }
        }

        public override bool GetBoolean(int ordinal)
        {
            return (bool)GetValue(ordinal);
        }

        public override byte GetByte(int ordinal)
        {
            return (byte)GetValue(ordinal);
        }

        public override long GetBytes(int ordinal, long dataOffset, byte[] buffer, int bufferOffset, int length)
        {
            throw new NotImplementedException();
        }

        public override char GetChar(int ordinal)
        {
            return (char)GetValue(ordinal);
        }

        public override long GetChars(int ordinal, long dataOffset, char[] buffer, int bufferOffset, int length)
        {
            throw new NotImplementedException();
        }

        public override string GetDataTypeName(int ordinal)
        {
            throw new NotImplementedException();
        }

        public override DateTime GetDateTime(int ordinal)
        {
            return (DateTime)GetValue(ordinal);
        }

        public override decimal GetDecimal(int ordinal)
        {
            return (decimal)GetValue(ordinal);
        }

        public override double GetDouble(int ordinal)
        {
            return (double)GetValue(ordinal);
        }

        public override IEnumerator GetEnumerator()
        {
            throw new NotImplementedException();
        }

        public override Type GetFieldType(int ordinal)
        {    
            // cannot handle nullable types, so get underlying type
            var propertyType =
                                Nullable.GetUnderlyingType(_propertyInfos[ordinal].PropertyType) ?? _propertyInfos[ordinal].PropertyType;

            return propertyType;
        }

        public override float GetFloat(int ordinal)
        {
            return (float)GetValue(ordinal);
        }

        public override Guid GetGuid(int ordinal)
        {
            return (Guid)GetValue(ordinal);
        }

        public override short GetInt16(int ordinal)
        {
            return (short)GetValue(ordinal);
        }

        public override int GetInt32(int ordinal)
        {
            return (int)GetValue(ordinal);
        }

        public override long GetInt64(int ordinal)
        {
            return (long)GetValue(ordinal);
        }

        public override string GetName(int ordinal)
        {
            string name;
            if (_ordinalToPropertyName.TryGetValue(ordinal, out name))
                return name;

            return null;
        }

        public override int GetOrdinal(string name)
        {
            int ordinal;
            if (_propertyNameToOrdinal.TryGetValue(name, out ordinal))
                return ordinal;

            return -1;
        }

        public override string GetString(int ordinal)
        {
            return (string)GetValue(ordinal);
        }

        public override object GetValue(int ordinal)
        {
            var func = _propertyInfos[ordinal].EvaluatePropertyFunction;
            return func(_iterator.Current);
        }

        public override int GetValues(object[] values)
        {
            int max = Math.Min(values.Length, FieldCount);
            for (var i = 0; i < max; i++)
            {
                values[i] = IsDBNull(i) ? DBNull.Value : GetValue(i);
            }

            return max;
        }

        public override bool IsDBNull(int ordinal)
        {
            return GetValue(ordinal) == null;
        }

        public override bool NextResult()
        {
            return false;
        }

        public override bool Read()
        {
            return _iterator.MoveNext();
        }
    }

Answer 6

DataTable is now supported in .NET CORE 2.0. See my answer at .Net Core how to implement SQLAdapter ./ DataTable function . Sample code below works in 2.0.

public static DataTable ExecuteDataTableSqlDA(SqlConnection conn, CommandType cmdType, string cmdText, SqlParameter[] cmdParms)
{
   System.Data.DataTable dt = new DataTable();
   System.Data.SqlClient.SqlDataAdapter da = new SqlDataAdapter(cmdText, conn);
   da.Fill(dt);
   return dt;
}