WCF data serialization : can it go faster?

Question

This question is sort of a sequel to that question.

When we want to build a WCF service which works with some kind of data, it\'s natural that we want it to be fast

Answer 1

Have you considered creating a View Model for your object and doing a projection in the select statement. That should be a lot faster so:

 var result = from person in DB.Entities.Persons
    .Include("District")
    .Include("District.City")
    .Include("District.City.State")
    .Include("Nationality")
    select new PersonViewModel()
    {
        Name = person.Name, 
        City = person.District.City, 
        State = person.District.City.State
        Nationality = person.Nationality.Name
    };

This would require you to create a ViewModel class to hold the flattened data for the PersonViewModel.

You might be able to further speed up things by creating a database view and letting Entity Framework select directly from there.

If you rally want the front-end to populate a grid with 500.000 records, then I'd remove the webservice layer altogether and use a DataReader to speed up the process. Entity Framework and WCF aren't suitable for transforming the data at a proper performance. What you're basically doing here is:

Database -> TDS -> .NET objects -> XML -> Plain text -> XML -> .NET Objects -> UI

While this could easily be reduced to:

Database -> TDS -> UI

Then use EntityFramwork to handle the changes to the entities in your business logic. This is in line with the Command and Query Separation pattern. Use a technology suitable for high performance querying of data and link that directly to your app. Then use a command strategy to implement your business logic.

OData services might also provide a better way to link your UI directly to the data, as it can be used to quickly query your data allowing you to implement quick filtering without the user really noticing.

If the security settings are prohibiting direct querying through OData or direct access to the SQL database, consider materializing the objects yourself. Select the data directly from either a view or a query and use a IDataReader to directly populate your ViewModel. That will probably give you the highest performance.

There are a lot of alternatives to Entity Framework created especially because EF isn't cut out for large datasets. See FluentData DapperDotNet, Massive or PetaPoco. You might want to use these side-by-side with entity Framework to handle your large, flat data queries.

Answer 2

Based on what I have gathered by looking at various reviews and performance benchmarks, I would choose protobuf-net as a serializer. It's just a matter of design whether it can be plugged into my service configuration. More info about that here.

Although not completely an answer to this question, jessehouwing had the best answer and I am marking it as accepted.

Answer 3

I use Json.Net's implementation of Bson in my RIA application. More info here.

I yield return an IEnumerable, as I read from the database and serialize the rows. I find the speed to be acceptable and I return Entities with roughly 20 properties. This approach should minimize the concurrent memory use on the server.