New cardinality estimator (SQL Server 2014) is way off

Question

I have a data warehouse database and I\'m facing problems with the new cardinality estimator of SQL Server 2014.

After upgrading the database server to SQL Server 2014 I

Answer 1

I think there is no simple answer today to this interesting question. The best answer I know is the following video: http://channel9.msdn.com/events/TechEd/NorthAmerica/2014/DBI-B331#fbid=. It has numerous examples of new and old estimators. The video is about 50+ mins long but it is worth the time.

A summary of the video that relates to this question:

Old assumptions of cardinality estimates:

1. Uniformity – data is uniformly distributed.
2. Independence – column 1 has no relation to column 2.
3. Containment – when two attributes might be the same, they are assumed to be the same.
4. Inclusion – there should be a match.

To Use SQL SERVER 2012 cardinality estimator in SQL SERVER 2014 use the following option:

• Option (querytraceon 9481) --revert to 2012

What is new estimator doing (based on video):

• SQL Server uses average selectivity in the index and estimates number of rows by multiplying density of the key to total numbers of rows in the index.
• New estimator does not work very well with jagged distributions.
• Most differences between the estimators are based on WHERE clause.
• New cardinality estimator believes there is a correlation between the tables.
• You can create filtered statistics to improve queries. (http://msdn.microsoft.com/en-us/library/ms188038.aspx )

To do / checklist:

1. Auto Create / Update Stats
2.  Check database compatibility mode (120/110)
3.  Test using query trace flags
4.  XML showplan

Update What's new in cardinality estimator (SQL Server 2016)

1. The more accurate.
2. The CE predicts how many rows your query will likely return
3. SQL Server 2016 the query store
4. Another option for tracking the cardinality predictions of the CE is to use the extended event named query_optimizer_estimate_cardinality
5. CE understands maximum value might be higher than when statistics were last gathered
6. CE understands that filtered predicates on the same table are often correlated
7. CE no longer assumes any correlation between filtered predicates from different tables

More details:

https://docs.microsoft.com/en-us/sql/relational-databases/performance/cardinality-estimation-sql-server

https://www.sqlshack.com/query-optimizer-changes-in-sql-server-2016-explained/