New cardinality estimator (SQL Server 2014) is way off

Question

I have a data warehouse database and I\'m facing problems with the new cardinality estimator of SQL Server 2014.

After upgrading the database server to SQL Server 2014 I

Answer 1

I wonder if you are running into this issue around multicolumn selectivity estimates:

http://www.sqlskills.com/blogs/kimberly/multi-column-statistics-exponential-backoff/

it seems that there are still some quirks with the new CE try also using TF 4137 as outlined and see if that helps.

finally make sure you are on the latest CU and are running with TF 4199 to blanket enable all query optimizer fixes as always test this in a non-production environment if possible first and be mindful of regressions in other queries when enable settings globally

Answer 2

This is not a direct answer to this question but it might help those who are facing similar performance issue related to that SCCM (aka ConfigMgr) database pertaining to Cardinality Estimator (CE) changes. SQL queries can timeout or your ConfigMgr console can run slow because of the new Cardinality Estimator (CE) changes in SQL Server 2014 and SQL Server 2016. Microsoft has given a solution to this problem here which suggests to apply an appropriate SQL Cardinality Estimator (CE) compatibility level as shown in the table below:

SQL Server version    Supported compatibility       Recommended compatibility   
                      level values                  level for ConfigMgr

SQL Server 2016       130, 120, 110, 100            130 

SQL Server 2014       120, 110, 100                 110

Hope this helps!

Answer 3

I think there is no simple answer today to this interesting question. The best answer I know is the following video: http://channel9.msdn.com/events/TechEd/NorthAmerica/2014/DBI-B331#fbid=. It has numerous examples of new and old estimators. The video is about 50+ mins long but it is worth the time.

A summary of the video that relates to this question:

Old assumptions of cardinality estimates:

1. Uniformity – data is uniformly distributed.
2. Independence – column 1 has no relation to column 2.
3. Containment – when two attributes might be the same, they are assumed to be the same.
4. Inclusion – there should be a match.

To Use SQL SERVER 2012 cardinality estimator in SQL SERVER 2014 use the following option:

• Option (querytraceon 9481) --revert to 2012

What is new estimator doing (based on video):

• SQL Server uses average selectivity in the index and estimates number of rows by multiplying density of the key to total numbers of rows in the index.
• New estimator does not work very well with jagged distributions.
• Most differences between the estimators are based on WHERE clause.
• New cardinality estimator believes there is a correlation between the tables.
• You can create filtered statistics to improve queries. (http://msdn.microsoft.com/en-us/library/ms188038.aspx )

To do / checklist:

1. Auto Create / Update Stats
2.  Check database compatibility mode (120/110)
3.  Test using query trace flags
4.  XML showplan

Update What's new in cardinality estimator (SQL Server 2016)

1. The more accurate.
2. The CE predicts how many rows your query will likely return
3. SQL Server 2016 the query store
4. Another option for tracking the cardinality predictions of the CE is to use the extended event named query_optimizer_estimate_cardinality
5. CE understands maximum value might be higher than when statistics were last gathered
6. CE understands that filtered predicates on the same table are often correlated
7. CE no longer assumes any correlation between filtered predicates from different tables

More details:

https://docs.microsoft.com/en-us/sql/relational-databases/performance/cardinality-estimation-sql-server

https://www.sqlshack.com/query-optimizer-changes-in-sql-server-2016-explained/