Select Count(*) over large amount of data

Question

I want to do this for a Report but i have 20,000,000 of records in my table and it causes an TimeOut in my application.

SELECT
        T.transactionStatusID,
        TS.shortName AS TransactionStatusDefShortName,
        count(*) AS qtyTransactions
    FROM
        Transactions T 

    INNER JOIN TransactionTypesCurrencies TTC
                ON  T.id_Ent = TTC.id_Ent
                    AND T.trnTypeCurrencyID = TTC.trnTypeCurrencyID
            INNER JOIN TransactionStatusDef TS
                ON  T.id_Ent = TS.ent_Ent
                AND T.transactionStatusID = TS.ID
WHERE
    T.id_Ent = @id_Ent
GROUP BY
        T.transactionStatusID,
        TS.shortName

as far as i know COUNT(*) causes a full table scan and it makes my query to take too much time, im Using MS SQL 2005, any help ?

Edit:

The project leader tells me that the Query is just for one day it could help?

Answer 1

as far as i know COUNT(*) causes a full table scan and it makes my query to take too much time, im Using MS SQL 2005, any help ?

COUNT(*) can use any source that is able to give the answer, this includes indexes.

In your very case, I'd create a covering index on (id_ent, transactionStatusID) with trnTypeCurrencyID:

CREATE INDEX ON Transactions (id_ent, transactionStatusID) INCLUDE (trnTypeCurrencyID)

and rewrite the query a little:

SELECT  transactionStatusID, qtyTransactions, TS.shortName
FROM    (
        SELECT  T.transactionStatusID,
                COUNT(*) AS qtyTransactions
        FROM    Transactions T
        JOIN    TransactionTypesCurrencies TTC
        ON      TTC.id_Ent = T.id_Ent
                AND TTC.trnTypeCurrencyID = T.trnTypeCurrencyID
        WHERE   T.id_Ent = @id_Ent
        GROUP BY
                T.transactionStatusID
        ) TD
JOIN    TransactionStatusDef TS
ON      TS.ent_Ent = @id_Ent
        AND TS.ID = TD.transactionStatusID

The index will filter on id_ent and parallelize on transactionStatusID. Since you have trnTypeCurrencyID covered, the engine will not have to lookup the value in the table, it's already present in the index.

The GROUP BY clause also includes only the columns from the index so it parallelizes much better.

Update:

By adding WITH (ONLINE = ON) you can leave the table operational for the time the index is being created:

CREATE INDEX ON Transactions (id_ent, transactionStatusID) INCLUDE (trnTypeCurrencyID) WITH (ONLINE = ON)

Answer 2

If you look at the execution plan for the query, that will highlight the bits that are performing badly. It will tell you whether it's doing a table scan, index scan or index seek. So that's the best place to start looking.

Do you have any indexes at the moment? The fields involved in the JOINs and WHERE clause are prime candidates - if you don't have indexes, that'll be a major factor.

Answer 3

Have you tried

COUNT(1)

Instead?

Also, is the join to TransactionTypesCurrencies required, does not seem that you use anything from it?

Answer 4

What is the clustered index on the transactions table? What other indexes exist? You could try this query to eliminate one join:

SELECT
    T.TransactionStatusID,
    TS.ShortName,
    qtyTransactions = COUNT(*)
FROM
    dbo.Transactions AS T
INNER JOIN
    dbo.TransactionStatusDef AS TS
    ON T.id_Ent = TS.ent_Ent
    AND T.transactionStatusID = TS.ID
WHERE EXISTS
(
    SELECT 1
        FROM do.TransactionTypeCurrencies AS TTC
        WHERE TTC.id_Ent = T.id_Ent
        AND TTC.trnTypeCurrencyID = T.trnTypeCurrencyID
)
AND T.id_Ent = @id_Ent
GROUP BY
    T.transactionStatusID,
    TS.shortName;

You could also try to use snapshot isolation before issuing the query, e.g.

SET TRANSACTION ISOLATION LEVEL SNAPSHOT;

To do this you must have:

ALTER DATABASE dbname SET ALLOW_SNAPSHOT_ISOLATION ON;

In general though you want to make sure you get the indexing right. If you can't apply proper indexing on the tables for this query because it will hurt other queries, then you can consider an indexed view to maintain this count for you (at the cost of insert/update performance), or table partitioning if you are on Enterprise Edition, or occasionally running a rollup of this data in the background so that your application doesn't have to wait for it (assuming it is ok that the actual count is a bit stale).