How to handle 5 million users? ASP.NET Identity

Question

I am running a ASP.NET mvc5 app which currently has 5 million users. It is hosted in the Azure cloud. For the authentication I use the Asp.Net Identity for EntityFramework.

Answer 1

I think I have solution for you. I have made another tests and the second option - index on computed column works. Here are steps in sql code, you can probably do the same using EF annotations.

1. Create computed column on table users as upper(username):

   alter table users add upper_username as upper(username)

2. Create index on that column:

   create index ix2 on a_upload(upper_username)

Thats all. The EF select will still use UPPER, but MS SQL optimizer should be able to use this index as it has the same definition as the function in where clause.

Here are test results on my PC:

test sql: select field001 from a_upload where upper(field001)='10'

BEFORE (SCAN means that the engine has to read all records one by one)

AFTER CREATING THE INDEX on functional column (SEEK=engine will utilize index)

Dont get confuzed that even in the BEFORE scenario, sql engine is using index (ix1). It is only because I am selecting only "field001" and the optimizer knows, that it is contained not only in the table but in the index too. And index has less bytes than whole table. But it does not mean that the system utilized index, it has to compute upper() for every row on each select anyway.

Answer 2

Yes, this query is probably the problem. This is why I don't like SQL frameworks.

The problem is here:

WHERE 
((UPPER([Extent3].[UserName])) = (UPPER(@p__linq__0))) 

The right side is constant and the left side "upper(username)" is SQL engine trying to find somewhere. Index on simple "username" is useless, because it contains simple "username" and not upper(username) values. SQL engine has to calculate upper(username) for all 5 millions lines EVERYTIME you add new user and then find the corresponding value. 7 seconds...

You need to create such index:

create index ix_name on Users(upper(UserName))

Such index will be used automatically by sql engine. But I am afraid, that MS SQL will not allow you to do it without adding computed column (and it is useless for your purpose, because you can not force EF to use another column for username value).

The other option is to force Identity framework not to use UPPER function. But I don't know how.

By the way, I am really surprised to find this type of code in such important package - entity framework. This "searching functional value in index" is on of the basic mistakes when writing efficient sql code.

And one more comment: This everything should have been done by the framework, it is responsible for correct handling of "username" column. It is a bug, this advice is only workaround.

Answer 3

Try to get the exact SQL command from some Azure audit tool and write it to the comment here. 7 second looks like problem in indexes (too long for access by index ant too short for transferring huge data to client).

INSERT command itself is always quick (if not slowed down by triggers) and does not get slower significantly with increasing number of rows. Identity framework is probably running some SELECT (e.g. to re-read new line...?). Having indexes on all affected columns is not enough. Select command can use only ONE index on each table, so you need to find only ONE index, but the proper one - the correct combination of columns, column order in index is also important. And all this depends on the SQL command, that is the Identity engine trying to execute. Try to find it, then I can help you.

Answer 4

While I had to replace the userId guid with an int, I created a CustomUserStore. In this UserStore you can simply overwrite the FindByNameAsync:

public class CustomUserStore<TUser> : UserStore<TUser, CustomRole, int, CustomUserLogin, CustomUserRole, CustomUserClaim> where TUser : MyUser
{
    public CustomUserStore(MyDbContext context) : base(context) { }

    public override Task<TUser> FindByNameAsync(string userName)
    {
        return this.GetUserAggregateAsync(u => u.UserName.Equals(userName, StringComparison.InvariantCultureIgnoreCase));
    }
}

This results in a query without UPPER().