EF Core with ABP performance issue in huge loops

旧巷老猫 提交于 2019-12-11 13:15:24

问题


I have a background job that processes a list of items. However, the larger the list, the time to complete it grows exponentially. Not just that, the further down the list, the app gradually eats up my server's 64gb ram and slows down the server to a crawl.

I use Hangfire as my background job manager, and the job takes a list of id and performs multiple database actions on them. I access the database through ef core, using the repository pattern provided by aspnetboilerplate framework.

I assume it is slow because of the growing db context, so here is what I have tried:

  • Split the list across multiple jobs.

        var splitIdList = ListExtensions.ChunkBy(idList, 100);
        foreach (var split in splitIdList)
        {
            await _backgroundJobManager.EnqueueAsync<MyJob, MyJobArgs>(new MyJobArgs(...))
    
        }
    
  • Split the job into multiple unit of work

    var splitIdList = ListExtensions.ChunkBy(idList, 100);
    foreach (var split in splitIdList)
    {
        using (var uow = UnitOfWorkManager.Begin())
        {
             foreach(var id in split)
             {
                 //create related entity
                 ....
                 //create barcode
                 ....
                 //link entity to created barcode
                 ....
             }
             uow.Complete();
        }
    }
    
  • Split the job into multiple unit of work with new transactions

    var splitIdList = ListExtensions.ChunkBy(idList, 100);
    foreach (var split in splitIdList)
    {
        using (var uow = UnitOfWorkManager.Begin(System.Transactions.TransactionScopeOption.RequiresNew))
        {
             foreach(var id in split)
             {
                 //create related entity
                 ....
                 //create barcode
                 ....
                 //link entity to created barcode
                 ....
             }
             uow.Complete();
        }
    }
    

None seems to solve the problem, any help would be appreciated, thank you.

EDIT Here is what I do to each entity in the job:

  1. I create another entity and link them together.
  2. I create a qrcode, which is a running number, and link it to the entity

Operations involved are create, get, insert, and update

来源:https://stackoverflow.com/questions/52755973/ef-core-with-abp-performance-issue-in-huge-loops

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!