How to make Filter Aggregation inside Bucket Aggregation?

问题

The bounty expires in 15 hours. Answers to this question are eligible for a +50 reputation bounty. priyanka.sarkar is looking for an answer from a reputable source.

I have the below requirement. I have some records which looks as under (as an example)

agreementid = 1, lastdispositioncode = PTP , feedbackdate = 30/11/2020 
agreementid = 1, lastdispositioncode = PTP  , feedbackdate = 29/11/2020
agreementid = 1, lastdispositioncode = BPTP , feedbackdate = 21/11/2020
agreementid = 2, lastdispositioncode = BPTP  , feedbackdate = 29/11/2020
agreementid = 2, lastdispositioncode = BPTP , feedbackdate = 11/11/2020
agreementid = 3, lastdispositioncode = SBPTP  , feedbackdate = 24/11/2020

Here , first we have to

• first group on agreementid,
• then sort on feedbackdate to get the latest record
• and after that have to count the number of lastdispositioncode.

The above data will give the below result

ptp = 1 (since latest record of PTP is on 30/11/2020 for AggrementID = 1)
bptp = 3 (since 

                for AggrementID = 1, latest record of BPTP is on 21/11/2020
                for AggrementID = 2, latest record of BPTP is on 29/11/2020
                for AggrementID = 3, latest record of SBPTP is on 24/11/2020)
            

I have attempted the below program but the query is failing.

class Program
{
    static void Main(string[] args)
    {
        var ptpDispositionCodes = new TermsQuery
        {
            IsVerbatim = true,
            Field = "lastdispositioncode",
            Terms = new string[] { "PTP" },
        };

        var bptpDispositionCodes = new TermsQuery
        {
            IsVerbatim = true,
            Field = "lastdispositioncode",
            Terms = new string[] { "BPTP","SBPTP" },
        };
        
        ISearchResponse<TestReportModel> searchResponse =
            ConnectionToES.EsClient()
            .Search<TestReportModel>
            (s => s
                .Index("feedbackdata")
                .From(0)
                .Size(50000)
                .Query(q =>q.MatchAll())
                .Aggregations(a => a
                    .Terms("Agreement_ID", t => t
                        .Field(f => f.agreementid.Suffix("keyword"))
                        .Aggregations(aa => aa
                            .TopHits("latest_feedbackdate_sort", th => th.Sort(so => so.Descending(f => f.feedbackdate))
                            
                                .Size(1) //get the latest record 
                            )

                        )
                        .Aggregations(fa => fa

                            .Filter("ptp_aggs", f => f.Filter(fd => ptpDispositionCodes))
                            .Filter("bptp_aggs", f => f.Filter(fd => bptpDispositionCodes))                         
                        )
                    )
                )
            );
        
        var ptpDocCount = ((Nest.SingleBucketAggregate)trailSearchResponse.Aggregations["ptp_aggs"]).DocCount;
        var bptpDocCount = ((Nest.SingleBucketAggregate)trailSearchResponse.Aggregations["bptp_aggs"]).DocCount;
    }
}

DTO

public class TestReportModel
{
    public string agreementid { get; set; }
    public string trailstatus { get; set; }        
    public string lastdispositioncode { get; set; }        
}

public class TestOutputAPIModel
{
    public List<TestModel> TestModelDetail { get; set; }    
}

public class TestModel
{   
    public string TrailStatus { get; set; }
    public int NoOfAccounts { get; set; }
    
}

Here is what the response I have received so far

As can be figured out that there are two problems

a) Aggregation on "agreementID" is happening only for the first 10 records (but there are many unique AgreementID's in the system say more than 5/6 lacs)

b) ptp_aggs / bptp_aggs are coming as zero where as there are matching Lastdispositioncodes for those.

What can I try next?

Edit

This is the raw query generated which I obtained from DebugInformation

ApiCall.RequestBodyInBytes

{"aggs":{"Agreement_ID":{"aggs":{"ptp_aggs":{"filter":{"terms":{"lastdispositioncode":["ptp"]}}},"bptp_aggs":{"filter":{"terms":{"lastdispositioncode":["bptp"]}}},"paid_aggs":{"filter":{"terms":{"lastdispositioncode":["clmpd","dib","exceemi","odp","partpaid","pr"]}}}},"terms":{"field":"agreementid.keyword"}}},"from":0,"query":{"match_all":{}},"size":50000}

ApiCall.ResponseBodyInBytes

"aggregations" : {
    "sterms#Agreement_ID" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 658,
      "buckets" : [
        {
          "key" : "2233585285",
          "doc_count" : 513,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 513
          }
        },
        {
          "key" : "ABCD123456",
          "doc_count" : 95,
          "filter#bptp_aggs" : {
            "doc_count" : 8
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 87
          }
        },
        {
          "key" : "8728441356",
          "doc_count" : 53,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 53
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "5385541121",
          "doc_count" : 39,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 39
          }
        },
        {
          "key" : "VEH001001",
          "doc_count" : 30,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "2671278479",
          "doc_count" : 29,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 29
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "4937845646",
          "doc_count" : 27,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "4472873928",
          "doc_count" : 25,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "LD1736377086",
          "doc_count" : 24,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 0
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        },
        {
          "key" : "5548762365",
          "doc_count" : 23,
          "filter#bptp_aggs" : {
            "doc_count" : 0
          },
          "filter#paid_aggs" : {
            "doc_count" : 23
          },
          "filter#ptp_aggs" : {
            "doc_count" : 0
          }
        }
      ]

来源：https://stackoverflow.com/questions/65073490/how-to-make-filter-aggregation-inside-bucket-aggregation