Spring Data “TimeoutException” when calling delete documents (with pagination) in Couchbase

北城以北 提交于 2020-02-25 04:26:18

问题


Our Spring Boot app is using a Couchbase DB and accessing it using Spring-Data

To delete records from the bucket we created the following method in the repository:

Slice<Dog> deleteAllByOwnerIdAndName(String ownerId, String name, Pageable pageable);

We also have relevant index on the bucket:

CREATE INDEX `dogs_by_ownerId_and_name_idx` ON `dogs`(`ownerId`,`name`) WHERE (`_class` = "com.example.Dog")

Our code is using the pagination when trying to delete the elements:

 Slice<Dog> dogsSlice = null;
 Pageable pageable = PageRequest.of(0, 1000, Sort.by("id"));
 int pageCounter = 0;
 do {
   log.debug("Deleting page No. {} of dogs", pageCounter++);
   dogsSlice = dogsRepository.deleteAllByOwnerIdAndName("3243242", "Max", pageable);
 } while (dogsSlice.hasNext());

But, too many times we get a Timeoutexceptioin:

Deleting page No. 0 of dogs

o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task.

org.springframework.data.couchbase.core.CouchbaseQueryExecutionException: Unable to execute query due to the following n1ql errors: {"msg":"Timeout 7.5s exceeded","code":1080} at org.springframework.data.couchbase.core.CouchbaseTemplate.findByN1QL(CouchbaseTemplate.java:458) ~[classes!/:5.1.40] at org.springframework.data.couchbase.repository.query.AbstractN1qlBasedQuery.executeSliced(AbstractN1qlBasedQuery.java:189) ~[classes!/:5.1.40] at org.springframework.data.couchbase.repository.query.AbstractN1qlBasedQuery.executeDependingOnType(AbstractN1qlBasedQuery.java:129) ~[classes!/:5.1.40] at org.springframework.data.couchbase.repository.query.AbstractN1qlBasedQuery.execute(AbstractN1qlBasedQuery.java:106) ~[classes!/:5.1.40] at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:590) ~[classes!/:5.1.40] at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.data.couchbase.repository.support.ViewPostProcessor$ViewInterceptor.invoke(ViewPostProcessor.java:87) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61) ~[classes!/:5.1.40] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[classes!/:5.1.40] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[classes!/:5.1.40] at com.sun.proxy.$Proxy130.deleteAllByOwnerIdAndName(Unknown Source) ~[na:na]

Is there anything else we need to do?


回答1:


You can improve a few things:

1) Change your index to sort by ownerId, then while deleting, sort also by ownerId

CREATE INDEX `dogs_by_ownerId_and_name_idx` ON `dogs`(`ownerId` ASC,`name`) WHERE (`_class` = "com.example.Dog") 

As your index is already sorted, couchbase won't spend extra time sorting it during the delete.

2) Do you really need to return all deleted objects? Couchbase has to bring all attributes which are not in your index before sending the documents back to you, and this operation will take some extra time. The best approach is to return only the ids.

@Override
public void updateFamilyName(String familyName, String familyId) {

    String queryString = "Delete from "+getBucketName()+" WHERE "+getClassFilter()+" " +
            " and familyId = '"+familyId+"' RETURNING meta().id";

    N1qlParams params = N1qlParams.build().consistency(ScanConsistency.REQUEST_PLUS).adhoc(true);
    ParameterizedN1qlQuery query = N1qlQuery.parameterized(queryString, JsonObject.create(), params);
    checklistRepository.getCouchbaseOperations().getCouchbaseBucket().query(query);
}

private String getBucketName(){
    return checklistRepository.getCouchbaseOperations().getCouchbaseBucket().bucketManager().info().name();
}

private String getClassFilter(){
    return "_class = '" + Checklist.class.getName() + "' ";
}

3) You could also improve on your pagination, but I don't think it will be necessary in your case.

https://blog.couchbase.com/offset-keyset-pagination-n1ql-query-couchbase/




回答2:


If query has ORDER BY, If possible optimizer try to use index order. If not possible it must produce all possible data set and sort the data to satisfies query even pagination needs few items.

Check Rule #7 https://blog.couchbase.com/create-right-index-get-right-performance/

Also see this article https://blog.couchbase.com/offset-keyset-pagination-n1ql-query-couchbase/




回答3:


The Sort.by("id") is causing the delay in the query, as it seems like Couchbase is sorting the entire collection of documents by that criteria.

So if there is not real need in sorting the results, it is better to use

Pageable pageable = PageRequest.of(0, 1000);



来源:https://stackoverflow.com/questions/53362185/spring-data-timeoutexception-when-calling-delete-documents-with-pagination-i

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!