问题
I deployed 3 Kaa clusters follow the kaa Cluster setup guide document. The databases are Cassandra and MariaDB . When I run C Client(I make a code change that every client can login in three kaa servers in random) . a lot of questions are reported in the Server log file. I will upload the one Server logs in Kaa Cluster . I summed up a bit,there are 7 errors.The first error will appear when C Clients login in kaa server firstly.After I stop C Clients and run those secondly,the error didn't appear. I think that The second error is caused by Database cluster, maybe is Database consistency . Because I made a limit that use 10 MB HUB connected to two Cassandra , the error often appears.I try to figure out the other errors and have no idea about those now.
1 -------------------
2016-12-09 00:00:43,950 [EPS-endpoint-dispatcher-597] ERROR o.k.k.s.c.n.c.d.AbstractVersionableCassandraDao - [class org.kaaproject.kaa.server.common.nosql.cassandra.dao.model.CassandraEndpointProfile] Can't update entity with version 2. Entity already changed!
2016-12-09 00:00:43,952 [EPS-endpoint-dispatcher-597] WARN o.k.k.s.c.d.s.EndpointServiceImpl - Optimistic lock detected in endpoint profile
org.kaaproject.kaa.server.common.dao.exception.KaaOptimisticLockingFailureException: Can't update entity with version 2. Entity already changed!
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.updateLocked(AbstractVersionableCassandraDao.java:100) ~[cassandra-dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.save(AbstractVersionableCassandraDao.java:59) ~[cassandra-dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.updateProfile(EndpointProfileCassandraDao.java:168) ~[cassandra-dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:129) ~[cassandra-dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:121) ~[cassandra-dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:90) ~[cassandra-dao-0.10.0.jar:na]
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at com.sun.proxy.$Proxy64.save(Unknown Source) ~[na:na]
at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.saveEndpointProfile(EndpointServiceImpl.java:382) ~[dao-0.10.0.jar:na]
at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.attachEndpointToUser(EndpointServiceImpl.java:421) ~[dao-0.10.0.jar:na]
at sun.reflect.GeneratedMethodAccessor180.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:121) [spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at com.sun.proxy.$Proxy70.attachEndpointToUser(Unknown Source) [na:na]
at org.kaaproject.kaa.server.operations.service.user.DefaultEndpointUserService.attachEndpointToUser(DefaultEndpointUserService.java:93) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.attachEndpointToUser(DefaultOperationsService.java:788) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processUserVerificationMessage(LocalEndpointActorMessageProcessor.java:1025) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processUserVerificationMessage(LocalEndpointActor.java:123) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:88) [kaa-node-0.10.0.jar:na]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]
2 -------------------
2016-12-09 00:17:12,473 [EPS-endpoint-dispatcher-586] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [R18PpFaouxXSrHQVbOfyTtiGKBo=][cd72eab3ccec4135a6ef4f553add7bb6] processEndpointRequest
java.security.InvalidParameterException: Cache key can't be null
at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.compute(CacheTemporaryMemorizer.java:77) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.cache.concurrent.ConcurrentCacheService.getConfByHash(ConcurrentCacheService.java:581) ~[kaa-node-0.10.0.jar:na]
at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:320) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:353) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:302) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]
at com.sun.proxy.$Proxy96.getConfByHash(Unknown Source) ~[na:na]
at org.kaaproject.kaa.server.operations.service.delta.DefaultDeltaService.getDelta(DefaultDeltaService.java:155) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.calculateConfigurationDelta(DefaultOperationsService.java:675) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.syncConfiguration(DefaultOperationsService.java:426) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:394) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:310) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processEndpointSync(LocalEndpointActorMessageProcessor.java:135) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processEndpointSync(LocalEndpointActor.java:131) [kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:78) [kaa-node-0.10.0.jar:na]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]
at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]
3 -----------------
2016-12-09 00:32:35,091 [EPS-core-dispatcher-1174] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null
java.lang.NullPointerException: null
2016-12-09 00:32:35,092 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null
java.lang.NullPointerException: null
2016-12-09 00:32:35,097 [EPS-endpoint-dispatcher-586] INFO o.k.k.s.o.s.u.DefaultEndpointUserService - Can't find listeners for unassigned endpoint!
4 --------------------
2016-12-09 01:15:46,710 [pool-22-thread-1] ERROR o.k.k.s.thrift.NeighborConnection - EventWorker [192.168.10.14:9090:OPERATIONS_SERVICE:d9c71186-e45e-4c33-88b0-0ad5f26c6d17] error sending event messages pack.
com.twitter.common.thrift.TResourceExhaustedException: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:91) ~[thrift-0.0.64.jar:na]
at com.twitter.common.thrift.Thrift$1.invoke(Thrift.java:363) ~[thrift-0.0.64.jar:na]
at com.sun.proxy.$Proxy141.sendMessages(Unknown Source) ~[na:na]
at org.kaaproject.kaa.server.operations.service.event.DefaultEventService$MessageTemplate.process(DefaultEventService.java:727) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.thrift.NeighborConnection$EventWorker.run(NeighborConnection.java:322) ~[server-shared-0.10.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
at com.twitter.common.thrift.ThriftFactory$ManagedThreadPool$1.rejectedExecution(ThriftFactory.java:237) ~[thrift-0.0.64.jar:na]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_111]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[na:1.8.0_111]
at com.twitter.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:66) ~[util-0.0.88.jar:na]
at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:71) ~[thrift-0.0.64.jar:na]
... 9 common frames omitted
5 ---------------------
2016-12-09 02:15:50,233 [EPS-endpoint-dispatcher-598] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [Nje8C3i2b6K7ukh7tP8G/pDzdrE=][3f5e75f03bd34c30be183807486176ba] processEndpointRequest
java.lang.NullPointerException: null
6 -------------------------
2016-12-09 00:53:05,649 [EPS-core-dispatcher-1117] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null
java.lang.NullPointerException: null
2016-12-09 00:53:05,650 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null
java.lang.NullPointerException: null
7 ------------------------
2016-12-09 09:20:33,680 [qtp1981719029-883] ERROR o.k.k.s.admin.servlet.SdkServlet - Unexpected error in SdkServlet.doGet:
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:402) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:577) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:548) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:488) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) ~[jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:606) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:535) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_111]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_111]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_111]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_111]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_111]
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]
... 9 common frames omitted
回答1:
Seems this is related to KAA-1643 bug. It's already fixed, but not released (will be in 0.10.1). You can try to build last version with that fix.
来源：https://stackoverflow.com/questions/41096922/the-issues-that-are-caused-by-kaa-cluster