问题
I'm going crazy because of this exception when starting up JanusGraph. It happened between restarts of the gremlin-server, without even touching the configuration files.
This error always appears at the very first startup of gremlin-server.
This is the stack trace from the logs:
6911 [main] INFO org.janusgraph.diskstorage.log.kcvs.KCVSLog - Loaded unidentified ReadMarker start time 2019-08-21T17:57:53.794212900Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@51efb731
8148 [main] WARN org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker - Skipping outdated lock on KeyColumn [k=0x 16-165-160-103-105- 30- 71-114- 97-112-104- 95- 78- 97-109-101- 95- 73-110-100-101-248, c=0x 0] with our rid ( 48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49) but mismatched timestamp (actual ts 2019-08-21T17:57:54.934645Z, expected ts 2019-08-21T17:57:54.934645300Z)
8149 [main] ERROR org.janusgraph.graphdb.database.StandardJanusGraph - Could not commit transaction [1] due to storage exception in system-commit
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196)
at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:716)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1380)
at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:246)
at org.janusgraph.graphdb.management.ConfigurationManagementGraph.createIndexIfDoesNotExist(ConfigurationManagementGraph.java:311)
at org.janusgraph.graphdb.management.ConfigurationManagementGraph.<init>(ConfigurationManagementGraph.java:81)
at org.janusgraph.graphdb.management.JanusGraphManager.lambda$new$0(JanusGraphManager.java:77)
at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.janusgraph.graphdb.management.JanusGraphManager.<init>(JanusGraphManager.java:74)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Permanent locking failure
at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:359)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
... 20 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid 48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49 but mismatched timestamps; no lock column contained our timestamp (2019-08-21T17:57:54.934645300Z)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
... 27 more
8152 [main] ERROR org.janusgraph.graphdb.database.StandardJanusGraph - Could not commit transaction [1] due to exception
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196)
at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:716)
at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1380)
at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:246)
at org.janusgraph.graphdb.management.ConfigurationManagementGraph.createIndexIfDoesNotExist(ConfigurationManagementGraph.java:311)
at org.janusgraph.graphdb.management.ConfigurationManagementGraph.<init>(ConfigurationManagementGraph.java:81)
at org.janusgraph.graphdb.management.JanusGraphManager.lambda$new$0(JanusGraphManager.java:77)
at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.janusgraph.graphdb.management.JanusGraphManager.<init>(JanusGraphManager.java:74)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Permanent locking failure
at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:359)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154)
at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94)
at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
... 20 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid 48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49 but mismatched timestamps; no lock column contained our timestamp (2019-08-21T17:57:54.934645300Z)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
... 27 more
8153 [main] ERROR org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor - Could not invoke constructor on class org.janusgraph.graphdb.management.JanusGraphManager (defined by the 'graphManager' setting) with one argument of class Settings
I'm on windows 10, the version of JanusGraph I'm using is the 0.4.0 and it's configured to use Cassandra 3.11.0 and Elastisearch 6.7.2, both running on docker (while I manually start gremlin-server).
I've already tried cleaning everything from docker (containers, images and volumes). I also tried running 'bin/JanusGraph.sh clean-, but with no luck. Even downloading again the janusgraph-0.4.0-hadoop2.zip and reconfiguring it from start does nothing.
It goes without saying that if I try to startup the server without touching the original configuration files it works. (But without the ConfigurationManagementGraph)
I don't understand how it could break without touching anything.
This is the gremlin-server configuration:
host: 0.0.0.0
port: 8182
scriptEvaluationTimeout: 180000
# channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
channelizer: org.janusgraph.channelizers.JanusGraphWebSocketChannelizer
graphManager: org.janusgraph.graphdb.management.JanusGraphManager
graphs: {
#graph: conf/gremlin-server/janusgraph-cql-es-server.properties,
ConfigurationManagementGraph: conf/gremlin-server/janusgraph-cql-es-server-configured.properties,
}
scriptEngines: {
gremlin-groovy: {
plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/init.groovy]}}}}
serializers:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
# - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 }
# - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }}
# Older serialization versions for backwards compatibility:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: true, interval: 180000},
csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: true},
slf4jReporter: {enabled: true, interval: 180000},
gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
graphiteReporter: {enabled: false, interval: 180000}}
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferLowWaterMark: 32768
writeBufferHighWaterMark: 65536
This is the janus graph configuration (I removed the comments for brevity):
gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
graph.graphname=ConfigurationManagementGraph
storage.backend=cql
storage.hostname=127.0.0.1
storage.cql.keyspace=janusgraph
cache.db-cache = true
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
index.search.backend=elasticsearch
index.search.hostname=127.0.0.1
index.search.elasticsearch.client-only=true
This is the script gremlin server runs at startup:
def globals = [:]
def getGraph() {
def graphNames = ConfiguredGraphFactory.getGraphNames();
def graphMaps = [:];
for (graphName in graphNames) {
def g = ConfiguredGraphFactory.open(graphName);
graphMaps.put(graphName, g.traversal())
}
return graphMaps;
}
globals << getGraph()
In the docker-compose console I see no exceptions, the last thing before gremlin-server shows the error is the following line:
cassandra | INFO [MigrationStage:1] 2019-08-21 17:57:53,725 ColumnFamilyStore.java:430 - Initializing janusgraph.system_properties_lock_
Please has anyone any idea what might be causing this problem?
Thank you!
来源：https://stackoverflow.com/questions/57606708/janusgraphexception-could-not-execute-operation-due-to-backend-exception