问题
We are using Nexus Repository version 3.2.0-01 and started getting the following error in nexus.log. OrientDB has got corrupted. Nexus is no longer starting.
2017-03-21 13:00:36,329+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - OrientDB version: 2.2.13
2017-03-21 13:00:36,348+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Server v2.2.13 is starting up...
2017-03-21 13:00:36,355+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Databases directory: /c9/setup/apps/nexus/sonatype-work/nexus3/db
2017-03-21 13:00:36,647+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies - Not enough physical memory available for DISKCACHE: 1,873MB (heap=1,161MB direct=2,048MB). Set lower Maximum Heap (-Xmx setting on JVM) and restart OrientDB. Now running with DISKCACHE=256MB
2017-03-21 13:00:36,648+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies - OrientDB config DISKCACHE=256MB (heap=1,161MB direct=2,048MB os=1,873MB)
2017-03-21 13:00:36,735+0000 WARN [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - Found ORIENTDB_ROOT_PASSWORD variable, using this value as root's password
2017-03-21 13:00:36,899+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.handler.OJMXPlugin - JMX plugin installed and active:
profilerManaged=true
2017-03-21 13:00:36,901+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - OrientDB Studio available at $ANSI{blue
http://localhost:2480/studio/index.html}
2017-03-21 13:00:36,902+0000 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.OServer - $ANSI{green:italic OrientDB Server is active} v2.2.13.
2017-03-21 13:00:36,902+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Activated
2017-03-21 13:00:36,904+0000 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
2017-03-21 13:00:37,632+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.orient.DatabaseInstanceImpl - Lifecycle operation start failed
com.orientechnologies.orient.core.exception.OSecurityAccessException: User or password not valid for database: 'config'^M
DB name="config"
at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
at com.orientechnologies.orient.core.metadata.security.OSecurityShared.authenticate(OSecurityShared.java:237) [na:na]
at com.orientechnologies.orient.core.metadata.security.OSecurityExternal.authenticate(OSecurityExternal.java:62) [na:na]
at com.orientechnologies.orient.core.metadata.security.OSecurityProxy.authenticate(OSecurityProxy.java:107) [na:na]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:274) [na:na]
at org.sonatype.nexus.orient.DatabaseManagerSupport.connect(DatabaseManagerSupport.java:142) [org.sonatype.nexus.orient:3.2.0.01]
at org.sonatype.nexus.orient.DatabaseInstanceImpl.doStart(DatabaseInstanceImpl.java:56) [na:na]
at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
at org.sonatype.goodies.lifecycle.Lifecycles.start(Lifecycles.java:44) [org.sonatype.goodies.lifecycle:2.2.3]
at org.sonatype.nexus.orient.DatabaseManagerSupport.createInstance(DatabaseManagerSupport.java:258) [org.sonatype.nexus.orient:3.2.0.01]
at org.sonatype.nexus.orient.DatabaseManagerSupport.instance(DatabaseManagerSupport.java:232) [org.sonatype.nexus.orient:3.2.0.01]
at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:61) [org.sonatype.nexus.base:3.2.0.01]
at org.sonatype.nexus.internal.orient.ConfigDatabase$ProviderImpl.get(ConfigDatabase.java:1) [org.sonatype.nexus.base:3.2.0.01]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:133) [com.google.inject:4.1.0]
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [com.google.inject:4.1.0]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [com.google.inject:4.1.0]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
at org.eclipse.sisu.inject.LazyBeanEntry.getValue(LazyBeanEntry.java:81) [org.eclipse.sisu.inject:0.3.2]
at org.eclipse.sisu.wire.BeanProviders.firstOf(BeanProviders.java:179) [org.eclipse.sisu.inject:0.3.2]
at org.eclipse.sisu.wire.PlaceholderBeanProvider.lookup(PlaceholderBeanProvider.java:122) [org.eclipse.sisu.inject:0.3.2]
at org.eclipse.sisu.wire.PlaceholderBeanProvider.get(PlaceholderBeanProvider.java:90) [org.eclipse.sisu.inject:0.3.2]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81) [com.google.inject:4.1.0]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderInternalFactory$1.call(ProviderInternalFactory.java:65) [com.google.inject:4.1.0]
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:115) [com.google.inject:4.1.0]
at org.eclipse.sisu.bean.BeanScheduler$Activator.onProvision(BeanScheduler.java:176) [org.eclipse.sisu.inject:0.3.2]
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:126) [com.google.inject:4.1.0]
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:68) [com.google.inject:4.1.0]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:63) [com.google.inject:4.1.0]
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [com.google.inject:4.1.0]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [com.google.inject:4.1.0]
at org.sonatype.nexus.upgrade.internal.ModelVersionStore.doStart(ModelVersionStore.java:82) [org.sonatype.nexus.upgrade:3.2.0.01]
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [org.sonatype.nexus.common:3.2.0.01]
at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.CGLIB$start$5(<generated>) [3.2.0-01:na]
at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f$$FastClassByGuice$$94ba58c4.invoke(<generated>) [3.2.0-01:na]
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [com.google.inject:4.1.0]
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [com.google.inject:4.1.0]
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [org.sonatype.nexus.common:3.2.0.01]
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [org.sonatype.nexus.common:3.2.0.01]
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [org.sonatype.nexus.common:3.2.0.01]
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [com.google.inject:4.1.0]
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [com.google.inject:4.1.0]
at org.sonatype.nexus.upgrade.internal.ModelVersionStore$$EnhancerByGuice$$b30c182f.start(<generated>) [3.2.0-01:na]
at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:80) [org.sonatype.nexus.upgrade:3.2.0.01]
at org.sonatype.goodies.lifecycle.LifecycleSupport.start(LifecycleSupport.java:104) [org.sonatype.goodies.lifecycle:2.2.3]
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:154) [org.sonatype.nexus.extender:3.2.0.01]
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:94) [org.sonatype.nexus.extender:3.2.0.01]
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:185) [org.sonatype.nexus.extender:3.2.0.01]
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [org.apache.felix.framework-5.4.0.jar:na]
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.4.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]Root cause Root cause of this was found to be an unclean shutdown of Nexus because of disk space getting full, this resulted in DB corruption.
2017-03-17 07:09:49,480+0000 WARN [elasticsearch[A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4][local_transport][T#1]] *SYSTEM org.elasticsearch.cluster.action.shard - [A82A95AC-5D396A2B-337B8736-5017E8FC-915232E4] [cab22f445c1892acef64d17ac2da6d0086023e21][0] received shard failed for [cab22f445c1892acef64d17ac2da6d0086023e21][0], node[HBjARpI7RyqqqruVqEJ8Ig], [P], v[11], s[INITIALIZING], a[id=FmxOHS7nQcGNz_93VlTpKg], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-03-17T07:09:49.101Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: IndexFailedEngineException[Index failed for [component#36e3dec8de528c9b41c5c6ac3b18e3f7]]; nested: NotSerializableExceptionWrapper[No space left on device]; ]], indexUUID [fLN4hdE8QPKH7Pj5RpLtmw], message [engine failure, reason [index]], failure [NotSerializableExceptionWrapper[No space left on device]]
org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) [na:1.8.0_121]
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) [na:1.8.0_121]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) [na:1.8.0_121]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) [na:1.8.0_121]
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) [na:1.8.0_121]
at org.elasticsearch.common.io.Channels.writeToChannel(Channels.java:211) [na:na]
at org.elasticsearch.common.bytes.PagedBytesReference.writeTo(PagedBytesReference.java:132) [na:na]
at org.elasticsearch.index.translog.BufferingTranslogWriter.add(BufferingTranslogWriter.java:57) [na:na]No backup We have not backed up Nexus VM, therefore we wont be able to restore it back. What is the possible way out - recreating the database - with least damage. Appreciate any help.
回答1:
Just in case that actually matters for somebody else - the above did not help me, but the json export / import one.
In my case nexus runs in the official docker image, so
docker exec -it nexus bash
Or however you access your nexus shell
cd /tmp
/opt/jdk1.8.0_141/bin/java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar
CONNECT plocal:/nexus-data/db/component admin admin
export database component-export
drop database
create database plocal:/nexus-data/db/component
import database component-export.json.gz
In your case the java-path could vary as also the data-path (nexus data) which we use to connect/create the database.
After that, restarting the service and your component database should be up and running again
回答2:
We had similar issue with Nexus OSS verison 3.2.0-01 and OrientDB corrupted state because of unclean shutdown.
In our case the coruppted storage was component located in
$install-dir/sonatype-work/nexus3/db/component
2017-04-14 15:09:20,007+0200 ERROR [qtp18266400-74] *UNKNOWN com.google.common.eventbus.EventBus.reentrant - Could not dispatch event AssetUpdatedEvent{metadata=AttachedEntityMetadata{schema=asset, document=asset#18:91856{bucket:#13:3,format:maven2,last_updated:Fri Apr 14 15:09:19 CEST 2017,attributes:[5],component:null,name:org/apache/maven/plugins/maven-jar-plugin/maven-metadata.xml,size:743,content_type:application/xml,blob_ref:default@ECF4439E-D7F184F8-578BAB14-ACFC0FB8-5589B163:3fb95745-4aed-460f-b711-b5f5b74b837c,last_accessed:Fri Apr 14 15:09:19 CEST 2017} v466}, remoteNodeId=null} to subscriber org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$a88d7327@83eca7 method [public void org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.on(org.sonatype.nexus.repository.storage.AssetEvent)]
com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
DB name="component"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.makeRollback(OAbstractPaginatedStorage.java:2210) [na:na]
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1433) [na:na]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:539) [com.orientechnologies.orientdb-core:2.2.13]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:104) [com.orientechnologies.orientdb-core:2.2.13]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2756) [com.orientechnologies.orientdb-core:2.2.13]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2725) [com.orientechnologies.orientdb-core:2.2.13]
at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:172) [org.sonatype.nexus.repository:3.2.0.01]
at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source) [na:na]
We managed to solve our problem with the following strategy
Scheduling Nexus tasks, particullary:
- Rebuild Maven repository metadata.
- Rebuild repository index.
- Compact blob store.
Orient DB manipulation:
- Rebuilding indexes.
- Repairing database.
According Nexus support you can access Orient DB console with Nexus OSS verison 3.2.1-01. Everything is explained here. Make sure your Nexus application is properly shut down before you apply any operations on database.
Once in Orient DB console you connect to local database with
> CONNECT PLOCAL:${your_install_dir}/sonatype-work/nexus3/db/${database} admin admin
And you perform the following operations:
> REBUILD INDEX *
> REPAIR DATABASE --fix-graph
> REPAIR DATABASE --fix-links
> REPAIR DATABASE --fix-ridbags
> REPAIR DATABASE --fix-bonsai
> DISCONNECT
I am absolutely aware my answer probably won't solve your concrete problem but I still wanted to give you some directions because you don't have any backups.
回答3:
The database write ahead logs are corrupt.
Make a backup of $install-dir/sonatype-work/nexus3/db/component, and then remove files with extension .wal from that directory. Then try the startup again
if the issue still persist, try clearing all .wal files under the "DB" directory, will resolve the issue,
I did tried this method and it worked like charm.
regards vinay
回答4:
With Nexus 3.16 there is now a task: Create Repair - Reconcile component database from blob store
super slow, but it helps recreating a accidentally deleted blobstore still on disk.
来源：https://stackoverflow.com/questions/42951710/orientdb-corruption-state-in-nexus-repository-version-3-2-0-01