Amazon DynamoDB Local - unknown error, exception or failure

后端未结

关注

 2  740

I have to test an application which relies heavily on Amazon\'s DynamoDB. I want the tests to be able to be run separately, which is why I opted for DynamoDB Local .ja


                      
              相关标签:


      
      
        
          2条回答        

        
                         				            
            
           
            
                              
                
              
              
                
                  执笔经年        
                
              
                            
                2020-12-06 17:15
              
            
            
                                                                       
There are at least 2 problems with DynamoDBLocal.  Resolve both, and you will be running embedded DynamoDB.

First, the -port parameter does not work correctly.  So Jetty isn't setup on the port you expect.  Instead something like port 51205 (or random?) is set as the Jetty listener as a default port.

Here is my code for how I start the server avoiding the built-in command-line parsing which is better anyway...   After starting the server in this way, http://localhost:19444/shell works, so Jetty is fine.  But then you might have another problem with Sqlite4java (see after the code block).

NOTE: code is Kotlin, but the Java would be very similar.  Also I have SLF4j configured and don't let the ServerRunner classes break the logging, so this is doubly better way to start the server and what ServerRunner does internally.



class TestAccountManager {
    companion object {
        private val localDbPort = 19444

        private lateinit var localDb: DynamoDBProxyServer
        private lateinit var dbClient: AmazonDynamoDBClient

        @BeforeClass @JvmStatic fun setup() {
            System.setProperty("org.eclipse.jetty.util.log.class", "org.eclipse.jetty.util.log.Slf4jLog")
            localDb = DynamoDBProxyServer(localDbPort, LocalDynamoDBServerHandler(
                    LocalDynamoDBRequestHandler(0, true, null, true, true), null)
            )
            localDb.start()

            val auth = BasicAWSCredentials("fakeKey", "fakeSecret")
            dbClient = AmazonDynamoDBClient(auth)
            dbClient.signerRegionOverride = "us-east-1"
            dbClient.setEndpoint("http://localhost:$localDbPort")
        }

        @AfterClass @JvmStatic fun teardown() {
           localDb.stop()
        }
    }

    @Test fun testSomething() {
        dbClient.listTables().tableNames.forEach {
            println(it)
        }
    }
}


Once you have this code, you now are running on the expected port.  

Now you likely have the 2nd error, such as Sqlite4java not finding its correct binary for your platform.  For some versions of Mac OSX it will generate a binary filename that actually does not exist.  And DynamoDBLocal hides all Sqlite4java logging forcibly (impossible to override it), so you will not see it.  

You can test the sqlite library by downloading a distribution, unzipping and then running:

java -jar sqlite4java-1.0.392.jar -d


And it will report what it try to load, and how it failed or not.  You just need to find that JAR on your system from wherever Gradle, Maven or you placed it.  My output with error was:

sqlite4java 392
160212:002049.833 FINE [sqlite] Internal: loading library
160212:002049.834 FINE [sqlite] Internal: java.library.path=/Users/jminard/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
160212:002049.834 FINE [sqlite] Internal: sqlite4java.library.path=null
160212:002049.834 FINE [sqlite] Internal: cwd=/Users/jminard/DEV/Collokia/repos/collokia-web-back/.
160212:002049.834 FINE [sqlite] Internal: default path=/Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d
160212:002049.834 FINE [sqlite] Internal: forced path=null
160212:002049.834 FINE [sqlite] Internal: os.name=mac os x; os=osx
160212:002049.835 FINE [sqlite] Internal: os.arch=x86_64
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-1.0.392.dylib
160212:002049.835 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d-1.0.392.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64.dylib
160212:002049.836 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-x86_64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-amd64-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-osx-d.dylib
160212:002049.837 FINE [sqlite] Internal: checking /Users/jminard/.gradle/caches/modules-2/files-2.1/com.almworks.sqlite4java/sqlite4java/1.0.392/d6234e08ff4e1607ff5321da2579571f05ff778d/libsqlite4java-d.dylib
160212:002049.837 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-1.0.392
160212:002049.838 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
160212:002049.838 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-1.0.392
160212:002049.839 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-1.0.392 in java.library.path
160212:002049.839 FINE [sqlite] Internal: trying to load sqlite4java-osx-1.0.392
160212:002049.840 FINE [sqlite] Internal: cannot load sqlite4java-osx-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-1.0.392 in java.library.path
160212:002049.840 FINE [sqlite] Internal: trying to load sqlite4java-1.0.392
160212:002049.841 FINE [sqlite] Internal: cannot load sqlite4java-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-1.0.392 in java.library.path
160212:002049.841 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d-1.0.392 in java.library.path
160212:002049.842 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d-1.0.392
160212:002049.842 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-osx-d-1.0.392
160212:002049.843 FINE [sqlite] Internal: cannot load sqlite4java-osx-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d-1.0.392 in java.library.path
160212:002049.843 FINE [sqlite] Internal: trying to load sqlite4java-d-1.0.392
160212:002049.844 FINE [sqlite] Internal: cannot load sqlite4java-d-1.0.392: java.lang.UnsatisfiedLinkError: no sqlite4java-d-1.0.392 in java.library.path
160212:002049.844 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64
160212:002049.845 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64 in java.library.path
160212:002049.845 FINE [sqlite] Internal: trying to load sqlite4java-osx
160212:002049.846 FINE [sqlite] Internal: cannot load sqlite4java-osx: java.lang.UnsatisfiedLinkError: no sqlite4java-osx in java.library.path
160212:002049.846 FINE [sqlite] Internal: trying to load sqlite4java
160212:002049.847 FINE [sqlite] Internal: cannot load sqlite4java: java.lang.UnsatisfiedLinkError: no sqlite4java in java.library.path
160212:002049.847 FINE [sqlite] Internal: trying to load sqlite4java-osx-x86_64-d
160212:002049.848 FINE [sqlite] Internal: cannot load sqlite4java-osx-x86_64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-d in java.library.path
160212:002049.848 FINE [sqlite] Internal: trying to load sqlite4java-osx-amd64-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-amd64-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-amd64-d in java.library.path
160212:002049.849 FINE [sqlite] Internal: trying to load sqlite4java-osx-d
160212:002049.849 FINE [sqlite] Internal: cannot load sqlite4java-osx-d: java.lang.UnsatisfiedLinkError: no sqlite4java-osx-d in java.library.path
160212:002049.850 FINE [sqlite] Internal: trying to load sqlite4java-d
160212:002049.850 FINE [sqlite] Internal: cannot load sqlite4java-d: java.lang.UnsatisfiedLinkError: no sqlite4java-d in java.library.path
Error: cannot load SQLite
java.lang.UnsatisfiedLinkError: no sqlite4java-osx-x86_64-1.0.392 in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.almworks.sqlite4java.Internal.tryLoadFromSystemPath(Internal.java:352)
    at com.almworks.sqlite4java.Internal.loadLibraryX(Internal.java:124)
    at com.almworks.sqlite4java.SQLite.main(SQLite.java:368)


If you have an error about the dynamic library loading, read these for how to resolve:


UnsatisfiedLinkError with sqlite4java Jar on Mac OS X NetBeans 
UnsatisfiedLinkError with sqlite4java Jar on Mac OS X 
and specifically related to this DynamoDBLocal Run DynamoDB Local with the java command on Mac OS X.  
https://groups.google.com/forum/#!topic/sqlite4java/9J1lmCuoKLA
https://groups.google.com/forum/#!topic/sqlite4java/jhwt44nYGvw


The only answer that seems to work reliably is to either:


add the libraries from a downloaded sqlite4java distribution to a known location and set the system property java.library.path
add the libraries from a downloaded sqlite4java distribution to /Library/Java/Extensions (macOS only)


Of these I do something like the first option, and add the libraries into the project and make sure the build adds a -Djava.library.path=./lib/sqlite4java where the dynamic libraries are unzipped.  For more resilient testing you can set the java.library.path programatically in your code using this trick (otherwise it is ignored when set in code): http://blog.cedarsoft.com/2010/11/setting-java-library-path-programmatically/

It is evil that DynamoDBLocal hides all Sqlite errors, so you have to debug to find silent failures.  The library is full of things that adjust logging levels and make it hard to debug since they break your ability to see errors.  For example every time when a SqlLite file is opened (class SQLiteDBAccess):

LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java");
LocalDBUtils.setLog4jToUtilsLogging("com.almworks.sqlite4java.Internal");
java.util.logging.Logger.getLogger("com.almworks.sqlite4java").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("com.almworks.sqlite4java.Internal").setLevel(Level.OFF);


Closing Notes: I am looking at alternatives, this is not the best built thing on the planet, looking at the code of DynamoDbLocal and decisions made for it give me very little confidence in it.  Alternator or Jcabi - Dynamo Mock are next on my list.
                                                                        
                                                        
            
            
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
                                                            
            
                      
                    


               
            
    发布评论:
    
         
                        
    
    提交评论 
  
  

                    
                    
                    
                        
                        
                         加载中...
                        
                    
                
          
          	          
            
           
            
                              
                
              
              
                
                  你的背包        
                
              
                            
                2020-12-06 17:21
              
            
            
                                                                       
In August 2018 Amazon announced new Docker image with Amazon DynamoDB Local onboard. It does not require downloading and running any JARs as well as adding using third-party OS-specific binaries (I'm talking about sqlite4java).

It is as simple as starting a Docker container before the tests:

docker run -p 8000:8000 amazon/dynamodb-local


You can do that manually for local development, as described above. IDEs usually provide a way to run arbitrary commands before executing a task, so you can make IDE to start the container for you.

Or you can use it in your CI pipeline. Many CI services provide an ability to start additional containers during the pipeline that can provide dependencies for your tests. Here is an example for Gitlab CI/CD:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test


Or Bitbucket Pipelines:

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test

                                                                        
                                                        
            
            
              
                
                0
              
                 
                
               讨论(0)
              
              
                                                   
              
                                                            
            
                      
                    


               
            
    发布评论:
    
         
                        
    
    提交评论 
  
  

                    
                    
                    
                        
                        
                         加载中...
                        
                    
                
          
          	          
                             
        
        
          
            
            
              
              
            
    


                                 
              
            
                          
    

        
         
                验证码
                
                  
                
                
                   看不清?
                
              
                                  
                    
   
                 
             
              提交回复