In-process HSQLDB database are not expected to be opened by others, even for file-based storage.
The documentation hints that this is possible: Server Modes, Advance
I've experience with starting HQLDB in server mode outside of application and connecting from several applications to the database.
As described in the doc the process is straight-forward.
The following works for me:
Start a server from your code, which is shown in the org.hsqldb.test.TestBase code in the HSQLDB source code. Something like:
Server server = new Server();
server.setDatabaseName(0, "test");
server.setDatabasePath(0, "file:/path/to/db");
server.start();
In the same JVM, open a Connection to the same database the way you would in a normal (non-Server) program, like:
conn = DriverManager.getConnection("jdbc:hsqldb:file:/path/to/db");
Then use conn to execute whatever SQL you want.
In my own experimentation this appears to work, and it works for in-memory and file database.
Other JVMs, of course, will need to connect to the server using TCP. Trying to open the database file directly in another JVM will result in the usual error about the database being locked.