Solr DataImportHandler CachedSqlEntityProcessor ClassCastException

Question

I am using Solr 4.6.0 and trying to import my data using CachedSqlEntityProcessor, but somehow I end up getting a ClassCastException.

Answer 1

This is a regression coming out of the introduction of pluggable cache support in https://issues.apache.org/jira/browse/SOLR-2382 and the workaround (working for me) is to cast to strings in all columns that you will use as either keys or values. In PostgreSQL cast syntax:

<entity name="par" dataSource="d" query="SELECT id, xyz, child_id::text FROM par;">
  <entity name="child" dataSource="d" query="SELECT id::text, abc FROM child;"
          processor="CachedSqlEntityProcessor"
          where="id=par.child_id"/>
</entity>

(and in other DBs I guess you need CAST(id AS VARCHAR(10) or such instead of id::text).

Answer 2

Solr CachedSqlEntityProcessor is depreciated somewhere around version 4.5. The new syntax uses cacheImpl, cacheKey & cacheLookup

I had to update the SQL in my data-config.xml

Old Syntax

processor="CachedSqlEntityProcessor"
where="fkId=parentTable.parentId"

New Syntax

cacheKey="fkId" cacheLookup="parenttable.parentId" cacheImpl="SortedMapBackedCache"

See full latest Solr wiki info, the old DataImportHanlder page is out of date https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler#UploadingStructuredDataStoreDatawiththeDataImportHandler-EntityProcessors

Answer 3

CachedSqlEntityProcessor relies on DIHCacheSupport(Map<String,Object> getIdCacheData(...) ) that is not adapted to work with keys of type Integer, which is expected behavior for any cache. (Object key = context.resolve(cacheForeignKey);, this key should really be of type String)

SELECT CAST(id as CHAR(32)) AS CID,full_name FROM sm_conferences 

Answer 4

This is a matter of case. You need to write the key in the cacheLookup attribute in upper-case. If you use

cacheLookup="publication.CONFERENCE_ID"

it will work.

All columns you gather for the entity publication are gathered within a Map internally. The keys to that map are uppercased. When you write cacheLookup="publication.CONFERENCE_ID" that short part after the equals sign publication.CONFERENCE_ID is a short-cut to that map. Apparently a String#toUpper is missing at some point in between.

---

I have added a test case about this, have a look at the DIHCachedTest, check it out and let it run. You will need maven for this. I create an embedded HSQLDB that mimics your schema and run an embedded Solr that does have 2 data-import configurations.

• src/main/resources/solr/dih-cached/conf/data-config-fault.xml
• src/main/resources/solr/dih-cached/conf/data-config.xml

The first one is used for the test case DIHCachedTest#runImportWithFault. This will run into the same error you have posted in your question.

The second one is used for the test case DIHCachedTest#runImport. This will succeed.

If you compare both configurations, you will see that the only difference is the case of cacheLookup="publication.CONFERENCE_ID" vs cacheLookup="publication.conference_id".

---

The issue SOLR-2483 DIH - an uppercase problem in query parameters describes this bug pattern.

Answer 5

I dont think there is something wrong with your dataimporthandler configuration, I had rather check that your database types match your corresponding sold field type

Something like a database type resulting in a java.lang.Integer is passed to a solr field based on java.lang.String

OUPS, I might have got it wrong, is your uniqueKey field for SOLR still "id"? you've set in your schema this field as being an integer, which is a bad idea.

http://wiki.apache.org/solr/SchemaXml#The_Unique_Key_Field

"Note that if you have enabled the QueryElevationComponent in solrconfig.xml it requires the schema to have a uniqueKey of type StrField. It cannot be, for example, an int field."