1. 技术文章
  2. Apache spark Hive, executable JAR with maven shade

Apache spark Hive, executable JAR with maven shade

由 心已入冬 提交于 2019-11-29 10:38:20
anebril

I have found solution to my problem. Solution is described in answer to original question for datanucleus in executable jar: https://stackoverflow.com/a/27030103/6390361

  1. Edit MANIFEST.MF to pretend that it is datanucleus OSGi bundle. This can be done by adding Bundle-SymbolicName and Premain-Class entries from datanucleus-core manifest.

  2. Create file plugin.xml in your classpath (resource folder) and use root tag from datanucleus-core project.

  3. Put all extension-point tags from datanucleus-core and datanucleus-rdbms at the beginning of the plugin tag. All extension-points from RDBMS projects has to be prefixed with store.rdbms. This is very important because datanucleus uses fully classified IDs including part from root plugin tag.

  4. Merge all extension tags from projects datanucleus-core, datanucleus-rdbms and datanucleus-api-jdo and put them behind all extension points. Be careful some extensions are present in more projects so you need to merge content of extensions with same IDs.

Manifest entries

Bundle-SymbolicName: org.datanucleus;singleton:=true
Premain-Class: org.datanucleus.enhancer.DataNucleusClassFileTransformer

plugin.xml

File plugin.xml is too big to be pasted here but you should be able to merge it by hand. Following code contains all RDBMS extension points with fixed IDs.

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.2"?>    
<plugin id="org.datanucleus" name="DataNucleus Core" provider-name="DataNucleus">

    <!-- Extension points from datanucleus-core -->
    <extension-point id="api_adapter" name="Api Adapter" schema="schema/apiadapter.exsd"/>
    ...

    <!-- extension points from datanucleus-rdbms - fixed IDs -->
    <extension-point id="store.rdbms.connectionprovider" name="Connection Provider" schema="schema/connectionprovider.exsd"/>
    <extension-point id="store.rdbms.connectionpool" name="ConnectionPool" schema="schema/connectionpool.exsd"/>
    <extension-point id="store.rdbms.sql_expression" name="SQL Expressions" schema="schema/sql_expression.exsd"/>
    <extension-point id="store.rdbms.sql_method" name="SQL Methods" schema="schema/sql_method.exsd"/>
    <extension-point id="store.rdbms.sql_operation" name="SQL Expressions" schema="schema/sql_operation.exsd"/>
    <extension-point id="store.rdbms.sql_tablenamer" name="SQL Table Namer" schema="schema/sql_tablenamer.exsd"/>
    <extension-point id="store.rdbms.rdbms_mapping" name="RDBMS Mapping" schema="schema/rdbms_mapping.exsd"/>

    <!-- Merged extensions from datanucleus-core, datanucleus-rdbms and datanucleus-api-jdo -->
    <extension point="org.datanucleus.persistence_properties">...</extension>
    ...
</plugin>

maven-shade-plugin

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <manifestEntries>
                            <Main-Class>${main.class}</Main-Class>
                            <Premain-Class>org.datanucleus.enhancer.DataNucleusClassFileTransformer</Premain-Class>
                            <Bundle-SymbolicName>org.datanucleus;singleton:=true</Bundle-SymbolicName>
                        </manifestEntries>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>reference.conf</resource>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                </transformers>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
        </execution>
    </executions>
</plugin>

Here is a link to a newer plugin.xml: https://pastebin.com/8b2deeCL build out of these dependencies:

  • datanucleus-core-4.1.6
  • datanucleus-rdbms-4.1.7
  • datanucleus-api-jdo-4.2.1

according to the answer by @anebril .

标签
maven
apache-spark
datanucleus
maven-shade-plugin
spark-hive
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!