How to specify system property in hadoop except modify hadoop-env.sh?

若如初见. 提交于 2019-12-05 06:55:27

问题


I'd like to set system property (not hadoop property) when running hadoop jobs. I found that it's not easy to setting system property. Even I set the property in shell

export HADOOP_OPTS="$HADOOP_OPTS:-Dproperty=value"

It's still not working. The "-D" option of hadoop command line is only for Configuration, not system property. So the "-D" option also not working

Anyone has ideas ? thanks


回答1:


Why not simply use -Dfoo.bar=example in-line when starting the job via command line, like so:

hadoop jar example.jar com.example.ExampleTool -Dfoo.bar=example argument

In order to get at the property in code, use conf.get("foo.bar");


Now if you genuinely need it to be set as a system property, you can set it at the start of your code using the value you got from Hadoop config like so:

String property = conf.get("foo.bar");
System.setProperty("foo.bar", property);



回答2:


The hadoop script invoke java class like this

exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"

so, we can pass system-wide property like this:

export HADOOP_OPTS="$HADOOP_OPTS -Dfoo=bar"



回答3:


Check the difference between system properties and the environment variables and how to set them here.



来源:https://stackoverflow.com/questions/15490090/how-to-specify-system-property-in-hadoop-except-modify-hadoop-env-sh

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!