Kafka performance test not available

问题

I'm a newbie for Kafka, using the version of kafka_2.8.0-0.8.1.1.

After build Kafka brokers and test it by producer and consumer that work fine, I decide to make some performance test for Kafka.

I found Kafka Benchmark by search kafka performance test and try it by execute

./bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196

but got

Error: Could not find or load main class org.apache.kafka.clients.tools.ProducerPerformance

Then I found another way to test kafka is the command

./bin/kafka-producer-perf-test.sh --broker-list 192.168.2.11:19092,192.168.2.11:19093,192.168.2.11:19094 --messages 100 --topic j1-soa --threads 1 --message-size 100 --batch-size 100 --compression-codec 1

got another error is

Error: Could not find or load main class kafka.perf.ProducerPerformance

I searched the errors. they told it should execute some command like

./sbt update
./sbt package
./sbt assembly-package-dependency

or

./gradlew jar

before executing ./bin/kafka-producer-perf-test.sh.

I realize the above commands only working for Source version instead Binary version.

Then I download the kafka-0.8.1.1-src.tgz and trying to execute above commands.

I do not know what is sbt and gradlew.

I just trying to running above commands.

And I got error when running sbt assembly-package-dependency

[info] Set current project to kafka (in build file:/Users/foxzen/Develop/kafka/)
[error] Not a valid command: assembly-package-dependency
[error] Not a valid project ID: assembly-package-dependency
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly-package-dependency (similar: sbt-dependency)
[error] assembly-package-dependency
[error]                            ^

Also error for ./gradlew jar

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'kafka-0.8.1.1-src'.
> Could not resolve all dependencies for configuration ':classpath'.
  > Could not resolve nl.javadude.gradle.plugins:license-gradle-plugin:0.6.1.
 Required by:
     org.apache.kafka:kafka-0.8.1.1-src:0.8.1.1
  > Could not GET 'http://repo1.maven.org/maven2/nl/javadude/gradle/plugins/license-gradle-plugin/0.6.1/license-gradle-plugin-0.6.1.pom'.
     > Connection to http://127.0.0.1:8888 refused
  > Could not GET 'http://dl.bintray.com/content/netflixoss/external-gradle-plugins/nl/javadude/gradle/plugins/license-gradle-plugin/0.6.1/license-gradle-plugin-0.6.1.pom'.
     > Connection to http://127.0.0.1:8888 refused

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 2.924 secs

I really do not know what the error message means, and can not found a solution.

How to make the command ./bin/kafka-producer-perf-test.sh work fine so I can test Kafka performance ?

---

EDIT

Its my fault to cause ./gradlew jar fail.

The internet connection have proxy setting to 127.0.0.1:8888,

I cant remember when did I do it...

After remove proxy setting of Internet connection. the ./gradlew jar command works ok.

But I got another error, again........

start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
[2015-02-04 10:37:22,936] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2015-02-04 10:37:22:821, 2015-02-04 10:37:23:451, 1, 100, 100, 0.01, 0.0151, 100, 158.7302

cause by ./bin/kafka-producer-perf-test.sh --broker-list 192.168.2.11:19092,192.168.2.11:19093,192.168.2.11:19094 --messages 100 --topic j1-soa --threads 1 --message-size 100 --batch-size 100 --compression-codec 1

Is this mean the performance test success ?

But I did not get a meaningful output.

I think the error info should be

Failed to load class "org.slf4j.impl.StaticLoggerBinder".

How to solve it ?

回答1:

Package name of ProducerPerformance has been changed from org.apache.kafka.clients.tools to org.apache.kafka.tools.

So, you should be running:

./bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196

The change has beed made in this commit.

回答2:

With kafka_2.11-0.10.1.1, I only managed to use it with:

./bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --throughput=1000 --topic=MY_TOPIC --num-records=50000000 --record-size=200 --producer-props bootstrap.servers=YOUR_KAFKA_IP:9092 buffer.memory=67108864 batch.size=64000

Be aware there's also a different class named kafka.tools.ProducerPerformance. But the one above looks better

来源：https://stackoverflow.com/questions/28296576/kafka-performance-test-not-available