How do I implement in memory or embedded kafka not for testing purposes? [closed]

Answer 1

What would be the purpose of such an application (produce and consume its own records)? The embedded broker is not designed for production use.

Since version 2.3.4, the container property missingTopicsFatal is false by default, which will allow the container to start even if the broker is not available. With earlier versions, you can set it to false to get the same effect.

When it is true, the container connects to the broker during startup to verify that the topic(s) exist.

You can also set the container's autoStartup=false to prevent the container from starting at all.

EDIT

I wouldn't recommend using this in production, but you can remove the test scope from spring-kafka-test and declare the broker as a @Bean...

        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka-test</artifactId>
<!--            <scope>test</scope> -->
        </dependency>

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerListProperty("spring.kafka.bootstrap-servers"); // override application property
}

I just tested it with this app...

@SpringBootApplication
public class So63812994Application {

    public static void main(String[] args) {
        SpringApplication.run(So63812994Application.class, args);
    }

    @Bean
    EmbeddedKafkaBroker broker() {
        return new EmbeddedKafkaBroker(1)
                .kafkaPorts(9092)
                .brokerListProperty("spring.kafka.bootstrap-servers");
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so63812994").partitions(1).replicas(1).build();
    }

    @KafkaListener(id = "so63812994", topics = "so63812994")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            template.send("so63812994", "foo");
        };
    }

}

spring.kafka.bootstrap-servers=realKafka:9092
spring.kafka.consumer.auto-offset-reset=earliest

EDIT2

With the above configuration, other applications on the same host can connect with localhost:9092.

If you need remote access to this embedded broker, you will need some additional configuration:

@Bean
EmbeddedKafkaBroker broker() {
    return new EmbeddedKafkaBroker(1)
            .kafkaPorts(9092)
            .brokerProperty("listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("advertised.listeners", "PLAINTEXT://localhost:9092,REMOTE://10.0.0.20:9093")
            .brokerProperty("listener.security.protocol.map", "PLAINTEXT:PLAINTEXT,REMOTE:PLAINTEXT")
            .brokerListProperty("spring.kafka.bootstrap-servers");
}

You can then connect from other servers with 10.0.0.20:9093.