Ksql: Left Join Displays columns from stream but not tables

Question

I have one steam and a table in KSQL as mentioned below:

Stream name: DEAL_STREAM

Table name: EXPENSE_TABLE

When I run the below queries it displays only columns from the stream but no table columns are being displays.

Is this the expected output. If not am I doing something wrong?

SELECT TD.EXPENSE_CODE, TD.BRANCH_CODE, TE.EXPENSE_DESC
FROM DEAL_STREAM TD
LEFT JOIN EXPENSE_TABLE TE ON TD.EXPENSE_CODE = TE.EXPENSE_CODE
WHERE TD.EXPENSE_CODE LIKE '%NL%' AND TD.BRANCH_CODE LIKE '%AM%';

An output of the query is as shown below.

NL8232@#0          | AM   | null
NL0232@#0          | AM   | null
NL6232@!0          | AM   | null
NL5232^%0          | AM   | null

Answer 1

When I run the below queries it displays only columns from the stream but no table columns are being displays.

In a stream-table (left) join, the output records will contain null columns (for table-side columns) if there is not matching record in the table at the time of the join/lookup.

Is this the expected output. If not am I doing something wrong?

Is it possible that, for example, you wrote the (1) input data into the stream before you wrote (2) the input data into the table? If so, then the stream-table join query would have attempted to perform table-lookups at the time of (1) when no such lookup data was available in the table yet (because that happened later at time (2)). Because there was no such table data available, the join wrote output records where the table-side columns were null.

Note: This stream-table join in KSQL (and, by extension, Apache Kafka's Streams API, on which KSQL is built) is the pretty much the norm for joins in the streaming world. Here, only the stream-side of the stream-table join will trigger downstream join outputs, and if there's no matching for a stream record on the table-side at the time when a new input record is being joined, then the table-side columns will be null. Since this is, however, a common cause of user confusion, we are currently working on adding table-side triggering of join output to Apache Kafka's Streams API and KSQL. When such a feature is available, then your issue above would not happen anymore.