see: https://docs.confluent.io/current/quickstart/ce-quickstart.html#ce-quickstart
This quick start shows you how to get up and running with Confluent Platform and its main components. This quick start demonstrates both the basic and most powerful capabilities of Confluent Platform, including using Control Center for topic management and event stream processing using KSQL. In this quick start you create Kafka topics, use Kafka Connect to generate mock data to those topics, and create KSQL streaming queries on those topics. You then go to Control Center to monitor and analyze the streaming queries.
You can also run an automated version of this quick start designed for Confluent Platform local installs.
Important
Java 1.8 is supported in this version of Confluent Platform (Java 1.9 and 1.10 are currently not supported). You should run with the Garbage-First (G1) garbage collector. For more information, see the Supported Versions and Interoperability.
- Prerequisites
-
- Internet connectivity
Step 1: Download and Start Confluent Platform
-
Go to the downloads page and choose Confluent Platform.
Tip
Download a previous version from Previous Versions.
-
Provide your name and email if applicable for the version and select Download.
-
Decompress the file. You should have these directories:
Folder Description /bin/ Driver scripts for starting and stopping services /etc/ Configuration files /lib/ Systemd services /logs/ Log files /share/ Jars and licenses /src/ Source files that require a platform-dependent build -
Add the install location of the Confluent bin directory to your PATH:
export PATH=<path-to-confluent>/bin:$PATH
-
Install the Kafka Connect Datagen source connector using the Confluent Hub client. This connector generates mock data for demonstration purposes and is not suitable for production. Confluent Hub is an online library of pre-packaged and ready-to-install extensions or add-ons for Confluent Platform and Apache Kafka.
<path-to-confluent>/bin/confluent-hub install \ --no-prompt confluentinc/kafka-connect-datagen:0.1.0
-
Start Confluent Platform using the Confluent CLI
start
command.<path-to-confluent>/bin/confluent start
This command starts all of the Confluent Platform components; including Kafka, ZooKeeper, Schema Registry, HTTP REST Proxy for Apache Kafka, Kafka Connect, KSQL, and Control Center.
Important
The Confluent CLI is meant for development purposes only and is not suitable for a production environment. The data that are produced are transient and are intended to be temporary.
Your output should resemble:
Starting zookeeper zookeeper is [UP] Starting kafka kafka is [UP] Starting schema-registry schema-registry is [UP] Starting kafka-rest kafka-rest is [UP] Starting connect connect is [UP] Starting ksql-server ksql-server is [UP] Starting control-center control-center is [UP]
Step 2: Create Kafka Topics
In this step, you create Kafka topics by using the Confluent Control Center. Confluent Control Center provides the functionality for building and monitoring production data pipelines and event streaming applications.
Navigate to the Control Center web interface at http://localhost:9021/.
-
Important
It may take a minute or two for Control Center to come online.
-
Select Management -> Topics and click Create topic.
-
Create a topic named
pageviews
and click Create with defaults. -
Select Management -> Topics, create a topic named
users
, and click Create with defaults.
Step 3: Install a Kafka Connector and Generate Sample Data
In this step, you use Kafka Connect to run a demo source connector called kafka-connect-datagen
that creates sample data for the Kafka topics pageviews
and users
.
Tip
The Kafka Connect Datagen connector was installed manually in Step 1: Download and Start Confluent Platform.
Run one instance of the Kafka Connect Datagen connector to produce Kafka data to the pageviews
topic in AVRO format.
-
-
Scroll down to select DatagenConnector from the Connector class list.
-
Name the connector
datagen-pageviews
. After naming the connector, new fields appear. Scroll down and specify the following configuration values:- In the Key converter class field, type
org.apache.kafka.connect.storage.StringConverter
. - In the kafka.topic field, type
pageviews
. - In the max.interval field, type
100
. - In the iterations field, type
1000000000
. - In the quickstart field, type
pageviews
.
- In the Key converter class field, type
-
Click Continue.
-
Review the connector configuration and click Launch.
-
-
Run another instance of the Kafka Connect Datagen connector to produce Kafka data to the
users
topic in AVRO format.-
Click Add connector.
-
Scroll down to select DatagenConnector from the Connector class list.
-
Name the connector
datagen-users
. After naming the connector, new fields appear. Scroll down and specify the following configuration values:- In the Key converter class field, type
org.apache.kafka.connect.storage.StringConverter
. - In the kafka.topic field, type
users
. - In the max.interval field, type
1000
. - In the iterations field, type
1000000000
. - In the quickstart field, type
users
.
- In the Key converter class field, type
-
Click Continue.
-
Review the connector configuration and click Launch.
-
Step 4: Create and Write to a Stream and Table using KSQL
In this step, KSQL queries are run on the pageviews
and users
topics that were created in the previous step. The KSQL commands are run using the KSQL tab in Control Center.
Tip
You can also run these commands using the KSQL CLI from your terminal with this command: <path-to-confluent>/bin/ksql http://localhost:8088
.
Create Streams and Tables
In this step, KSQL is used to create streams and tables for the pageviews
and users
topics.
-
From the Control Center navigation menu, click Development -> KSQL. By default, you are on the KSQL EDITOR page.
-
Click the STREAMS tab -> Add a Stream and select the
pageviews
topic. -
Choose your stream options:
- In the How are your messages encoded? field, select
AVRO
. - In the Field(s) you'd like to include in your STREAM field, ensure fields are set as follows:
viewtime
with typeBIGINT
userid
with typeVARCHAR
pageid
with typeVARCHAR
- In the How are your messages encoded? field, select
-
Click Save STREAM.
-
Click the TABLES tab -> Add a Table and select the
users
topic. -
Choose your table options:
- In the How are your messages encoded? field, select
AVRO
. - In the Key field, select
userid
. - In the Field(s) you'd like to include in your TABLE field, set fields as follows:
registertime
with typeBIGINT
userid
with typeVARCHAR
regionid
with typeVARCHAR
gender
with typeVARCHAR
interests
with typeARRAY<VARCHAR>
contact_info
with typeMAP<VARCHAR, VARCHAR>
- In the How are your messages encoded? field, select
-
Click Save TABLE.
Write Queries
These examples write queries using the KSQL tab in Control Center.
-
From the Control Center navigation menu, click Development -> KSQL. By default, you are on the KSQL EDITOR page. Click Query properties to add a custom query property. Set the
auto.offset.reset
parameter toearliest
.This instructs KSQL queries to read all available topic data from the beginning. This configuration is used for each subsequent query. For more information, see the KSQL Configuration Parameter Reference.
-
Run the following queries.
-
Create a query that returns data from a stream with the results limited to three rows.
SELECT pageid FROM pageviews LIMIT 3;
Your output should resemble:
-
Create a persistent query that filters for female users. The results from this query are written to the Kafka
PAGEVIEWS_FEMALE
topic. This query enriches thepageviews
STREAM by doing aLEFT JOIN
with theusers
TABLE on the user ID, where a condition (gender = 'FEMALE'
) is met.CREATE STREAM pageviews_female AS SELECT users.userid AS userid, pageid, regionid, gender FROM pageviews LEFT JOIN users ON pageviews.userid = users.userid WHERE gender = 'FEMALE';
Your output should resemble:
-
Create a persistent query where a condition (
regionid
) is met, usingLIKE
. Results from this query are written to a Kafka topic namedpageviews_enriched_r8_r9
.CREATE STREAM pageviews_female_like_89 WITH (kafka_topic='pageviews_enriched_r8_r9', value_format='AVRO') AS SELECT * FROM pageviews_female WHERE regionid LIKE '%_8' OR regionid LIKE '%_9';
Your output should resemble:
-
Create a persistent query that counts the pageviews for each region and gender combination in a tumbling window of 30 seconds when the count is greater than 1. Because the procedure is grouping and counting, the result is now a table, rather than a stream. Results from this query are written to a Kafka topic called
PAGEVIEWS_REGIONS
.CREATE TABLE pageviews_regions AS SELECT gender, regionid , COUNT(*) AS numusers FROM pageviews_female WINDOW TUMBLING (size 30 second) GROUP BY gender, regionid HAVING COUNT(*) > 1;
Your output should resemble:
-
Click RUNNING QUERIES and you should see the following persisted queries:
-
Step 5: View Your Stream in Control Center
From the Control Center interface you can view all of your streaming KSQL queries.
Navigate to the Control Center web interface Monitoring -> Data streams tab at http://localhost:9021/monitoring/streams. The monitoring page shows the total number of messages produced and consumed on the cluster. You can scroll down to see more details on the consumer groups for your queries.
Tip
Depending on your machine, these charts may take a few minutes to populate and you might need to refresh your browser.
Now that your streams are running you can monitor them.
Step 6: Stop Confluent Platform
When you are done working with the local install, you can stop Confluent Platform.
-
Stop Confluent Platform using the Confluent CLI
stop
command.<path-to-confluent>/bin/confluent stop
-
Destroy the Confluent Platform instance with the
destroy
command.<path-to-confluent>/bin/confluent destroy
https://docs.confluent.io/current/streams/index.html#kafka-streams
来源:oschina
链接:https://my.oschina.net/u/4381342/blog/3625456