WeBank/Exchangis | 易学教程

English | 中文

Introduction

Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources. On the application layer, it has business features such as data permission management and control, high availability of node services and multi-tenant resource isolation. On the data layer, it also has architectural characteristics such as diversified transmission architecture, module plug-in and low coupling of components.

Exchnagis's transmission and exchange capabilities depend on its underlying aggregated transmission engines. It defines a unified parameter model for various data sources on the top layer. It maps and configures the parameter model for each transmission engine, and then converts it into the engine's input model. Each type of engine will add Exchangis features, and the enhancement of certain engine features will improve the Exchangis features. Exchangis's default engine aggregated and enhanced is Alibaba's DataX transmission engine.

Features

Data Source Management
Share your own data source in a bound project；
Set the external authority of the data source to control the inflow and outflow of data。
Muti-transport Engine Support
Transmission engine scales horizontally；
The current version fully aggregates the offline batch engine DataX and partially aggregates the big data batch derivative engine SQOOP
Near Real-time Task Control
Quickly capture the transmission task log, transmission rate and other information, close the task in real time；
Dynamically limit transmission rate based on bandwidth
Support Unstructured Transmission
Transform the DataX framework and build a binary stream fast channel separately, suitable for pure data synchronization scenarios without data conversion。
Task Status Self-check
Monitor long-running tasks and tasks with abnormal status, release occupied resources in time and issue alarms。

Comparison With Existing Systems

Comparison of some existing data exchange tools and platforms：

Function module	Description	Exchangis	DataX	Sqoop	DataLink	DBus
UI	Integrated the convenient management interface and monitoring window	Integrated	None	None	Integrated	Integrated
Installation and deployment	Ease of deployment and third-party dependencies	One-click deployment, no dependencies	No dependencies	Rely on Hadoop environment	Rely on Zookeeper	Rely on a large number of third-party components
Data authority management	Multi-tenant permission configuration and data source permission control	Support	Not support	Not support	Not support	Support
	Dynamic limit transmission	Support	Partially supported, unable to adjust dynamically	Partially supported, unable to adjust dynamically	Support	Support，with Kafka
Data transmission	Unstructured data binary transmission	Support, fast channel	Not support	Not support	Not support，only transport record	Not support，need to be converted to a unified message format
	Embed processing code	Support，dynamic compilation	Not support	Not support	Not support	Partial support
	Transmission breakpoint recovery	Support（Not open source）	Not support	Not support	Support	Support
High availability	Mutiple services, failure does not affect the use	Application high availability, transmission single point（Distributed architecture planning）	Single point service（Open source version）	Multipoint transmission	Application、transmission high availability	Application、transmission high availability
System Management	Nodes、resources management	Support	Not support	Not support	Support	Support