关于canal具体的原理,以及应用场景,可以参考开发文档:https://github.com/alibaba/canal
下面给出canal的入门Demo
(一)部署canal服务器
可以参考官方文档的QuickStart:https://github.com/alibaba/canal/wiki/QuickStart
为了完整性,下面重复给出如何配置canal服务器
开启mysql的binlog功能,并配置binlog模式为row
1. Windows环境下,是修改my.ini文件:
[mysqld] log-bin=mysql-bin #添加这一行就ok binlog-format=ROW #选择row模式 server_id=1 #配置mysql replaction需要定义,不能和canal的slaveId重复
2. 在mysql中 配置canal数据库管理用户,配置相应权限(repication权限),运行mysql后依次运行这四条代码:
1 CREATE USER canal IDENTIFIED BY 'canal'; 2 GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; 3 -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; 4 FLUSH PRIVILEGES;
3. 下载好canal,修改配置 instance.properties:
################################################# ## mysql serverId canal.instance.mysql.slaveId = 1234 # position info,需要改成自己的数据库信息 canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = # username/password,需要改成自己的数据库信息 canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal_test canal.instance.connectionCharset = UTF-8 # table regex canal.instance.filter.regex = .*\\..* #################################################
4. 启动startup.bat,并且查看日志log:
如果日志中有记录,证明canal服务器部署成功了。
(二)运行canal客户端
运行canal客户端代码时,一定要先启动canal服务器!!!
1. 建立实例maven工程:
- 不选择任何Maven模板
2. 添加pom依赖:
<dependency> <groupId>com.alibaba.otter</groupId> <artifactId>canal.client</artifactId> <version>1.0.12</version> </dependency>
3. 更新依赖
4. canal客户端代码:
1 import java.net.InetSocketAddress; 2 import java.util.List; 3 4 import com.alibaba.otter.canal.client.CanalConnector; 5 import com.alibaba.otter.canal.protocol.Message; 6 import com.alibaba.otter.canal.protocol.CanalEntry.Column; 7 import com.alibaba.otter.canal.protocol.CanalEntry.Entry; 8 import com.alibaba.otter.canal.protocol.CanalEntry.EntryType; 9 import com.alibaba.otter.canal.protocol.CanalEntry.EventType; 10 import com.alibaba.otter.canal.protocol.CanalEntry.RowChange; 11 import com.alibaba.otter.canal.protocol.CanalEntry.RowData; 12 import com.alibaba.otter.canal.client.*; 13 14 public class canal_client { 15 16 public static void main(String args[]) { 17 // 创建链接 18 CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("127.0.0.1", 19 11111), "example", "", ""); 20 int batchSize = 1000; 21 int emptyCount = 0; 22 try { 23 connector.connect(); 24 connector.subscribe(".*\\..*"); 25 connector.rollback(); 26 int totalEntryCount = 1200; 27 while (emptyCount < totalEntryCount) { 28 Message message = connector.getWithoutAck(batchSize); // 获取指定数量的数据 29 long batchId = message.getId(); 30 int size = message.getEntries().size(); 31 if (batchId == -1 || size == 0) { 32 emptyCount++; 33 System.out.println("empty count : " + emptyCount); 34 try { 35 Thread.sleep(5000); 36 } catch (InterruptedException e) { 37 e.printStackTrace(); 38 } 39 } else { 40 emptyCount = 0; 41 printEntry(message.getEntries()); 42 } 43 connector.ack(batchId); // 提交确认 44 } 45 System.out.println("empty too many times, exit"); 46 }catch (Exception e){ 47 //connector.rollback(batchId); // 处理失败, 回滚数据 48 } 49 finally { 50 connector.disconnect(); 51 } 52 } 53 54 private static void printEntry( List<Entry> entrys) { 55 for (Entry entry : entrys) { 56 if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) { 57 continue; 58 } 59 RowChange rowChage = null; 60 try { 61 rowChage = RowChange.parseFrom(entry.getStoreValue()); 62 } catch (Exception e) { 63 throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(), e); 64 } 65 66 EventType eventType = rowChage.getEventType(); 67 System.out.println(String.format("================> binlog[%s:%s] , name[%s,%s] , eventType : %s", 68 entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(), 69 entry.getHeader().getSchemaName(), entry.getHeader().getTableName(), 70 eventType)); 71 for (RowData rowData : rowChage.getRowDatasList()) { 72 if (eventType == EventType.DELETE) { 73 printColumn(rowData.getBeforeColumnsList()); 74 } else if (eventType == EventType.INSERT) { 75 printColumn(rowData.getAfterColumnsList()); 76 } else { 77 System.out.println("-------> before"); 78 printColumn(rowData.getBeforeColumnsList()); 79 System.out.println("-------> after"); 80 printColumn(rowData.getAfterColumnsList()); 81 } 82 } 83 } 84 } 85 86 private static void printColumn( List<Column> columns) { 87 for (Column column : columns) { 88 System.out.println(column.getName() + " : " + column.getValue() + " update=" + column.getUpdated()); 89 } 90 } 91 }
5. 运行客户端实例:
6. 触发数据库变更:
总结:参考网上的资料,运行这个canal的Demo,对canal的机制有一点了解;当MySQL将binary log发送给canal服务器,然后canal client从服务器获取binary log,同时解析出来,尤其是解析的过程对于理解canal会更深刻一点。
建议运行的代码的过程中打断点调试处理!
来源:https://www.cnblogs.com/jayinnn/p/9606466.html