ETLCloud CDC中如何监听多表？
在当前数据驱动的时代，CDC（Change Data Capture）数据库实时增量数据采集的重要性愈发凸显。随着企业业务规模扩大和数字化转型加速，数据实时性已成为决策优化和业务敏捷的关键。CDC技术通过精准捕获源数据库的增量变更，以毫秒级延迟同步至目标系统，有效解决了传统批量ETL的高延迟问题。在金融交易、电商库存、物联网监测等场景中，实时数据流能够确保风控系统及时识别异常、库存状态精准可视、设备数据动态分析，避免因数据滞后导致的业务损失。本文将基于ETLCloud的CDC实时数据采集能力，详细演示如何实现对MySQL多张数据表的实时监听，准确捕获源表的变更数据，并将其高效同步至目标数据库表。一、准备工作对数据库进行CDC采集需要做好以下工作：1.部署好ETLCloud。2.因为CDC功能是基于数据库的日志采集，因此要按照文档开启数据库CDC。3.配置好要监听的数据源以及目标端的数据源。以上配置的文档可以参考ETLCLoud官网帮助文档。二、使用直接传输的方式实现CDC功能1.创建CDC监听器来到ETLCloud首页，找到实时数据集成模块，点击进入https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0de9d148120fb25e62545425631ce1efbd53521b662244905772923aac53f03c98e58ad4407492801a5756ea65ad9a9897?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af新建数据库监听器https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d4a4f6d66f73f81a9bc0302f135c4ccf33591eb0585a81ef780a483ea2584171d4a8d1d80ca44567e4b2d1ff9b9dff0a4?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af配置监听器接收端https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d2c56e705bbee31f01954b1bcd290e1a5a35a7e870ed489f39a1c2a7d3026eddde5a5d377a6fa5baa9205d036076fe348?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d2acc3f23661283c72fd528f5b8dfa53960cfb48e01030eae535828d32bce402cfd3a3023627f9cf0b39e7aa7792a8557?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af配置监听器启动模式https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d1e3976710e1b271dc3843fd18a97c573096587fb79cfe82bda4b39f36c19bbdbc08e3e8885147a0b92f995b3d4d72bc0?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af2.启动监听器配置完成监听器后启动监听器。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d180af3383b0b06bc90d951661a935904da37ac14ac4b2c05c31578eb9712824be9fe802d3827a57697a367de4f63fd64?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af3.验证功能稍等片刻，看到增量已启动代表监听器启动成功。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d951778c999c38cb5e80869a2329f5408ca7ef1306151589bda5503731c39975497bafb5ee9beeca5de4075cc0a5b8a0d?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af之后，一旦源表的数据发生了增、删、改，会把变化的数据传到平台，并把源表的增删改操作同步到目标库https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d00e652a286771433f9976ffd925dea82a9381f19f104e268fb60f2722f2df41f2ed5bc84e7f50c703c538177a9f3e0f0?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d1e3976710e1b271da86f27e37746195efaa5695181002c5e805f9443c6ff75a84599f0e1c199eae877a7ada4764ed499?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af三、通过传输到ETL流程的方式实现CDC功能1.创建离线流程首先来到离线数据集成模块创建一个流程用以处理监听到的数据https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d0ac92b56c5c57a36f7429b4c98f4d1352404f93f8c8fd26e3d485ef23bf018a10807c93d1cdabb185247a00effead888?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d620925421bdb9898ced758f79c97a178c0c103578054a9f349126e6049b80991ff880931741aed7df9081f3e3ee7ab89?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d2d9bb4d026d8248cae9dff50534ba461a9e3271c306fd171d30f1bdeecf70d1d4a14f0a82fd7135a4eb673a7a36f678b?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d7a936c7509af19e7a498479c02dea4fc8b559b360252d3605aaf9b220412cf707a6918e0ac61f45411f3a0446a5ceec1?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d3a7ced29c5772a2af148bb1efacf6c758ad43f144581774d2722c84315a17f02571cff2b91e161365756ea65ad9a9897?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d0122f13d8ddfc0bb5c779af0f4a59d7f574c81689dd767d88d8ba1b870638605cbf11f80c39b095350ab44cff799417f?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af动态库表输出组件配置：https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d1b1f22ea4caae8c7d9ee44b6f7ab0ed8959a5f1dc202dc5694ad1c43cfa66329999bf6fd0d5a136a3e7c2dcc7608a1cb?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af2.创建CDC监听器同样创建一个新的监听器https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d4a4f6d66f73f81a9bc0302f135c4ccf33591eb0585a81ef780a483ea2584171d4a8d1d80ca44567e4b2d1ff9b9dff0a4?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0daba574a945976e419903e4505f9bfb73fd9bfc8f01f4e5b27ab04c2f0c09c348467e78731987b66a34674a31359abff3?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af3.启动监听器https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d311f0bd38b2a38ea70652d3072af592ebcb4dbfbc3fa9570ecf2ff93ff431f05b040881582b6b1b3291dca4bb196748b?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af监听器启动后同样可以看到监听记录https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d2d9bb4d026d8248c504d910d948161497067278dac27b227ddae8f8000c88a1914534878ac0bf85594cf45718c973510?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af详细情况可以在接受监听数据的离线流程的运行记录看https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0df8ed0ea74b05ad8a03a46bcb35c7b63371f44ad1e7dc020cd5a2b4b27577dc70eec6c75fed3b9b854558dd92c5501916?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af4.传到etl流程的另一种方式上面的情况是针对源表和目标表表名与表结构一致的情况，如果表结构不一样或者需要对监听到的数据做一些清洗等处理，可以选择使用多分支的情况。监听器的配置无需改变，流程设计为下：https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d516c655f43d37927c2526191c51eaa42ae2ac4769bf55752cef4b9aba09b5a2f26dc8928ff2188bb829f9ade368cd2dd?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d4a4f6d66f73f81a9b088bf4a35237f9f4beed47fbc9e0eea7486ae4dfa35792d4d4003a25a51e66a86abc59b84edd64e?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d238eababefb5fa098ae0c3104aa28a9628ee95e0882f9133c68a172ba854d3eba6b800843dde73273c02c5ae4a5ac1f7?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d21236fd7bb961673762d869fa6b3f0b673e42fe1b339fc60f788df96ef46b3d76209beb82b8230f03b17ce1361c6c5db?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288afhttps://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d9b6d6801cd4af704ee02df1dc6b0797bd7c5fcfa327be8f06a96de21aa30a3632ebd954f5c2fe244ceaa06b3cb6e8e34?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af采集数据后流程的运行效果https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d82219af8c676e69ae3d82aa5c278903bf182dd704a70bd4813c5e2bbe1fe037d349b7330612e248194cf45718c973510?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af这样做的好处是可以为每张表单独配置处理的分支https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde5fe84c8a8a0d941845a9213274799c6ee75b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d8e63107301917b6856087dd5e8f82ba4fca97265303de9bbb61887d066f366b7501362cf02960d4f3f7b3228700ac3ae?tmpCode=cef0c6d0-5415-4476-8979-94a17a7288af四、最后以上便是使用ETLCloud进行实时监听的展示，ETLCloud作为一款数据集成平台，在CDC数据集成方面有着可视化的面板配置，非常易于使用，配置过程中有着多种方式以及大量组件可以针对性做各种个性化的数据处理工作，还有着完善的日志监控方便用户进行错误排查。除了ETL工具，我们还可以通过不同的方式进行数据同步和监听多个库表工作，以此来提升工作效率。
页:
[1]