如何通过ETL把StarRocks中的数据同步到数仓
在数据驱动决策的时代，企业对于数据处理和分析的需求日益增加。ETL作为数据处理的核心环节，扮演着将原始数据转化为有价值信息的关键角色。而StarRocks作为一款极速全场景MPP（Massively Parallel Processing）企业级数据库产品，凭借其创新的架构设计与卓越的性能表现，正成为企业实现高效数据处理与实时分析的优选方案。那如何通过ETL工具把StarRocks中的数据同步到数仓呢？接下来我们通过实操演示下过程。一、StarRocks数据同步到Doris演示新建数据源创建StarRocks源数据库：进入数据源管理选择新建数据源，在数据源中找到StarRocks进行创建。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0dc1c34aa9260b3c7481f58341c111f3edae80be644b9365c5e639db4153b0a858f652765c06851f5a31848e28ced537ce?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1填写StarRocks相关配置https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d3b53009c48601f3ee5f94e0dd63b8a22b06a6f928704a17937bcde02b4aa20a00b6ab13f8552df7da0a0797189bcfd59?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0de1b70ccccf24b8b6f1107d165af439c50511e596f51e15c39461912ff20dedbc96e67f7cfed83cb8bcb03efae328c241?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1新建数据源创建Doris源数据库：Doris数据源创建步骤和上述相同https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d1ace3178a4e9505a235f1e9d4dfc3597c0d32d9fb00cd68d45837d5093d0444daa8db3943d9751d3db6439fa4e7d4331?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1新建流程https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0daba574a945976e41ce34d69996879e0a9d473e28fd8a98f77d9be630c79167e59843b7ef521f8e581337e86b5484ef44?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1新建流程，在流程中配置库表输入组件和doris输出组件，没有组件的需要前往官网购买。库表输入用于读取StarRocks数据，Doris输出用于往Doris中同步数据。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d130fad487eb80085fc727031fd675b25e83d99f5a5f86856c8834c73bc0e6463ec5121dd2a2584644d17dc54c35c5919?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1配置库表输入组件，只需选择刚才创建的数据源和数据源中表。当前表中有30万条数据。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0dc4813196551de50982c6105624618dc28b897bdbc92b4caef7aacc159a1cb437bbf108ca8c793d7cd15100ddb4b8427f?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1选中表后会默认生成查询语句，也可以更具需要更改语句。后续的输入字段也会自动识别。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d8aaea2aa4e4c368792528201726e50d96c38e60667dae76fd8173b3d691e034a9ba8619e500002e85041a2bbb5b9d13b?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d3a7ced29c5772a2a431342d9d2b1165389a04d488076a2180b9a926607e79594c01147b102b6e8fd50fe90c46b42e5b8?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1配置Doris快速输出组件，同样的Doris选择数据源和目标表。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d4d760f6d709fd6c9e502e713048ef095cb069bdbd4a68a5577572bc6e215d7b15d0cccff74215ecb5f49936cf4b0ac0b?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0dfff7bb7417e4ff7b296aefd09b5f8114465195f435e78502b1156812f065ca3b09aaee4ce9bbabae2179a03ef8d91278?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1同时使用了自动建表功能在目标端Doris数据库中自动创建表。https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d67b37aec8ae5f1b40fd93ce9cbca88e53c934db7d4c6f973fbb1ef300cffda94f44624745799b1d3d15100ddb4b8427f?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1在路由线中开启5个并发线程优化同步速度https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0deeb14c25b92dc341669bba2e16214f70b7261eca848d40430aba63f68774e915416cbc3784b98d4532be85da2889710d?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1执行流程并查看结果https://alidocs.dingtalk.com/core/api/resources/img/5eecdaf48460cde56caaf215f75d7c9ff4a0dcea308c585975b8339e1c4c2483ced7199cdd5c984339e8703ac5556d0d7fdfeb4ed86d939425b1e0f556faf4d38bfb23bb367bd94cfbbde890a50ed54fd8e9b2d8c9062e5494cf45718c973510?tmpCode=ca92c41a-3e2f-470e-b1f9-89bb83b938b1
页:
[1]