Sqoop Incremental Import

后端 未结 8 1611
别那么骄傲
别那么骄傲 2021-01-30 15:27

Need advice on Sqoop Incremental Imports. Say I have a Customer with Policy 1 on Day 1 and I imported those records in HDFS on Day 1 and I see them in Part Files.
On Day 2,

8条回答
  •  庸人自扰
    2021-01-30 16:04

    Step1 : The entire table is imported. This will be available as part-m file in your specified HDFS location (say /user/abc/def/part-m-00000) Step2 : Only the incremental records are imported. This will be available in another location (say /user/abc/def1/part-m-00000)

    Now that both the data are available, you can use the sqoop merge option to consolidate both based on the key column.

    Refer to the below doc. for more details

    https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_literal_sqoop_merge_literal

提交回复
热议问题