Delete or change records in ETL

问题

I have a table over which I have built an ETL service. Goods records (arrival / departure) go to the table. I have done that my table will be erased. When the item identifier arrives in the database for the second time, both records are deleted.

label   cost   time
x2       29    14/5/2020 01:00:00
x3       20    14/5/2020 01:02:00
x2       29    15/5/2020 03:12:02

Now ETL service remove records (every 30s):

label   cost   time
x3       20    14/5/2020 01:02:00

I delete it using the function:

with todelete as (
      select *, count(*) over (partition by label) as cnt, ROW_NUMBER() over (partition by label order by time DESC) as r_number
      from Table1
     )
delete from todelete
    where cnt >= 2

And another problem gets in the way. And when it comes to the table, which only means a change in price.

Variant 1:

label   cost   time
x2       29    14/5/2020 01:00:00
x3       20    14/5/2020 01:02:00
x2       30    15/5/2020 03:12:02

Now "delete function" and

My goal:

label   cost   time
x3       20    14/5/2020 01:02:00
x2       30    15/5/2020 03:12:02

I don't know how to treat both of these things in a delete function.

回答1:

To meet both your requirements you should check for change in cost like below

 ; with todelete as (
      select *, 
           count(*) over (partition by label) as cnt, 
           lag(cost) over (partition by label order by time ASC) as lastcost
           ROW_NUMBER() over (partition by label order by time ASC) as r_number
      from Table1
     )
delete from todelete 
    where cnt > 1 and r_number between 1 and (cnt/2)*2 and  cost=ISNULL(lastcost,cost)

来源：https://stackoverflow.com/questions/63830172/delete-or-change-records-in-etl

标签

sql

sql-server

ssis

etl