Hi all,
I am developing an ETL wherein the requirement is to do an incremental load and at the same time, if there is a record that got deleted in the Source delete it from the destination too, makes sense.
The approach am doing is, pick data from the SRC and Destination, pass it onto a Merge join component, do a Full Outer join, then pass the rows to a conditional split. Newly Added records and updated records I can handle, how do I handle the Deleted records?
Am I correct in the way I am doing or there is something better to handle this?
Thanks in advance
MShetty wrote:
Hi all,
I am developing an ETL wherein the requirement is to do an incremental load and at the same time, if there is a record that got deleted in the Source delete it from the destination too, makes sense.
The approach am doing is, pick data from the SRC and Destination, pass it onto a Merge join component, do a Full Outer join, then pass the rows to a conditional split. Newly Added records and updated records I can handle, how do I handle the Deleted records?
Am I correct in the way I am doing or there is something better to handle this?
Thanks in advance
That sounds like it will work. You basically need to compare the source and destination. Any records that are in the destination but not the source have to be removed - it sounds as if that is what you are attempting.
They can be deleted using an OLE DB Command. or you can push the records to be deleted into a temporary table and delete them using an Execute SQL Task.
-Jamie
|||
Hi Jamie,
Thx for the comments, I started working exactly the same way, but now am into an issue here. I am starting with two small tables as the first step. The table has this schema.
UserId (int) (PK)
UserName (varchar)
IsActive (bit) i am taking two DFT's one each to the two Databases and the selected records are passed onto a Merge Join. UserId is the join key here. Now the records that are having a match already are returned from this component right ? If I make the join type as a Ful Outer JOin and pass on to conditional split transform, how do I get the records that are in Destination DB but were deleted from the Source, I am just getting the newly added records and the updated one's. Please guide..
Thanks a lot
|||I don't think you need a fullt outer join, just a left join will do it (with the incoming data on the left input).
Described here:
Get all from Table A that isn't in Table B
(http://www.sqlis.com/default.aspx?311)
-Jamie
|||
Thx for the inputs Jamie.
My problem of deleting the records in the destination that were removed from the Source DB was solved by using an IsNull (SrcTable.PK) in a conditional split and then direct those rows to an OLE DB Command Component.
