What are some ways to perform incremental loads and extract data?
I’ve had to perform incremental loads when extracting data to populate a data mart. To do this I wrote extraction stored procedures that were executed as a SQL task within my Data Transformation Service (DTS) packages. Our source data tables were designed to allow this to happen quite easily. Here are the columns in each source table that allowed us to easily perform an incremental load/extract: Identity Column as PK or Unique Constraint (int column) This column allowed us to see if a row in the source was new or already existed by comparing the source ID value with those that exist in the destination. LastUpdateDate (DateTime Column) This column was automatically updated by our application. If we found that our row was an existing row (based on the identity column value above) we were able to compare the source LastUpdateDate column with the DestinationUpdateDate column to see if the row’s data changed since the last extraction. To make things easier for us, we created a table that he
