Delta Management
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Delta Load Management Framework Overview
CAF
and SAP BW integration supports delta loading for DataSources created
by entity and application service extractor methods. When working
with applications with large data volumes, it is logical to prevent
long loading times and unnecessary locks on the database by only
loading new or modified data records into SAP BW.
Features
Generic
delta management works as follows:
1. A data
request is combined with particular selection criteria in an
InfoPackage and is to be extracted in delta mode.
2. The
request is sent to the source system and then received by the SAPI
(service application programming interface) request broker.
3. The
generic delta management is initiated before the data request is
transferred to the extractor corresponding to the DataSource. This
enhances the selection criteria of the request in accordance with the
update mode of the request. If the delta-relevant field is a
timestamp, the system then adds a time interval to the selection
criteria.
Delta
management can take the lower limit from the last extraction. The
upper limit is taken from the current time. For example, date of
application server minus a safety margin (SY-DATE,
SY-TIME).
4. The
enhanced request is transferred to the extractor. The update mode is
‘translated’ by the generic delta management into a selection
criteria. For this reason, the update mode is set first to full.
5. At the
end of the extraction, the system informs generic delta management
that the pointer can now be set to the upper limit of the previously
returned interval.
You
can find a description of this transfer process in the figure below.
Structure
Delta Fields
The
delta-relevant field of the extract structure meets one of the
following criteria:
●
Field
type is timestamp.
○
New
records that are to be loaded into BW using a delta upload each have
a higher entry in this field than records that have already been
loaded.
○
The same
criteria applies for new records as in the case of a timestamp field.
●
Field
type is not
timestamp.
This case is only supported for SAP Content DataSources. At the start
of delta extraction, the maximum value to be read must be returned
using a DataSource-specific exit.
You
can use special data field to achieve more reliable delta loading
from different source systems. They are integrated into the delta
management framework. They are:
●
Safety
interval upper limit
●
Safety
interval low limit
Safety Interval Upper Limit
The
upper limit for safety interval contains the difference between the
current highest value at the time of the delta or initial delta
extraction and the data that has actually been read. If this value is
initial,
records that are created during extraction cannot be
extracted.
A
timestamp is used to determine the delta value. The timestamp that
was read last stands at 12:00:00. The next data extraction begins at
12:30:00. The selection interval is therefore 12:00:00 to 12:30:00.
At the end of the extraction, the pointer is set to
12:30:00.
This
transaction is saved as a record. It is created at 12:25 but not
saved until 12:35. As a result, it is not contained in the extracted
data and the timestamp means the record is not included in the
subsequent
extraction.
To
avoid this discrepancy, the safety margin between read and
transferred data must always be longer than the maximum time the
creation of a record for this DataSource can take (for timestamp
deltas), or a sufficiently large interval (for deltas using a serial
number).
Safety Interval Lower Limit
The
lower limit for safety interval contains the value that needs to be
taken from the highest value of the previous extraction to obtain the
lowest value of the following extraction.
A
timestamp is used to determine the delta. The master data is
extracted. Only images taken after the extraction are transferred and
overwrite the status in BW. Therefore, with such data, a record can
be extracted more than once into BW without too much difficulty.
Taking
this into account, the current timestamp can always be used as the
upper limit in an extraction and the lower limit of the
subsequent extraction does not immediately follow on from the upper
limit of the previous one. Instead, it takes a value corresponding to
this upper limit minus a safety margin.
This
safety interval needs to sufficiently large so that all values that
already contain a timestamp at the time of the last extraction, but
which have yet to be read (see type 1), are now contained in the
extraction. This implies that some records will be transferred twice.
However, due to the reasons outlined previously, this is irrelevant.
You
should not fill the safety intervals fields with an additive delta
update, as duplicate records will invariably lead to incorrect data.
It
is not necessary to set safety intervals for DataSources used in CAF
and SAP BW integration.
,,
