VDM incorrectly parses and loads data that hasn’t happened yet

Products

Visitor Data Mart

Issue

One the most recent successful load of a VDM profile, no further data will be loaded into the profile for a period of time. The profile will not be failing, it will simply stop loading any new data.

Cause

There can be two causes to this:

1) The parse of logs caused the database to think that the watermark is at some point in the future.
2) There is an entry or there are entries in the log file in the future, causing the high water mark to be set in the future.

The result of this is that ever parse since this issue occurs, will result in the parser deleting the log files instead of parsing them as it thinks the data has already been parsed.

Resolution

To resolve this, there are a number of steps that need to be taken:

1. Check the hitload table to verify that indeed this issue is occurring. You will see load that will have an end time of some point in the future if this has occurred.
2. Check the log files for the day and around the day in question that the issue occurred on. If there are hits that show up in the future, these hits will need to be removed from the log prior to going to step 3
3. The following steps need to be performed in order to get the parser and loader working again from the day the issue occured

• The log needs to be trimmed to the exact last watermark to the exact point of the latest log in the EDB Source Folder
• Once the logs are copied in, the following similar scripts need to be run in order to remove
• Invalidate the data load from the bad parse: (note that the DataLoadID and HitloadID will be different for every database)

• ?Make the parse a “Bad parse”

Once these are run and the logs get put back into place, the profile can start back up again. It will pick up the logs and parse them for the real watermark forward.