Products
Webtrends Analytics 9.x
Webtrends Analytics 8.x
Cause
The LowestEndingTimeTimeout setting is a safety feature built into the application for profiles analyzing data sources from web server clusters for which multiple log files with divergent closing time stamps increase the chance of data loss. When all prerequisites are met, this feature is intended to halt analysis at the earliest closing time stamp among the most-recently generated log files in all data sources, thereby preventing the profile’s “high water mark” from moving forward beyond this point until either new log files are available for the data source with this earliest closing time stamp, or the number of hours specified in the LowestEndingTimeTimeout setting have elapsed.
Example:
A profile configured to use web server clusters as a data source has two data sources applied to it, one for each of a pair of web servers. Each data source path specifies the log file repository on the respective web server. The following events take place:
Day 1 – Analysis 1
Log files are available on Server 1 with a closing time stamp of 2024-03-01 11:59:59
Log files are available on Server 2 with a closing time stamp of 2024-03-01 11:59:59
Analysis completes successfully and the profile’s high water mark is set to 2024-03-01 11:59:59.
Day 1 – Analysis 2
Log files are available on Server 1 with a closing time stamp of 2024-03-01 23:00:00
Log files are available on Server 2 with a closing time stamp of 2024-03-01 23:59:59
With the default LowestEndingTimeTimeout setting of 24, the application detects a gap of less than 24 hours between closing time stamps of the most recent log files in each of the data sources. Analysis completes successfully and the profile’s high water mark is set to 2024-03-01 23:00:00, stopping at the earliest closing time stamp. Unless a user initiates a manual analysis, the additional hour of data from Server 2 will be analyzed during the next scheduled analysis job when, presumably, Server 1 will have log files for this time frame as well.
Day 2 – Analysis 1
Log files are available on Server 1 with a closing time stamp of 2024-03-02 11:59:59
Log files are available on Server 2 with a closing time stamp of 2024-03-02 11:59:59
Analysis completes successfully and the profile’s high water mark is set to 2024-03-02 11:59:59. The additional hour of data from Server 2 that was not analyzed in the previous scheduled job has now been analyzed along with data for the same time frame from Server 1. No data is missing, everything is current.
Day 2 – Analysis 2
Log files are available on Server 1 with a closing time stamp of 2024-03-02 23:59:59
Log files are available on Server 2 with a closing time stamp of 2024-03-02 14:00:00
Analysis completes successfully and the profile’s high water mark is set to 2024-03-02 14:00:00, stopping at the earliest closing time stamp. As the gap between the two data sources is approximately ten hours, and under the default LowestEndingTimeTimeout value of 24 hours, analysis does not proceed any further.
Day 3 – Analysis 1
Log files are available on Server 1 with a closing time stamp of 2024-03-03 11:59:59
Log files are available on Server 2 with a closing time stamp of 2024-03-02 14:00:00 (No new log file data is available.)
Analysis completes successfully but does not advance beyond the previous high water mark. No additional log file data can be analyzed until newer log files are available on Server 2, or 24 hours have elapsed. At the time of this analysis, 22 hours have elapsed since the log file data from Server 2 was included in the analysis.
Day 3 – Analysis 2
Log files are available on Server 1 with a closing time stamp of 2024-03-03 23:59:59
Log files are available on Server 2 with a closing time stamp of 2024-03-02 14:00:00 (No new log file data is available.)
Analysis completes successfully and the profile’s high water mark is set to 2024-03-03 23:59:59, stopping at the earliest (and only) closing time stamp. Approximately 34 hours have elapsed since the already-analyzed log file data from Server 2 was included in the analysis, and as a result, only the log files on Server 1 will be analyzed from this point forward. If the log files from Server 2 become available at a later date, only entries with time stamps that fall after the profile’s high water mark will be included in analysis. Until that time, analysis data will represent only half of the visitor traffic for the web site. In this situation, the only options to recover the lost data are to either reanalyze the profile or to restore the profile from a backup of the profile pre-dating the period when Server 2 did not have log files available, and then let the profile analyze forward.
Resolution
To avoid a reanalysis and potential loss of data, the following prerequisites must be met in order to invoke the LowestEndingTimeTimeout setting.
Define the LowestEndingTimeTimeout value
Decide what the appropriate value for LowestEndingTimeTimeout should be in this configuration. While 24 is the default value, some environments may be better served by increasing this to 72 or higher to account for extended periods of time such as weekends or holidays. To modify this value for all profiles:
1. Navigate to the following location: \Webtrends\storage\config\wtm_wtx\
2. Edit the following file in a text editor: wtm_wtx.ini
3. Locate the [profile] section and modify the following line: LowestEndingTimeTimeout = 24
4. Change to the desired value and then save the change.
To modify this value for an individual profile, edit the profile and under the “General” tab, in the “Profile File Name” field, note the value for the profile’s globally unique identifier (guid).
1. Navigate to the following location: \Webtrends\storage\config\wtm_wtx\datfiles\profiles
2. Edit the .wlp file.
3. Locate the [profile] section and add the following line: LowestEndingTimeTimeout = …where is the number of hours for this setting on the profile.
4. Save the change.
New profiles inherit the global value so if the global default is left at 24 and a new profile should use a different value, first create the profile, then modify the .wlp file afterward. It may be preferable to select the option to “Disable Analysis Job” under the “General” settings at the time of creation.
Specify the Data Sources
Navigate to “Application Settings” > “Data Sources” and create a separate data source for each of the servers. Each data source points to the entirety of a single server’s log files. It is acceptable to add multiple paths on separate lines.
Configure the profile
If a new profile is being created, select “New” under “Web Analysis” > “Reports & Profiles” and select the check box for “Advanced profile options” and then proceed through the Profile Wizard. Under “Site Configuration” select the option for “My web site is on multiple physical machines” and continue through the Profile Wizard to “Data Sources.” When “My web site is on multiple physical machines” is selected under “Site Configuration” the option to create a new server entry under “Data Sources” becomes available. Select “New” and enter the IP address or name of the server(s) where the data source is located and then select one data source from the list below that corresponds to the designated server(s). When all servers have been added and matched to data sources, complete the Profile Wizard and save the profile.
For existing profiles, edit the profile and under “Analysis” > “Site Configuration” select “My web site is on multiple physical machines” and then navigate to “Analysis” > “Data Sources” and create new server entries as needed, selecting one data source per server. Select “Save” to save the changes.
While not a requirement, it may be desirable to ensure the following settings are applied to this configuration: IgnoreMissingServerLogfiles
The IgnoreMissingServerLogfiles feature, when set to true, tells the application to proceed with the analysis in the event that the expected log files are not present in the data source. When set to false, analysis will not continue if no log files are present in any of the data sources. If log files are expected, but not yet available, the LowestEndingTimeTimeout feature delays analysis until log files are once again available. The recommended setting for IgnoreMissingServerLogfiles in a configuration using LowestEndingTimeTimeout is false. For more information about IgnoreMissingServerLogfiles, refer to KB How do I configure Webtrends to continue analysis if log(s) aren’t available?
Date Macros
If date macros are used in data source path definitions in conjunction with LowestEndingTimeTimeout, date macros spanning a range less than the LowestEndingTimeTimeout value can omit log files that would have otherwise been included. For example, if LowestEndingTimeTimeout is set to 96 hours and date macros specify a date range of only three days, log files which become available beyond the third day will not be included in the data source. If date macros are used, it is recommended the date macros include date ranges which exceed the equivalent number of days expressed in hours by the LowestEndingTimeTimeout value. For more information about date macros, refer to KB How do I set up a datasource in Webtrends Analytics Software?
More Information
The same type of configuration can be used when performing analysis for multiple, discrete web sites. As stated above, add new servers and a single corresponding data source for each server of a web site to the profile, then save the change.