Custom Logs

Introduction

In CMX version 3.22.0 - b3211 the Custom Logs were introduced, providing the possibility for the user to create their own logfiles for any value - also strings - which CMX produces in a webtag. This called for a procedure in CumulusUtils to handle those logs. In version 6.21.0 this was realised. This Custom Logs procedure is based on the External Sensors.

Goal

The CustomLogs module's goal is to chart webtags and display its current realtime value in a table left of the chart.

IMPORTANT

NOTE: The formatting of the datafiles is important as the Custom Logs are basically free format: you can put into the file any text and numbers with any separator you wish. CUtils will NOT handle that Free Format in general. The user has to take care the information in the resulting datafiles is as follows:

1) There MUST be at least one space or your local separator between the webtags; 
2) There cannot be any plain text in the logfile
3) The date/time must be untouched i.e. the format is the local date format e.g. dd-mm-yy;hh:mm;

If there is free text in the definition of the log, the CustomLogs module will ignore that and parse only the webtags which result in a number.

NOTE: Be aware that for every webtag an entry in the JSON datafile will be created for the period of number of hours you have defined for the charts. This may lead to large datafiles being transferred every interval, data duplication (data already present in the standard datafiles of CMX or CUtils) and high memory use and performance of your server while displaying the charts may be the result. So the user is warned to think about what he really wants to log and view.

NOTE: Webtag values are the actual momentane values when asked at CMX. Be aware that a webtag value is NEVER an average of the period before it. That is also not the case for the DAILY values where it would maybe be kind of natural to expect such value. A better name for Daily in this case would be the Midnight value.

NOTE: When you define or modify a CustomLog in CMX, you have to run CUtils with CustomLogs (or Website) as argument because there is no connection between CMX and CUtils: running CUtils will fetch the info from CMX and update the website accordingly. It will reset the JSON and a full dataset will be transferred to the webserver. However: if a new CustomLog did not yet contain any data, you will have to run CUtils again when it has. If you do not, the JSON will not adhere to the new definition and the PHP upload will not be able to do incremental uploads which will produce errors in the logfile. This should not be an issue for FTP uploads which are not incremental. Also note that the frequency for the CustomLogs has no connection with the Internet Interval with which the data on the webserver (the JSONs) are updated. That may give unexpected effects for you.

Dependency

Like the Extra Sensors module the Customs Logs module is dependent on the ChartsCompiler which means that the file CutilsCharts.def must be present in the utils directory. This may contain a user made charts definition but for a starting user it will be easiest to just copy the file CutilsCharts-default-for-use.def from the distribution and rename it to CutilsCharts.def. Make sure that your CMX configured datafiles and the charts definitions agree, if they don't than you won't see data lines or/and get errors in the console.

If the user has already run Extra Sensors module, the file CutilsCharts.def will contain the line

 ; ExtraSensorCharts

And everything below these lines will be replaced by generated ExtraSensors charts (up to but not including the following CustomLogs demarcation line).

A similar section will be created for the Custom Logs charts with the CustomLogs demarcation line:

 ; CustomLogsCharts

and the parameter

 UserModificationCustomLogsCharts=true 

will provide the possibility to modify the charts (default is false).

Again: be aware that making the modification of the charts manual will no longer overwrite. If you lose track and want to start over again with automatic chart generation set this parameter to false.

Translation/Language

The translation of of this module is accommodated via two sections in the CUstringsXX.ini file: Compiler and CustomLogs analogue to the Extra Sensors.

You will find all translation tags for the plottable parameters in the section Compiler. This is analogue to all standard plot variables. Normal terms to translate and the fixed chart title section you will find in the CustomLogs section.

Operation

1. CumulusUtils must be run once with the command CustomLogs (fastest) or Website to create the required parameters in the section [CustomLogs] in the inifile.
2. The module requires that the user has Custom Logs logging enabled in CumulusMX (Define and ectivate Custom Logs) AND the user must set the parameter CustomLogs in the cumulusutils.ini file to true. This last parameter makes it possible to have CustomLogs operational in CMX but not display them on the web (if you are generating the website).
3. When configuring the CustomLogs make sure you use meaningful names for the individual logs. This makes it easy to interpret the output as the names are used for the menu and the datapresentation.

For the Website Generator this means everything is now automatically available in the menu of the website.

The customlogsrealtime.txt file needs to be processed in the CMX Extra Webfiles utility and send to the webroot on the realtime frequency.

Using as a module requires some additional editing to make use of the table with the current values (Note the libraries as well). The charts are always available (see outputs). It is advised not to use this module outside the context of the Website Generator but skilled users can use the module as the basis for their own website (as has been done with AirLink and other modules). If you only want the charts then life is more easy and the ChartsCompiler makes it possible to combine virtually every plotparameter with another plotparameter (of RECENT or EXTRA type)

To receive the data for the CustomLogs you have to run CumulusUtils UserAskedData. It would be logical to do this on the normal interval frequency on which all other chart data are generated by CMX and by the requirements of the ChartsCompiler and the AirLink module.

The commands for the above named steps are:

 utils/bin/cumulusutils.exe CustomLogs         => this command creates the outputfiles. 

or:

 utils/bin/cumulusutils.exe website

and:

 utils/bin/cumulusutils.exe UserAskedData                => this command produces the CustomLogs datafiles with only those data required for the present and activated CustomLogs.

NOTE: UserAskedData is typically used in an interval through the scheduler. See also the AirLink and the ChartsCompiler
NOTE: CUtils creates its own JSON data files for the CustomLogs based on the CustomLogs defined by you.

Output

The CustomLogs module has multiple outputs:

1. customlogs.txt => contains the activation code for the module, activated by the menu choice
2. customlogscharts.txt => contains the charts which go with the extra sensors (analogue to the trend charts produced by CMX with the same periodic generation and the same period shown)
3. customlogsrealtime.txt => contains the webtags for the current value of the Extra Sensors used. Has to be entered in Extra Webfiles as source to be processed and FTP'd to the webroot at realtime.
4. customlogsdata.json => contains all data for the extra sensors specified (by the command utils/cumulusutils.exe UserAskedData).

The user can optionally create or modify the charts himself by setting the parameter UserModificationCustomLogsCharts to true and subsequently edit CutilsCharts.def.
The outputfile for the charts MUST always be customlogscharts.txt so the first chart in the CustomLogs section ends with: EndChart Output customlogscharts.txt but the user may change this with manual modification.

Inifile Parameters

The CustomLogs module has the following inifile parameters (see cumulusutils.ini):

 [ExtraSensors]
 CustomLogs=false                        => If false (default) the module is skipped, no menu entry will be made
 UserModificationCustomLogsCharts=false  => If true the CustomLogs charts will not be overwritten in CutilsCharts.def and give the user the possibility to create his own charts
 DoneToday=2023-10-16T12:41:00           => Exists and has a value when the JSON datafile(s) have been uploaded at least once. In case of PHP upload, removing this parameter makes sure next upload will non-incremental.