Calculate Missing Values: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
m (Sfws moved page Correcting Missing Values to Calculate Missing Values without leaving a redirect: More descriptive title)
m (→‎CreateMissing.exe: That sentence still read wrongly!)
(31 intermediate revisions by the same user not shown)
Line 1: Line 1:
=Introduction=
{{Template:WorkInProgressBanner}}{{Template:Version badge Mx}}{{Version badge 1}}This page applies to both the legacy Cumulus 1 and to MX.  It is intended that this page will cover all the different ways in which you add back any missing data. This means it should cover:
* Data that you want to add from another system for a period when you were not running Cumulus
* How Cumulus captures archive data if it is available for a period when Cumulus was stopped
* Derivatives that have been added since particular log files lines were stored
* Solving problems when a line was not successfully written to the daily summary log file


As Cumulus has developed, it has included the ability to do more and more, so for many [[:Category:Log_Files]], the number of fields that are stored can vary for different lines, depending on which version the user was running when that line was created.
[[Category:Cumulus MX]][[Category:Cumulus Files]][[Category:Files_with_Comma_Separated_Values]][[Category:Cumulus 1]]


Cumulus MX,  release 3.9.2 - b3097 (released 7 Dec 2020), added the ability to display historic charts, these read old lines from the log file, so there is now a demand for all the fields that are available now to be available for historic data as well.
=How Cumulus Works=


This has been created to explain how derived values are calculated, and how they can be added into older lines in the log files.
{{TOCright}}
 
The way that Cumulus works is that:
# It reads what we can call '''source values''' (defined below) from your weather station (based on measurements from sensors being transmitted in some way by the station)
# It calculates what we can call '''derived values''' which may be either the source re-expressed in different units, or calculated by combining two or three source values to get a new derivative
# It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various [[:Category:Ini_Files|period tracking files]]
#* You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the [[Correcting_Extremes]] page.
# It periodically stores the spot (current) derived values in a collection of [[Monthly_log_files]], (and so when you move to a different device, or upgrade to a new release, then providing you copy these files to the new location you will not lose any data)
#* for MX these files include  [[Standard log files]],  [[Extra Sensor Files]], and [[Air_Link_Log.txt]], at time of writing
#* for Cumulus 1 these files are [[Speciallog.txt]], [[Standard log files]], and [[Extra Sensor Files]], for all version 1.9.3 and 1.9.4 releases
# At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]]
 
==Reading archive data==
 
If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.
 
This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can '''rewind''', by stopping Cumulus, optionally replacing files in [[Data folder| '''data''' folder]] with earlier copies of those files from [[Backup folder| '''backup''' folder or ''backup/daily'' folder]], and restarting Cumulus.  Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.
 
* For Cumulus 1, to stop Cumulus, you select [[Cumulus Screenshots|Exit]] from the main menu.
* For MX, how you stop MX depends on your device, and whether running as a service, please see [[MX on Linux]] or [[MX on Windows OS]] pages as appropriate for advice.
Put simply, Cumulus stores the latest time it successfully read data from the weather station in [[Today.ini|today.ini]]. When Cumulus is restarted, if it is possible to read the historic data from the weather station, then any entries between the time stored and the current time will be read.
 
With Cumulus 1, there is some dependence on weather station type, but usually two passes are made through the external logging memory, the first pass investigates what records are available in the weather station, by reading backwards in time, and the second pass works forward in past time reading and processing those records.
 
With MX, again there will be some dependence on weather station type, and the process has not been documented by the developer; it appears just a single pass is made.
 
=Importing data from other systems for periods when Cumulus was not running=
 
<big>This section needs more work on it</big>
 
Essentially, what we need to do, is to compare the format of the data we have available against the format of the relevant Cumulus file.
 
==CSV outputs from software like EasyWeather==
 
You might want to read [[EasyWeather_Format#Transferring_past_observations_from_EasyWeather.dat_to_Cumulus|Transferring_past_observations_from_EasyWeather.dat_to_Cumulus]], although that is now an obsolete article, Easy Weather has changed, and Cumulus MX is different from the original Cumulus described there.
 
The CSV output from such files has fields in the wrong order, may not match our Cumulus units, and does not have a match for each of the fields we need in our Cumulus - see [[Standard_log_files#Importing_pre-Cumulus_data|importing into standard log file fields]].  The solution is to use a CSV file editor, or a spreadsheet, to move the fields into a better order, to apply a formula where necessary to change units, and create any missing fields. 
 
Cumulus does not allow for "null", field not available, except in that it can allow shorter lines with just the fields applying to an earlier release.  For MX, [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases for the [[Standard_log_files]], and that utility can generate the necessary [[Dayfile.txt|dayfile.txt]] lines.
 
On the Support Forum - Dane - offered a translation service - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=13&t=250 here], but the last post is May 2018, so I am unsure whether it is still available as I type this in July 2021.
 
==Weather Display==
 
Please see [[Software#Weather_Display_Converter|Software page - Weather_Display_Converter]].  When you run this routine, it brings up a screen where you select the units that you want the Cumulus [[Standard_log_files]] it produces to use, as the units that weather display uses for the export are fixed.
 
The routine was written for Cumulus 1, so will not produce all the fields that MX uses, but [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases, as well as producing [[Dayfile.txt|dayfile.txt]].
 
==Weather Link log file==
 
There is a routine, see [[Software#WeatherLink_Converter|Software page - WeatherLink_Converter]] for this.  When you run this routine, it brings up a screen where you select the units.  Please note, this utility assumes the units you select on that screen apply both to the file (OR MULTIPLE FILES) being converted and to the (multiple) Cumulus [[Standard_log_files]] it produces.
 
It was written for Cumulus 1, so will not produce all the fields that MX uses, but [[#CreateMissing.exe|CreateMissing.exe]] can calculate some of the derivatives missing from earlier releases, as well as producing [[Dayfile.txt|dayfile.txt]].
 
==Weather Link Live==
 
PLEASE CAN SOMEBODY FULLY DOCUMENT THIS.
 
My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data.  I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.
 
==OTHERS????==
 
PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.
 
Meanwhile, people should ask for help in the support forum.


=Some definitions=
=Some definitions=
To make sense of explanations on this page, you need to understand the terminology used here.
==Source value==
A weather station sends values based on its sensors to Cumulus. If Cumulus reports that value without changing it (an offset and/or multiplier might be applied to convert it to the unit wanted by the Cumulus User), the value is described as a '''source value''' because Cumulus is reporting something that has its source elsewhere.
There is not a single list of what weather values are called "source values", because this varies depending on the weather station, and in some cases, a Cumulus User can ask Cumulus to recalculate a value instead of using what is sent by their weather station.
However, Cumulus does include code that expects a weather station to provide ''a defined minimum set of source values'':
# Current air temperature
# Current Relative Humidity
# At least one wind speed
# Current air pressure (absolute or sea-level)
Cumulus will stop processing any information from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these).
This requirement is a default, but it can be changed:
*For recent releases of MX, this is classified as an advanced setting "No sensor check" (see [[Cumulus.ini]])
*For earlier releases of MX, this is "No sensor check" (see [[Cumulus.ini (MX_3.0.0_to_3.7.0)#Read-only_parameters_in_the_Station_section_.28releases_3.0.0_to_3.6.12.29|here for how to change default]])
*For legacy Cumulus 1, [[Cumulus.ini_(Cumulus_1)#Read-only_parameters_in_the_Station_section|see here for how to change default]]
Cumulus also expects that your weather station can provide:
* A rainfall counter (this could be annual rainfall, or count of rocker bucket gauge tips)
Although the lack of that rainfall counter source value will affect functionality, Cumulus will continue to process other source values that are available.
Some weather stations may also provide one, or more, of these optional source values (not a complete definitive list):
* Dew-point Temperature
* Wind Chill Temperature
* Evaporation
* Sunshine hours
* Solar radiation
* UV index
* Air pollution measurement


==Derived value==
==Derived value==


In general, if there is a rogue value read from a weather station, it will affect not just that reading, but all values that are derived from it. So it is necessary to explain what is meant by derived values.
A dictionary will define '''derived''' as "obtained from a source", and that is the meaning adopted on this page.  Steve Loft (in the Cumulus Support Forum) used the terminology "derived" for two purposes.
# One type of derived value takes a source value, applies any multiplier (may be both first order and second order multipliers) and/or constant that has been defined in calibration settings, and converts the output to the units selected by the Cumulus user.
# The other type of derived value takes more than one source value, applies a standard calculation, and ouputs a new derivative
#* Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file
 
=="Calculate Missing"==
 
This also has two meanings in a Cumulus context:
# If a particular standard log file line has fewer fields than the latest line;
#* '''Calculate Missing''' is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
# If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
#* Please see [[Amending dayfile]] page for full details.
#* '''Calculate Missing''' is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.
 
If you are using Cumulus MX, there is a download [[Software#Create_Missing| linked from here]] that does both of these. There are also editors within the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|admin interface]] for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.
 
If you are using the legacy Cumulus 1 software:
#  For the standard log file meaning above, provided you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] includes a script that produces a HTML form where you specify [[Standard_log_files#Introduction|the log file name]] you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated.  Please note that script was written to run on an old version of PHP that was current at the time the script was written, it will need some editing to work on latest PHP.
# For the daily summary log meaning above, go to the '''Edit''' menu, and select ''Dayfile.txt''.  This brings up an editor with a button labelled "Create Missing", that will not affect any existing line, but can insert missing lines, see [[Amending_dayfile#Create_Missing]].
 
===Accurate or Not?===
 
This Wiki page describes some techniques for calculating and inserting values that are missing from standard log files and from daily summary log file.
 
Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your [[:Category:Ini Files|.ini]] files is missing some fields, these missing fields cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.
 
However, the techniques for correcting rogue values described on the [[Correcting_Extremes]] page, can be used for inserting missing values in the daily and longer period extreme records.
 
For the standard log files, all the fields in any one line relate to the same time, therefore for derived values calculated from other fields in the same line, you should have the same value whether it was calculated when that line was originally stored, or calculated afterwards.  I say '''should''' because the calculation formula is not always the same for all releases, in particular there are differences between how Cumulus 1 and how MX calculate some derivatives.
 
For entries in today.ini, month.ini, year.ini, alltime.ini, and monthlyalltime.ini files, you don't have access to the source values used for the original calculation afterwards.
* The original values are calculated as Cumulus is running
**Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini (and the other files listed) each minute if an extreme happens
** Obviously, dayfile.txt is updated from today.ini, so it is just as accurate
* Any "Calculate Missing" operation, done subsequently, does not have access to old data, it can only look in the spot values that have been logged.
**If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
** Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)
 
=Derived spot values=
 
Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.
 
Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):
* '''Dew point''', a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
** The original legacy Cumulus, and MX, use [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters|different formulae to calculate dew point]], so there is a continuity break if some of your data logs were created by the original Cumulus software and some by MX.
* '''Wet Bulb''', is only calculated by Cumulus 1, not MX
* [[Wind_chill|'''Wind Chill''']], again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
* [[Humidex|'''Canadian Humidity Index (Humidex)''']], [[Heat_index|'''USA Heat Index''']], and [[Apparent_temperature|'''Apparent Temperature''']] are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
** (The implementation of these by Cumulus software is briefly mentioned [[Feels_Like#The_various_ways_to_express_Feels_Like|here]]).
** The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
* [[Feels_Like|'''Feels Like Temperature''']] is calculated by the Cumulus MX flavour only, the actual calculation formula [[Feels_Like#The_various_ways_to_express_Feels_Like|has varied]] in different releases, but use a variation on
* [[Heat/cold_degree_days_and_Chill_hours|'''Heating Degree Days''' and '''Cooling Degree Days''']]; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
**A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.
 
The links above will take you to where the derived values are explained in the [[:Category:Terminology]] pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.
 
 
There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.
* For MX, please see [[MX_Administrative_Interface#Changing_Settings]] and [[Cumulus.ini]] pages for how to find the settings
* For legacy Cumulus 1, please see [[Cumulus_Screenshots#Station|Configuration -->> Station]] menu and [[Cumulus.ini_(Cumulus_1)]] pages
 
=Field Count Variations=
 
When the [[Standard_log_files#List_of_fields_in_the_file|standard data logging]] file was introduced it only had 16 (or fewer?) fields. As time has gone by, extra fields have been added to the file. At time of writing, 29 fields have been in the file since release 3.6.12 (build 3088), and currently the "To Do" database does not include any suggestions that would add more fields.
 
When the [[Dayfile.txt#List_of_fields_in_dayfile.txt|daily summary log]] file was introduced it had 15 fields. As time has gone by, extra fields have been added to the file. At release 3.6.12 there were 54 fields, but at earlier and later releases there are fewer fields. At the last update of this page (release 3.7.0) there were 52 fields. The number of fields in a line of the file might be changed in a future release.
 
When you use Cumulus to edit any of these files, it expects the file to have the number of fields defined in the release you are using. If an existing line in the file has fewer fields, Cumulus can still read it, but Cumulus will add trailing field separators if the file line is edited.
 
Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.
 
=Why do "Calculate Missing"?=
 
Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the [[Correcting_Extremes]] page.  You are unlikely to worry about missing past values.
 
However, if you want to be sure that your all-time extremes, or monthly-all-time extremes, are correct, then [[Correcting_Extremes#All-time_extreme_functionality|this table]] shows how the start date for these extremes varies.  You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".
 
If you are using the [[Highcharts_-_Historic|'''Historic Charts''']] feature introduced from release 3.9.2 - b3097 (7 December 2020), you may notice that these new charts have gaps in available data, and the dates with/without data vary depending on what is being plotted.  You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".
 
 
=How to do "Calculate Missing"=
 
As mentioned earlier, there are a number of options, here are the detailed instructions for each option.
 
==CreateMissing.exe==
 
This utility is only for those who have already installed Cumulus MX:
* It uses some files that are in the MX installation package
* It uses some files that are created when MX is run
 
 
''Please note the developer does not fully describe his utility at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his github page] so the author of this Wiki update cannot guarantee the documentation here is correct''
 
This utility was written by Mark Crossley specifically to insert any missing data:
* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]
** The utility will not run if a file called  '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
* If any of the following fields are not populated in [[Standard_log_files#List of fields in the file|MMMYYlog.txt]] lines, the utility can populate them as it works through that file to work out daily extremes (see calculations in table below).
** [[Wind chill|Wind Chill]]
** [[Apparent_temperature|Australian Apparent Temperature]]
** [[Feels Like|Feels Like temperature]]
** [[Heat index|North American Heat Index]]
** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below
 
 
===Obtaining the Create Missing Utility===
 
There is a download link on [[Software#Create_Missing]] page, unzip to reveal the components in the package, and install all of those in same folder as CumulusMX.exe.
 
If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[MX_on_Linux#chmod]])
 
Please note that "Create Missing Version 1.1.0" can only be used with "Cumulus MX release 3.12.0" and above. It makes use of components not included in earlier MX releases.
 
Earlier "Create Missing" versions can be found at https://github.com/cumulusmx/CreateMissing, and these will work with earlier MX releases, but be aware that Create Missing version 1.0.2 had been tested and was found to have a bug.
 
===Running the Create Missing Utility===
 
The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days).
 
To run CumulusMX.exe needs .NET or MONO to be installed, and because MX is already running, CreateMissing.exe can be run, as it too requires .NET or MONO.
 
# Open up the MX interface in a browser, and navigate to  Settings menu -> Station Settings -> General Settings -> Advanced Options -> Records Began Date
#* The date that is shown there is the date where "Create Missing" will start by default, so if you have MMMYYlog.txt log files with an earlier date, edit the date here, keeping to same format
#* Click '''Save Settings''' button if you have made a change to this date
# Close your browser, and (if using an interactive screen for your computer) open up your file manager, or (if using a terminal session for access to your computer) navigate to your CumulusMX folder
# Navigate to your [[Data_folder|data sub-folder]]
# If there is a file there called '''dayfile.txt.sav''', rename that file to '''dayfile.txt.sav.bak''' (or any other name that does not already exist)
# Now change directory back up to parent folder <code>cd ..</code>, i.e the folder containing "CreateMissing.exe"
#* On Linux operating systems, you need execute rights on that file (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
#* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.
 
Remember, as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.
 
===How the Create Missing Utility works===
 
The utility program will output to any terminal session open and to a file saved in [[MXdiags_folder|MXdiags sub-folder]].
 
This utility program looks in [[Cumulus.ini]] for:
# The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (as mentioned above it can be edited to another date, to include imported earlier data or to exclude data that relates to a former location).
#* That will be the earliest date the utility program processes.
#* However, if a dayfile.txt file exists and that has an earlier date, then "Create Missing" will only continue if you accept that earlier date.
# The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters.
#* This identifies which standard log lines belong to each day by checking against date and time of that line.
# The thresholds for Heating Degree Days, Cooling Degree Days, and Chill Hours
# The starting month for Chill Hours Season
 
This utility program looks in the [[Data_folder|data sub-folder]]:
# If there is a file there called '''dayfile.txt.sav''', the utility aborts
# If there is a file there called '''dayfile.txt''', the utility renames that to ''dayfile.txt.sav''
# The utility creates an empty file, naming it '''dayfile.txt'''
# The utility continues by opening the MMMYYlog.txt file for the earliest date (see above) it is going to process
#* See table below for how contents of this file are read/updated and used to create lines in the new '''dayfile.txt''' file
# Each subsequent  MMMYYlog.txt file is processed in turn
 
===How the utility reports progress===
 
Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month):
<pre>
2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt
</pre>
 
 
 
=== How the utility creates a dayfile.txt line ===
{| class="wikitable" border="1"
|-
!style="background:pink; width:250px" | dayfile.txt field
|colspan="4" style="background:lightgray; width:150px" | Standard log file fields
!style="background:pink; width:400px" | Description
|-
|style="background:pink;"| Daily derivative
|style="background:lightgray;"| Preferred field
|style="background:lightgray;"| First source
|style="background:lightgray;"| Second source
|style="background:lightgray;"| Third source
|style="background:pink;"| (how calculated)
|-
| [[Meteorological_day|date]]
|
| Day-Month-Year
| Hour-Minute
|
| From processing lines linked with that Meteorological day.
|-
|Highest wind [[Wind_measurement#Weather_Stations_and_Cumulus|gust]] speed
| colspan="4" | Cumulus '''Gust''' wind speed
| Stores highest value of that log file field in that Meteorological day.
|-
|[[Wind_measurement#Wind_Direction | Bearing]] of highest wind gust
| colspan="4" | Average wind bearing (in degrees)
| Stores the bearing recorded at same time as maximum value in previous field
|-
|Time of highest wind gust
| colspan="4" | Hour-Minute
| Stores the time in log file line used in two previous fields
|-
|Minimum [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | temperature]]
| colspan="4" | Current temperature
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum temperature
| colspan="4" | Current temperature
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Minimum [[Pressure_Measurement | sea level pressure]]
| colspan="4" | Current sea level pressure
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of minimum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum sea level pressure
| colspan="4" | Current sea level pressure
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum pressure
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Maximum [[Rain_measurement#Rain_Rate | rainfall rate]]
| colspan="4" | [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]]
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of maximum rainfall rate
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total rainfall for the day
| colspan="4" | Total rainfall today so far
| Stores the entry in the last log file field in that Meteorological day.
|-
|[[Average temperature]] for the day
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in minutes obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current temperature" field
# Sum the interval times in step 1 for whole day
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3 divided by the sum in step 4
|-
|Daily [[Windrun | wind run]]
|
| Hour-Minute
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out product of above interval time times "Current average wind speed" field
# Sum the products in step 2 for whole day
# When completed loop, store the sum in step 3
|-
|Highest [[Wind_measurement#Weather_Stations_and_Cumulus|Average Wind Speed]]
| colspan="4" | Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of Highest Avg. Wind speed
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Lowest [[Temperature_(and_humidity)_measurement | humidity]]
| colspan="4" | Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity]
| Stores the lowest value of that log file field in that Meteorological day.
|-
|Time of lowest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Highest humidity
| colspan="4" | Current relative humidity
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of highest humidity
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Total evapotranspiration
| colspan="4" | Evapotranspiration
| Stores highest value of that log file field in that Meteorological day.
|-
|Total hours of sunshine
| colspan="4" | Hours of sunshine so far today
| Stores highest value of that log file field in that '''calendar''' day (i.e. midnight to midnight)
|-
|High USA [[Heat index]]
| Heat Index
| Current relative humidity
| Current temperature
|
| The heat index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
| Time of high heat index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| High [[Apparent temperature]]
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low apparent temperature
| Apparent temperature
| Current relative humidity
| Current temperature
|
| Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low apparent temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High hourly rain
| colspan="4" | Total rainfall today so far
| High hourly rain is a derived value.  Loop through every log file field in that Meteorological day, build up a series of hourly values (total rainfall in this entry minus total rainfall an hour earlier), find maximum of all those hourly values, and store that.
|-
|Time of high hourly rain
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Greatest [[wind chill]] (high wind speed, low temperature)
| Wind chill
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
|
| Wind Chill can be reported by weather station or it can be derived. If the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of greatest wind chill
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Temperature_(and_humidity)_measurement#Cumulus_Calculated_Parameters | dew point]]
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores highest value of that log file field in that Meteorological day.
|-
|Time of high dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low dew point
| colspan="4" | Current dew point
| Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores lowest value of that log file field in that Meteorological day.
|-
|Time of low dew point
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Today's dominant/average wind direction
|
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Average wind bearing (in degrees)
|
| The dominant/average wind direction is a derived value. 
# Loop through every log file pair of fields in that Meteorological day:
#* Calculate increment in X as product of wind speed times sine of bearing, and sum those increments
#* Calculate increment in Y as product of wind speed times cosine of bearing, and sum those increments
# Convert final X and Y coordinates back to a bearing in degrees
|-
|[[Heat/cold degree days and Chill hours | Heating degree days]] (HDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting current temperature from  HDD threshold
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|[[Heat/cold degree days and Chill hours | Cooling degree days]] (CDD)
|
| Hour-Minute
| Current temperature
|
|Loop through every log file pair of fields in that Meteorological day:
# Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out increment in HDD by subtracting CDD threshold from current temperature
# Work out product multiplying result in step 1 by result in step 2, and sum those products
# At end of loop store the final sum
|-
|High solar radiation
| colspan="4" | current solar radiation
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high solar radiation
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High UV Index
| colspan="4" | UV Index
| Stores highest value of that log file field in that Meteorological day.
|-
|Time of high UV Index
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High [[Feels Like]] temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|Low Feels Like temperature
| Feels Like temperature
| Current relative humidity
| Cumulus moving ''''Average'''' of wind speed measurements over a particular period
| Current temperature
| Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
|-
|Time of low feels like temperature
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
|High Canadian Humidity Index or [[Humidex]]
| Humidex
| Current relative humidity
| Current temperature
|
| The Canadian Humidity index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
|-
|Time of high Humidex
| colspan="4" | Hour-Minute
| Stores the time in log file line used in the previous field
|-
| Cumulative Seasonal Chill Hours
|
| Current temperature
| Hour-Minute
|
| "Chill Hours" is a derived value, loop through every log file field in that Meteorological day:
# Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
# Work out if there is increment in Chill hours by seeing if "Current temperature" field is below Chill Hours threshold temperature
# If there is an increment, sum value from step 1
# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day
|}
 
==Using a PHP script on your web server==
 
If you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] includes a script that produces a HTML form where you specify [[Standard_log_files#Introduction|the log file name]] you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated.
 
That might sound a bit technical, so here are some step by step instructions:
#Download the '''processStandardLog.php''' script from [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 here].
#File transfer (or copy for local web server) that script to your web server
#File transfer (or copy for local web server) all the standard log files from the data sub-folder in your Cumulus installation to a suitable holding folder (you may need to create it) on your web server
#Open in a browser '''PATH/processStandardLog.php''' by preceding the file name with the path as defined from the root on your web server
#This loads a web page where you have a field asking you to enter a path and file name for the data log you want to process
#Continue to follow the instructions on the web page
#When it has created a replacement file, you can enter details for another data log, and continue until all your data logs have been processed
#Take a backup of your existing Cumulus installation (you should be doing that on a regular basis anyway, so I will not give instructions here)
#Carefully delete any non-current data log in your data sub-folder that you have a replacement for, and file transfer (or copy back) the replacement data logs from your web server into the local data sub-folder, noting that the file extension will need to be renamed from '''.csv''' to '''.txt'''.
 
==Using the '''data log''' editor provided in MX==
 
<div style="background: LemonChiffon;padding:5px; margin:2px;">
[[File:Crystal Clear info.png|40px]] This document was written for the (legacy) Cumulus 1 software.  It has been updated to cover MX, but that was for a MX release that is no longer latest!
 
When this section was written, the number of lines shown was fixed at a maximum of 10; later releases have given the option to display different numbers of lines, and there may be other changes still to be documented here.
</div>
In the [[MX_Administrative_Interface#The_Data_Log_Viewing_and_Editing_interface|MX admin interface]] go to the '''Data Logs''' menu tab, and select the ''Data Logs'' page.
 
There is a box for selecting the data log you want to edit. Once you have loaded that, the first (up to) 10 lines are shown. Navigation links let you select 'First', 'Previous', 'Next', and 'Last' pages, also a small number of pages can be selected directly.
 
Once you select a line, an '''Edit''' button is enabled, click that and you can manually input the missing values for that line. '''Save''' that edit, and you can select another line. Once you have edited all the lines on that page, you can select another page, and repeat the process. Then you can select another log, and repeat the process.
 
It is a long-winded way to edit, and the MX editor does not even validate what you have entered. An alternative is to edit each log file externally, and you can read how to do that in the "Work around for standard log files" section below.
 
<small>Some readers of the Cumulus support forum will know that a third-party replacement for the MX editor was worked on, but never got incorporated into MX. The idea was to replace the alt_editor software used by Mark Crossley, with a standard HTML form script.  This allowed in-line editing, it allowed the derived values to be calculated and displayed (so you simply decided whether to accept the replacements as suggested for the various fields), and finally it applied some validation to each field to ensure any manual edit inserted a value that was within the allowed range.  The main reason for its rejection from the public MX was the complex way in which different files included in the admin interface interact, and the consequent issue that changes made for this replacement had a knock on effect on other pages in the admin interface. The author could not afford the time to redesign the whole admin interface so the proposed replacement could be integrated.</small>
 
 
==Lack of editor in Cumulus 1==
 
Cumulus 1 provides a [[Cumulus_Screenshots#View_Menu|viewer]] for the data logs, that does not permit editing of the file.
 
On the '''View''' menu, select ''''Data Logs''', then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which.  You cannot do any editing.
 
If you find that this viewer cannot load a data log, it is probably because you ignored the '''read me''' that is part of the Cumulus 1 installation procedure, see  [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]].  If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.
 
Cumulus 1 does not provide any functionality to edit the standard data logs, whether to correct a rogue value, or to add a missing derivative.
 
==Work around for standard log files==
 
An option is to edit the file outside Cumulus using a '''Comma Separated File''' editor, a plain text editor, or a spreadsheet program (like the free open source '''Libre Office Calc''' or the commercially charged for ''Microsoft Excel'').


===Derived spot values===
Note: Cumulus 1 applies an exclusive lock to current standard log file, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while the original Cumulus software is running. A full discussion of the problems with conflicts of access to the standard log file can be found [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 in this support forum topic].


It is important to understand how Cumulus works. Some weather software when it stores the values into a data log will store both the highest and lowest value since the previous log entry. Cumulus only stores the spot values read (or derived) just before that entry is made into the data logs; for details see [[Standard_log_files]] or [[Extra_Sensor_Files]]. The default logging interval is every 10 minutes, although that can be adjusted, this means that there is a high chance that any rogue value read from a weather station will not be found in these data logs, as it is unlikely it happened just before a log line was written.
If you decide to edit the current log outside Cumulus, then remember that, if you leave Cumulus running it will continue to append new lines. Therefore, you either need to close Cumulus while you are doing the edit; or (if you are able to merge two files) close Cumulus while you replace its file with a merge of what you have edited and the extra lines added since you took away a copy to edit.


*For some weather data processed by Cumulus the value sent by the weather station and that used by Cumulus will be different:
It is best practice, to take a back-up copy of all your Cumulus installation before starting any editing. It is also best practice to take a further copy of any file you want to amend, and to do your edit on that copy, so you do not edit any Cumulus file directly. The original full backup will preserve the existing file, so you can regress to it, should Cumulus find an error in your edit.
** This can be because the units used by the weather station, and the units required by the Cumulus user are different (and Cumulus allows the user to select the units in which values will be stored)
** This can be because the user has entered some calibration settings. The exact settings available vary for different weather data, but in some cases a second order multiplier is available (the square of the input value is multiplied by the secord order multiplier), a first order multiplier is available (the input value is multiplied by the first order multiplier), and an offset is available (added to the input value). If you want to understand how this works, please see [[Cumulus.ini#Offsets]] for the details
Also note these log files do not include a header line, and should not be edited to include it. All flavours of Cumulus provide, in the data sub-folder, a file called '''monthlyfileheader.txt''' which contains the headers appropriate to the release you are running, and you can add that temporarily to your spreadsheet if it helps you with editing, but don't forget to delete it before saving the file ready to make it available to Cumulus.
*Other values are derived in the sense that a particular weather station does not output that weather data, and Cumulus calculates it. Here are the ones that Cumulus can calculate:
** '''Dew point''', a weather station might output dew point temperatures, or Cumulus might calculate it from the values that are output (outdoor temperature and outdoor humidity)
**'''Wind Chill''', again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
**'''Canadian Humidity Indes''', '''USA Heat Index''', '''Apparent Temperature''', '''Feels Like Temperature'''; all of these are output by MX (not all in early versions) and some are output by later Cumulus 1 versions (the readings from your weather station used obviously vary depending on derived value).
**'''Heating Degree Days''' and '''Cooling Degree Days'''; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)


You can find many of these explained in the [[:Category:Terminology]] pages of this Wiki, otherwise search online and find them in for example Wikipedia.
Here are some other rules to follow when editing the standard log files:
* You can't edit any log file with a word processor, as they add control characters, and other information, that Cumulus cannot understand.
* Editing is straight forward if you use a specialised comma separated value file editor, such editors will split the content by field so it is easy to ensure you only amend field content and do not accidentally change a field divider, plus these editors will not add additional content to any line as they can cope with the number of fields varying in different lines and don't change encoding.
* If you want to use a text editor, it is best if you choose one designed for computer programmers or developers. Such an editor will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM).
* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected.  Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ...").
* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer).
* Ensure no blank lines are introduced by your editing.
*Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.


===Over time derived values===


This is an attempt to summarise how Cumulus produces derived values by looking for extremes, or cumulative total values over various periods of time:
=General External Editing Rules=
*'''Daily''', one big feature of Cumulus software is that it calculates daily totals and extremes (regardless of whether the weather station can supply them or not), the feature was added because originally Cumulus was designed to match the way that meteorological services reported values for 9 a.m. to 9 a.m. days (and Cumulus can swap to using 10 a.m. in summer just as official statistics do)
** Only daily maximum is stored by Cumulus for those weather values that have a fixed minimum (wind speed, Humidex, rainfall rate, are examples)
** Both daily low and daily high are stored for those weather values where a spot value can vary up and down during the day without a fixed minimum (temperature and pressure are examples
** Cumulative values are stored for some items (e.g. rainfall, Heating degree days, sunshine hours)
** What Cumulus does not do is fully obey professional meterological practice (as a few examples: for a particular date the lowest temperature in the 24 hours prior to 9 a.m. is stored, while for highest temperature it is that in the 24 hours following 9 a.m., for snow records it is whether snow fell on the calendar day i.e. since midnight that determines what is assigned to that date)
*'''This month''', from version 1.2 Cumulus has added the facility to output extremes and totals for the current month to date.
**Note 'month-to-date rainfall' is treated differently, it was introduced in version 1.1 and shown on the main screen in Cumulus 1, it also appears on the "current conditions" sample web page provided in Cumulus instead of the this month sample web page. For MX the admin interface also displays it on the current conditions page.
**Whether a cumulative total, just a maximum, or both high and low are stored by Cumulus depends on the particular derivative just as for daily values
**from version 1.7.5 onwards the original Cumulus counts air frosts and gale days in the current month-to date (it can do it for any period actually), this feature is not in MX
*'''This year''', similar derivatives are maintained on a yearly basis
**Again rainfall is treated differently, the cumululative total is seasonal, you can choose which month it starts counting from, and you can even ask it to add a fixed amount while you are in a particular calendar year (yes the seasonality is ignored for that offset)
**Another seasonal parameter is added, cumulative '''Chill Hours''' are maintained by Cumulus from version 1.9.2 onwards, again you specify the month when the season is to start, but you also can vary the threshold.


[[Category:Log Files]]
*  Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
*  Take another copy and use that for your editing, don't edit the actual file being used by the software.
**  This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
**  It also means that you can go back to the last working copy, you can't upset your "revert" copy.
*  The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
*  Generally, it is easiest if you use either a specialised "Comma Separated Value" file editor or a text editor.
** These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
*  You can use a spreadsheet application, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
** You need to ensure that your spreadsheet treats every column as plain text, don't let it recognise dates or times and convert them into another format, don't let it convert any numeric field into another format
**  If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
* Don't remove any figures from fields where figures currently exist, simply overtype one entry with another entry in same format.
* If your file has previously been edited by the relevant editor in MX, a field that looks empty may actually contain one or two space characters.
* If you are editing a field which was empty previously, remember that Cumulus does not accept the concept of nulls (entering -999 or "Null"), there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted in one field if any subsequent field in same line is not empty.
**Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
**  Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and for up to 6 times it will repeat the last valid value if the station fails to send a value it should provide (normally six successive readings will happen between entries in the standard data log, so repeated values are less likely to affect log files).
* The character (or in a few locales, two characters) used for separating both the day of the month from the month, and for separating the month from the year, must be consistent throughout the whole file (and must not be a single space). Normally, the separator will be either "-" or "/". Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus. Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
* USA date format with month before day of month, and finally year, is not permitted for log files.
* All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
* Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
* Don't add a header line to the file, Cumulus expects all lines to be data lines.
*Be aware that different devices use different line terminators, so ensure that after editing a file, the line terminator is correct for the device that is running Cumulus:
**The single character representing line feed (in most encodings, LF is binary equivalent of a decimal 10) is used for both UNIX and Linux devices (including Raspberry Pi Operating System)
**The single character representing carriage return (in most encodings, CR is binary equivalent of a decimal 13) is used for Apple operating systems (like Mac)
**The two character sequence first CR then LF is used to terminate lines in all Windows operating systems (part of Microsoft's determination to be different)
**Problems with terminating characters are normally intercepted by operating system, before the contents of a line reaches any software like Cumulus, but if partial editing or merging has produced a file with mixed line terminators, there is a high possibility this will stop any software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
**Finally, if you are going to use a script (such as JavaScript or PHP) to attempt to read a Cumulus file, that script might only recognise a different line terminator to that your device operating system recognises (most likely with processing on a windows device, the script will treat one of the terminating characters (CR) as part of the adjacent field's text, and only treat the LF as a line terminator).

Revision as of 10:33, 10 April 2022

Crystal Clear info.png This document is 'Work In Progress' so content may not be complete.

Request for help from Wiki Readers

  • Do you understand how MX works?
  • Do you use hardware, or MX functionality, that is not yet documented? Can you begin that documenting?
  • Can you contribute simple text for novice users, examples of what you have done, correction of typing or factual errors, or supply missing details?
  • Will you make this page more useful by bringing content up-to-date as new releases change some information written for older releases?
  • Does any page need a section for novices, so they don't need to read more technical information further down that page?
  • Is there some information on this page, that should be on a separate page? Can you create the new page and move the less relevant information off this page, don't forget this page needs a link to the new page so people who expect to find it here know where it has moved to?

If you plan on contributing to the Wiki, then you will need an account.

  • Please use the Request Account form to apply for an account. Note that the Wiki is currently undergoing restructuring and is largely locked for editing, but please apply for an account if you wish to contribute in the future.
  • You will find help on how to contribute to this wiki at How to Edit.
  • If you need to consult others, please use the Cumulus Wiki suggestions forum.

Please be aware that information on this page may be incorrect.

Cumulus Version MX SpecificCumulus Version 1 SpecificThis page applies to both the legacy Cumulus 1 and to MX. It is intended that this page will cover all the different ways in which you add back any missing data. This means it should cover:

  • Data that you want to add from another system for a period when you were not running Cumulus
  • How Cumulus captures archive data if it is available for a period when Cumulus was stopped
  • Derivatives that have been added since particular log files lines were stored
  • Solving problems when a line was not successfully written to the daily summary log file

How Cumulus Works

The way that Cumulus works is that:

  1. It reads what we can call source values (defined below) from your weather station (based on measurements from sensors being transmitted in some way by the station)
  2. It calculates what we can call derived values which may be either the source re-expressed in different units, or calculated by combining two or three source values to get a new derivative
  3. It tracks various extremes, and cumulative totals, by comparing the derived values against existing extremes and sums stored in various period tracking files
    • You can find out about how a rogue source value can affect extreme records derived from it, and how to correct such issues on the Correcting_Extremes page.
  4. It periodically stores the spot (current) derived values in a collection of Monthly_log_files, (and so when you move to a different device, or upgrade to a new release, then providing you copy these files to the new location you will not lose any data)
  5. At the end of each day, Cumulus logs the daily extremes or daily sums, from monitoring changes in each derived value into daily summary log

Reading archive data

If you are using a weather station type that has a internal memory storing weather data that Cumulus can read, then when Cumulus is restarted it can read historic data from that logging memory.

This means that if you discover that Cumulus has missed some data, soon after it misses that data, you can rewind, by stopping Cumulus, optionally replacing files in data folder with earlier copies of those files from backup folder or backup/daily folder, and restarting Cumulus. Typical reasons for missing some data would include power blips and problems with the interface between Cumulus and the weather station.

  • For Cumulus 1, to stop Cumulus, you select Exit from the main menu.
  • For MX, how you stop MX depends on your device, and whether running as a service, please see MX on Linux or MX on Windows OS pages as appropriate for advice.

Put simply, Cumulus stores the latest time it successfully read data from the weather station in today.ini. When Cumulus is restarted, if it is possible to read the historic data from the weather station, then any entries between the time stored and the current time will be read.

With Cumulus 1, there is some dependence on weather station type, but usually two passes are made through the external logging memory, the first pass investigates what records are available in the weather station, by reading backwards in time, and the second pass works forward in past time reading and processing those records.

With MX, again there will be some dependence on weather station type, and the process has not been documented by the developer; it appears just a single pass is made.

Importing data from other systems for periods when Cumulus was not running

This section needs more work on it

Essentially, what we need to do, is to compare the format of the data we have available against the format of the relevant Cumulus file.

CSV outputs from software like EasyWeather

You might want to read Transferring_past_observations_from_EasyWeather.dat_to_Cumulus, although that is now an obsolete article, Easy Weather has changed, and Cumulus MX is different from the original Cumulus described there.

The CSV output from such files has fields in the wrong order, may not match our Cumulus units, and does not have a match for each of the fields we need in our Cumulus - see importing into standard log file fields. The solution is to use a CSV file editor, or a spreadsheet, to move the fields into a better order, to apply a formula where necessary to change units, and create any missing fields.

Cumulus does not allow for "null", field not available, except in that it can allow shorter lines with just the fields applying to an earlier release. For MX, CreateMissing.exe can calculate some of the derivatives missing from earlier releases for the Standard_log_files, and that utility can generate the necessary dayfile.txt lines.

On the Support Forum - Dane - offered a translation service - see here, but the last post is May 2018, so I am unsure whether it is still available as I type this in July 2021.

Weather Display

Please see Software page - Weather_Display_Converter. When you run this routine, it brings up a screen where you select the units that you want the Cumulus Standard_log_files it produces to use, as the units that weather display uses for the export are fixed.

The routine was written for Cumulus 1, so will not produce all the fields that MX uses, but CreateMissing.exe can calculate some of the derivatives missing from earlier releases, as well as producing dayfile.txt.

Weather Link log file

There is a routine, see Software page - WeatherLink_Converter for this. When you run this routine, it brings up a screen where you select the units. Please note, this utility assumes the units you select on that screen apply both to the file (OR MULTIPLE FILES) being converted and to the (multiple) Cumulus Standard_log_files it produces.

It was written for Cumulus 1, so will not produce all the fields that MX uses, but CreateMissing.exe can calculate some of the derivatives missing from earlier releases, as well as producing dayfile.txt.

Weather Link Live

PLEASE CAN SOMEBODY FULLY DOCUMENT THIS.

My understanding is that while MX can read current data from the WLL, you need a "pro subscription" to export past data. I don't know the format of that past data export, but can only guess it can be easily imported into Cumulus.

OTHERS????

PLEASE CAN SOMEBODY CONTRIBUTE WAYS OF IMPORTING FROM OTHER SOFTWARE HERE.

Meanwhile, people should ask for help in the support forum.

Some definitions

To make sense of explanations on this page, you need to understand the terminology used here.

Source value

A weather station sends values based on its sensors to Cumulus. If Cumulus reports that value without changing it (an offset and/or multiplier might be applied to convert it to the unit wanted by the Cumulus User), the value is described as a source value because Cumulus is reporting something that has its source elsewhere.

There is not a single list of what weather values are called "source values", because this varies depending on the weather station, and in some cases, a Cumulus User can ask Cumulus to recalculate a value instead of using what is sent by their weather station.

However, Cumulus does include code that expects a weather station to provide a defined minimum set of source values:

  1. Current air temperature
  2. Current Relative Humidity
  3. At least one wind speed
  4. Current air pressure (absolute or sea-level)

Cumulus will stop processing any information from a weather station unless the above 4 source values are being supplied and reveal they are being updated (failure is set is after a total of 6 unsuccessful consecutive attempts to read each of these).

This requirement is a default, but it can be changed:

Cumulus also expects that your weather station can provide:

  • A rainfall counter (this could be annual rainfall, or count of rocker bucket gauge tips)

Although the lack of that rainfall counter source value will affect functionality, Cumulus will continue to process other source values that are available.

Some weather stations may also provide one, or more, of these optional source values (not a complete definitive list):

  • Dew-point Temperature
  • Wind Chill Temperature
  • Evaporation
  • Sunshine hours
  • Solar radiation
  • UV index
  • Air pollution measurement

Derived value

A dictionary will define derived as "obtained from a source", and that is the meaning adopted on this page. Steve Loft (in the Cumulus Support Forum) used the terminology "derived" for two purposes.

  1. One type of derived value takes a source value, applies any multiplier (may be both first order and second order multipliers) and/or constant that has been defined in calibration settings, and converts the output to the units selected by the Cumulus user.
  2. The other type of derived value takes more than one source value, applies a standard calculation, and ouputs a new derivative
    • Because newer releases calculate more derivatives than older releases, extra fields have been added to the standard log file

"Calculate Missing"

This also has two meanings in a Cumulus context:

  1. If a particular standard log file line has fewer fields than the latest line;
    • Calculate Missing is the process of looking at the derived values of first type above, and calculating any derivative (second type of derived value) that is missing in that particular line
  2. If a particular daily summary log file, either does not have a line for a particular meteorological date, or does not have all fields defined in a line for a particular meteorological date;
    • Please see Amending dayfile page for full details.
    • Calculate Missing is the process of scanning all the lines in the standard log file that relate to the meteorological date and recalulating approximate extremes, or sums, for the missing fields.

If you are using Cumulus MX, there is a download linked from here that does both of these. There are also editors within the admin interface for manually editing the files on a line by line basis. You can also use the PHP Hypertext Pre-processor (PHP) script specified for Cumulus 1 below, although be aware it was written for a very old PHP version.

If you are using the legacy Cumulus 1 software:

  1. For the standard log file meaning above, provided you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then this post in support forum includes a script that produces a HTML form where you specify the log file name you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated. Please note that script was written to run on an old version of PHP that was current at the time the script was written, it will need some editing to work on latest PHP.
  2. For the daily summary log meaning above, go to the Edit menu, and select Dayfile.txt. This brings up an editor with a button labelled "Create Missing", that will not affect any existing line, but can insert missing lines, see Amending_dayfile#Create_Missing.

Accurate or Not?

This Wiki page describes some techniques for calculating and inserting values that are missing from standard log files and from daily summary log file.

Since the derived values this page is discussing are spot values, they have to be calculated from source values measured at the same time. This means that if one of your .ini files is missing some fields, these missing fields cannot be calculated from other fields. This applies to any missing extreme records for today, this month, this year, monthly-all-time, or all-time.

However, the techniques for correcting rogue values described on the Correcting_Extremes page, can be used for inserting missing values in the daily and longer period extreme records.

For the standard log files, all the fields in any one line relate to the same time, therefore for derived values calculated from other fields in the same line, you should have the same value whether it was calculated when that line was originally stored, or calculated afterwards. I say should because the calculation formula is not always the same for all releases, in particular there are differences between how Cumulus 1 and how MX calculate some derivatives.

For entries in today.ini, month.ini, year.ini, alltime.ini, and monthlyalltime.ini files, you don't have access to the source values used for the original calculation afterwards.

  • The original values are calculated as Cumulus is running
    • Depending on your weather station, Cumulus is able to read values every minute, and consequently update today.ini (and the other files listed) each minute if an extreme happens
    • Obviously, dayfile.txt is updated from today.ini, so it is just as accurate
  • Any "Calculate Missing" operation, done subsequently, does not have access to old data, it can only look in the spot values that have been logged.
    • If Cumulus is set up to only log the readings every half an hour, create missing is only able to see 1/30th of the data,
    • Due to this mismatch, the derived values (averages, highs, lows) this approach can store are much less accurate (hence getting missing lines from a backup is better)

Derived spot values

Cumulus software code as it reads source spot values, will detect if that source value is required for the calculation of an instant derived spot value.

Here are all the derived spot values that Cumulus can calculate (depending on Cumulus configuration settings, and what your weather station can output):

  • Dew point, a weather station might output dew point temperatures, but Cumulus can calculate it from source values for outdoor temperature and outdoor humidity
  • Wet Bulb, is only calculated by Cumulus 1, not MX
  • Wind Chill, again this might be output by your weather station, but Cumulus can calculate it from outdoor temperature and average wind speed.
  • Canadian Humidity Index (Humidex), USA Heat Index, and Apparent Temperature are not output by your weather station, but both the original Cumulus 1 and the newer Cumulus MX will derive these spot values for you (except if you are running a very old release)
    • (The implementation of these by Cumulus software is briefly mentioned here).
    • The calculation formulae used for these may not be consistent for all releases, so again there is a possibility a data log might have continuity breaks.
  • Feels Like Temperature is calculated by the Cumulus MX flavour only, the actual calculation formula has varied in different releases, but use a variation on
  • Heating Degree Days and Cooling Degree Days; these are further examples of derived values that most versions of Cumulus will calculate for you (from all processed outdoor temperatures in a day)
    • A bug in some versions of the original Cumulus software could result in these derived values being swapped and therefore tracked wrongly when reporting extremes.

The links above will take you to where the derived values are explained in the Category:Terminology pages of this Wiki, however at the time of writing this page, many of those links have very little information, so you may wish to search online to find more information in for example Wikipedia.


There are some configuration settings where you can decide whether to use a weather station supplied dew point temperature and whether to use a weather station supplied wind chill temperature.

Field Count Variations

When the standard data logging file was introduced it only had 16 (or fewer?) fields. As time has gone by, extra fields have been added to the file. At time of writing, 29 fields have been in the file since release 3.6.12 (build 3088), and currently the "To Do" database does not include any suggestions that would add more fields.

When the daily summary log file was introduced it had 15 fields. As time has gone by, extra fields have been added to the file. At release 3.6.12 there were 54 fields, but at earlier and later releases there are fewer fields. At the last update of this page (release 3.7.0) there were 52 fields. The number of fields in a line of the file might be changed in a future release.

When you use Cumulus to edit any of these files, it expects the file to have the number of fields defined in the release you are using. If an existing line in the file has fewer fields, Cumulus can still read it, but Cumulus will add trailing field separators if the file line is edited.

Consequently, those people who have used Cumulus for a while may have files that include some lines with fewer fields stored than their latest lines.

Why do "Calculate Missing"?

Most functionality in Cumulus is concerned with current data or extremes/sums that are derived for a hour, a day, or longer, periods. For these contexts, you might encounter an odd rogue value that needs to be corrected as described on the Correcting_Extremes page. You are unlikely to worry about missing past values.

However, if you want to be sure that your all-time extremes, or monthly-all-time extremes, are correct, then this table shows how the start date for these extremes varies. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".

If you are using the Historic Charts feature introduced from release 3.9.2 - b3097 (7 December 2020), you may notice that these new charts have gaps in available data, and the dates with/without data vary depending on what is being plotted. You might want to achieve better consistency by adding missing fields to earlier lines in the log files, if so you want to do a "create missing".


How to do "Calculate Missing"

As mentioned earlier, there are a number of options, here are the detailed instructions for each option.

CreateMissing.exe

This utility is only for those who have already installed Cumulus MX:

  • It uses some files that are in the MX installation package
  • It uses some files that are created when MX is run


Please note the developer does not fully describe his utility at his github page so the author of this Wiki update cannot guarantee the documentation here is correct

This utility was written by Mark Crossley specifically to insert any missing data:


Obtaining the Create Missing Utility

There is a download link on Software#Create_Missing page, unzip to reveal the components in the package, and install all of those in same folder as CumulusMX.exe.

If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the CreateMissing.exe file may need to be given execute access (see MX_on_Linux#chmod)

Please note that "Create Missing Version 1.1.0" can only be used with "Cumulus MX release 3.12.0" and above. It makes use of components not included in earlier MX releases.

Earlier "Create Missing" versions can be found at https://github.com/cumulusmx/CreateMissing, and these will work with earlier MX releases, but be aware that Create Missing version 1.0.2 had been tested and was found to have a bug.

Running the Create Missing Utility

The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days).

To run CumulusMX.exe needs .NET or MONO to be installed, and because MX is already running, CreateMissing.exe can be run, as it too requires .NET or MONO.

  1. Open up the MX interface in a browser, and navigate to Settings menu -> Station Settings -> General Settings -> Advanced Options -> Records Began Date
    • The date that is shown there is the date where "Create Missing" will start by default, so if you have MMMYYlog.txt log files with an earlier date, edit the date here, keeping to same format
    • Click Save Settings button if you have made a change to this date
  2. Close your browser, and (if using an interactive screen for your computer) open up your file manager, or (if using a terminal session for access to your computer) navigate to your CumulusMX folder
  3. Navigate to your data sub-folder
  4. If there is a file there called dayfile.txt.sav, rename that file to dayfile.txt.sav.bak (or any other name that does not already exist)
  5. Now change directory back up to parent folder cd .., i.e the folder containing "CreateMissing.exe"
    • On Linux operating systems, you need execute rights on that file (prefix with sudo if you don't have rights) type mono CreateMissing.exe.
    • On Microsoft Windows operating systems, type CreateMissing in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.

Remember, as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.

How the Create Missing Utility works

The utility program will output to any terminal session open and to a file saved in MXdiags sub-folder.

This utility program looks in Cumulus.ini for:

  1. The Cumulus start date in "StartDate=" parameter, which defaults to the date you first ran Cumulus (as mentioned above it can be edited to another date, to include imported earlier data or to exclude data that relates to a former location).
    • That will be the earliest date the utility program processes.
    • However, if a dayfile.txt file exists and that has an earlier date, then "Create Missing" will only continue if you accept that earlier date.
  2. The meteorological day start time in "RolloverHour=" and "Use10amInSummer=" parameters.
    • This identifies which standard log lines belong to each day by checking against date and time of that line.
  3. The thresholds for Heating Degree Days, Cooling Degree Days, and Chill Hours
  4. The starting month for Chill Hours Season

This utility program looks in the data sub-folder:

  1. If there is a file there called dayfile.txt.sav, the utility aborts
  2. If there is a file there called dayfile.txt, the utility renames that to dayfile.txt.sav
  3. The utility creates an empty file, naming it dayfile.txt
  4. The utility continues by opening the MMMYYlog.txt file for the earliest date (see above) it is going to process
    • See table below for how contents of this file are read/updated and used to create lines in the new dayfile.txt file
  5. Each subsequent MMMYYlog.txt file is processed in turn

How the utility reports progress

Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month):

2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt


How the utility creates a dayfile.txt line

dayfile.txt field Standard log file fields Description
Daily derivative Preferred field First source Second source Third source (how calculated)
date Day-Month-Year Hour-Minute From processing lines linked with that Meteorological day.
Highest wind gust speed Cumulus Gust wind speed Stores highest value of that log file field in that Meteorological day.
Bearing of highest wind gust Average wind bearing (in degrees) Stores the bearing recorded at same time as maximum value in previous field
Time of highest wind gust Hour-Minute Stores the time in log file line used in two previous fields
Minimum temperature Current temperature Stores the lowest value of that log file field in that Meteorological day.
Time of minimum temperature Hour-Minute Stores the time in log file line used in the previous field
Maximum temperature Current temperature Stores highest value of that log file field in that Meteorological day.
Time of maximum temperature Hour-Minute Stores the time in log file line used in the previous field
Minimum sea level pressure Current sea level pressure Stores the lowest value of that log file field in that Meteorological day.
Time of minimum pressure Hour-Minute Stores the time in log file line used in the previous field
Maximum sea level pressure Current sea level pressure Stores highest value of that log file field in that Meteorological day.
Time of maximum pressure Hour-Minute Stores the time in log file line used in the previous field
Maximum rainfall rate Current rainfall rate Stores highest value of that log file field in that Meteorological day.
Time of maximum rainfall rate Hour-Minute Stores the time in log file line used in the previous field
Total rainfall for the day Total rainfall today so far Stores the entry in the last log file field in that Meteorological day.
Average temperature for the day Hour-Minute Current temperature Loop through every log file pair of fields in that Meteorological day:
  1. Work out interval time in minutes obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
  2. Work out product of above interval time times "Current temperature" field
  3. Sum the interval times in step 1 for whole day
  4. Sum the products in step 2 for whole day
  5. When completed loop, store the sum in step 3 divided by the sum in step 4
Daily wind run Hour-Minute Cumulus moving 'Average' of wind speed measurements over a particular period Loop through every log file pair of fields in that Meteorological day:
  1. Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
  2. Work out product of above interval time times "Current average wind speed" field
  3. Sum the products in step 2 for whole day
  4. When completed loop, store the sum in step 3
Highest Average Wind Speed Cumulus moving 'Average' of wind speed measurements over a particular period Stores highest value of that log file field in that Meteorological day.
Time of Highest Avg. Wind speed Hour-Minute Stores the time in log file line used in the previous field
Lowest humidity Current relative humidity Stores the lowest value of that log file field in that Meteorological day.
Time of lowest humidity Hour-Minute Stores the time in log file line used in the previous field
Highest humidity Current relative humidity Stores highest value of that log file field in that Meteorological day.
Time of highest humidity Hour-Minute Stores the time in log file line used in the previous field
Total evapotranspiration Evapotranspiration Stores highest value of that log file field in that Meteorological day.
Total hours of sunshine Hours of sunshine so far today Stores highest value of that log file field in that calendar day (i.e. midnight to midnight)
High USA Heat index Heat Index Current relative humidity Current temperature The heat index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
Time of high heat index Hour-Minute Stores the time in log file line used in the previous field
High Apparent temperature Apparent temperature Current relative humidity Current temperature Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
Time of high apparent temperature Hour-Minute Stores the time in log file line used in the previous field
Low apparent temperature Apparent temperature Current relative humidity Current temperature Apparent temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
Time of low apparent temperature Hour-Minute Stores the time in log file line used in the previous field
High hourly rain Total rainfall today so far High hourly rain is a derived value. Loop through every log file field in that Meteorological day, build up a series of hourly values (total rainfall in this entry minus total rainfall an hour earlier), find maximum of all those hourly values, and store that.
Time of high hourly rain Hour-Minute Stores the time in log file line used in the previous field
Greatest wind chill (high wind speed, low temperature) Wind chill Cumulus moving 'Average' of wind speed measurements over a particular period Current temperature Wind Chill can be reported by weather station or it can be derived. If the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
Time of greatest wind chill Hour-Minute Stores the time in log file line used in the previous field
High dew point Current dew point Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores highest value of that log file field in that Meteorological day.
Time of high dew point Hour-Minute Stores the time in log file line used in the previous field
Low dew point Current dew point Dew Point can be reported by weather station or it can be derived. However, all Cumulus releases have this log file field. Stores lowest value of that log file field in that Meteorological day.
Time of low dew point Hour-Minute Stores the time in log file line used in the previous field
Today's dominant/average wind direction Cumulus moving 'Average' of wind speed measurements over a particular period Average wind bearing (in degrees) The dominant/average wind direction is a derived value.
  1. Loop through every log file pair of fields in that Meteorological day:
    • Calculate increment in X as product of wind speed times sine of bearing, and sum those increments
    • Calculate increment in Y as product of wind speed times cosine of bearing, and sum those increments
  2. Convert final X and Y coordinates back to a bearing in degrees
Heating degree days (HDD) Hour-Minute Current temperature Loop through every log file pair of fields in that Meteorological day:
  1. Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
  2. Work out increment in HDD by subtracting current temperature from HDD threshold
  3. Work out product multiplying result in step 1 by result in step 2, and sum those products
  4. At end of loop store the final sum
Cooling degree days (CDD) Hour-Minute Current temperature Loop through every log file pair of fields in that Meteorological day:
  1. Work out interval time in days obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
  2. Work out increment in HDD by subtracting CDD threshold from current temperature
  3. Work out product multiplying result in step 1 by result in step 2, and sum those products
  4. At end of loop store the final sum
High solar radiation current solar radiation Stores highest value of that log file field in that Meteorological day.
Time of high solar radiation Hour-Minute Stores the time in log file line used in the previous field
High UV Index UV Index Stores highest value of that log file field in that Meteorological day.
Time of high UV Index Hour-Minute Stores the time in log file line used in the previous field
High Feels Like temperature Feels Like temperature Current relative humidity Cumulus moving 'Average' of wind speed measurements over a particular period Current temperature Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
Time of high feels like temperature Hour-Minute Stores the time in log file line used in the previous field
Low Feels Like temperature Feels Like temperature Current relative humidity Cumulus moving 'Average' of wind speed measurements over a particular period Current temperature Feels Like temperature is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the lowest is stored.
Time of low feels like temperature Hour-Minute Stores the time in log file line used in the previous field
High Canadian Humidity Index or Humidex Humidex Current relative humidity Current temperature The Canadian Humidity index is a derived value, if the field in "preferred field" does not contain a valid number, then that field is populated for each line linked with that Meteorological day using the values in fields named in the the other columns of this table. When all the preferred field in day have a value, the highest is stored.
Time of high Humidex Hour-Minute Stores the time in log file line used in the previous field
Cumulative Seasonal Chill Hours Current temperature Hour-Minute "Chill Hours" is a derived value, loop through every log file field in that Meteorological day:
  1. Work out interval time in hours obtained by subtracting previous "Hour-Minute" field from current "Hour-Minute" field
  2. Work out if there is increment in Chill hours by seeing if "Current temperature" field is below Chill Hours threshold temperature
  3. If there is an increment, sum value from step 1
  4. At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day

Using a PHP script on your web server

If you have access to a web server that can run PHP Hypertext Pre-processor (PHP) scripts, then this post in support forum includes a script that produces a HTML form where you specify the log file name you wish to edit. The script will read that file, and output a replacement file with all possible spot derived fields populated.

That might sound a bit technical, so here are some step by step instructions:

  1. Download the processStandardLog.php script from here.
  2. File transfer (or copy for local web server) that script to your web server
  3. File transfer (or copy for local web server) all the standard log files from the data sub-folder in your Cumulus installation to a suitable holding folder (you may need to create it) on your web server
  4. Open in a browser PATH/processStandardLog.php by preceding the file name with the path as defined from the root on your web server
  5. This loads a web page where you have a field asking you to enter a path and file name for the data log you want to process
  6. Continue to follow the instructions on the web page
  7. When it has created a replacement file, you can enter details for another data log, and continue until all your data logs have been processed
  8. Take a backup of your existing Cumulus installation (you should be doing that on a regular basis anyway, so I will not give instructions here)
  9. Carefully delete any non-current data log in your data sub-folder that you have a replacement for, and file transfer (or copy back) the replacement data logs from your web server into the local data sub-folder, noting that the file extension will need to be renamed from .csv to .txt.

Using the data log editor provided in MX

Crystal Clear info.png This document was written for the (legacy) Cumulus 1 software. It has been updated to cover MX, but that was for a MX release that is no longer latest!

When this section was written, the number of lines shown was fixed at a maximum of 10; later releases have given the option to display different numbers of lines, and there may be other changes still to be documented here.

In the MX admin interface go to the Data Logs menu tab, and select the Data Logs page.

There is a box for selecting the data log you want to edit. Once you have loaded that, the first (up to) 10 lines are shown. Navigation links let you select 'First', 'Previous', 'Next', and 'Last' pages, also a small number of pages can be selected directly.

Once you select a line, an Edit button is enabled, click that and you can manually input the missing values for that line. Save that edit, and you can select another line. Once you have edited all the lines on that page, you can select another page, and repeat the process. Then you can select another log, and repeat the process.

It is a long-winded way to edit, and the MX editor does not even validate what you have entered. An alternative is to edit each log file externally, and you can read how to do that in the "Work around for standard log files" section below.

Some readers of the Cumulus support forum will know that a third-party replacement for the MX editor was worked on, but never got incorporated into MX. The idea was to replace the alt_editor software used by Mark Crossley, with a standard HTML form script. This allowed in-line editing, it allowed the derived values to be calculated and displayed (so you simply decided whether to accept the replacements as suggested for the various fields), and finally it applied some validation to each field to ensure any manual edit inserted a value that was within the allowed range. The main reason for its rejection from the public MX was the complex way in which different files included in the admin interface interact, and the consequent issue that changes made for this replacement had a knock on effect on other pages in the admin interface. The author could not afford the time to redesign the whole admin interface so the proposed replacement could be integrated.


Lack of editor in Cumulus 1

Cumulus 1 provides a viewer for the data logs, that does not permit editing of the file.

On the View menu, select 'Data Logs, then enter the file name you want to view and load it. You can scroll left to right through the fields, and you can scroll up and down through the lines. The viewer shows a header row so you know which field is which. You cannot do any editing.

If you find that this viewer cannot load a data log, it is probably because you ignored the read me that is part of the Cumulus 1 installation procedure, see FAQ: I can't find my data files. If the displayed headings do not match the data shown, you have not read the caution on the screen, which says the viewer is only for standard data logs, not extra sensor data logs , nor the daily summary log.

Cumulus 1 does not provide any functionality to edit the standard data logs, whether to correct a rogue value, or to add a missing derivative.

Work around for standard log files

An option is to edit the file outside Cumulus using a Comma Separated File editor, a plain text editor, or a spreadsheet program (like the free open source Libre Office Calc or the commercially charged for Microsoft Excel).

Note: Cumulus 1 applies an exclusive lock to current standard log file, and conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while the original Cumulus software is running. A full discussion of the problems with conflicts of access to the standard log file can be found in this support forum topic.

If you decide to edit the current log outside Cumulus, then remember that, if you leave Cumulus running it will continue to append new lines. Therefore, you either need to close Cumulus while you are doing the edit; or (if you are able to merge two files) close Cumulus while you replace its file with a merge of what you have edited and the extra lines added since you took away a copy to edit.

It is best practice, to take a back-up copy of all your Cumulus installation before starting any editing. It is also best practice to take a further copy of any file you want to amend, and to do your edit on that copy, so you do not edit any Cumulus file directly. The original full backup will preserve the existing file, so you can regress to it, should Cumulus find an error in your edit.

Also note these log files do not include a header line, and should not be edited to include it. All flavours of Cumulus provide, in the data sub-folder, a file called monthlyfileheader.txt which contains the headers appropriate to the release you are running, and you can add that temporarily to your spreadsheet if it helps you with editing, but don't forget to delete it before saving the file ready to make it available to Cumulus.

Here are some other rules to follow when editing the standard log files:

  • You can't edit any log file with a word processor, as they add control characters, and other information, that Cumulus cannot understand.
  • Editing is straight forward if you use a specialised comma separated value file editor, such editors will split the content by field so it is easy to ensure you only amend field content and do not accidentally change a field divider, plus these editors will not add additional content to any line as they can cope with the number of fields varying in different lines and don't change encoding.
  • If you want to use a text editor, it is best if you choose one designed for computer programmers or developers. Such an editor will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM).
  • If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ...").
  • If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer).
  • Ensure no blank lines are introduced by your editing.
  • Ensure that all lines continue to have date and time information at the start of the line, and that the format of that identifier is not changed (same sequence, same character(s) separating each element of date, and a colon separating hours and minutes, and that the time does not have a seconds element added.


General External Editing Rules

  • Take a copy of the file that can be reverted to if there is a subsequent problem, and you have messed up the file that Cumulus (1 or MX) is now trying to use.
  • Take another copy and use that for your editing, don't edit the actual file being used by the software.
    • This prevents any conflicts between access by the software and access by your script or tool being used to modify the file.
    • It also means that you can go back to the last working copy, you can't upset your "revert" copy.
  • The file must never be edited with a word processor, as they store many control and identification characters that prevent Cumulus correctly reading the values.
  • Generally, it is easiest if you use either a specialised "Comma Separated Value" file editor or a text editor.
    • These tools have the advantage that they can cope with different lines having a different number of fields depending on which version number of Cumulus created each line.
  • You can use a spreadsheet application, but if you do, there may be a number of settings to change from their defaults to ensure the file remains in a readable format for Cumulus.
    • You need to ensure that your spreadsheet treats every column as plain text, don't let it recognise dates or times and convert them into another format, don't let it convert any numeric field into another format
    • If you do use a spreadsheet, extra field separators may be added at end of shorter lines as these make all lines end up with same number of fields.
  • Don't remove any figures from fields where figures currently exist, simply overtype one entry with another entry in same format.
  • If your file has previously been edited by the relevant editor in MX, a field that looks empty may actually contain one or two space characters.
  • If you are editing a field which was empty previously, remember that Cumulus does not accept the concept of nulls (entering -999 or "Null"), there is nothing that can be placed as a place-holder when the correct figure is not known, and empty fields are not permitted in one field if any subsequent field in same line is not empty.
    • Beware - if you do insert zero or an obviously wrong extreme value, Cumulus will display those in any editing screen where you wish to update the all-time, monthly-all-time, this month, or this year, extremes. This can make editing by picking values in logs harder.
    • Cumulus itself will use zero for any parameters (e.g. solar) not provided by your station, and for up to 6 times it will repeat the last valid value if the station fails to send a value it should provide (normally six successive readings will happen between entries in the standard data log, so repeated values are less likely to affect log files).
  • The character (or in a few locales, two characters) used for separating both the day of the month from the month, and for separating the month from the year, must be consistent throughout the whole file (and must not be a single space). Normally, the separator will be either "-" or "/". Whether Cumulus expects a hyphen or a slash is determined by the locale, you must keep to the same locale for the whole file, you cannot change the locale when you do an edit, nor when you update the device running Cumulus. Although, use of comma or point for separating parts of the date is in some locales, and therefore allowed by Cumulus, those locale settings are not recommended as these date separators can cause issues for subsequent edits.
  • USA date format with month before day of month, and finally year, is not permitted for log files.
  • All figures must be within the range of sensible figures for that field (no hour 24 or higher, no signed numbers when accepted values must be positive, don't put in 200 for a relative humidity)
  • Make sure that any editing does not create any blank lines in the file. Cumulus assumes an empty line means end of processing.
  • Don't add a header line to the file, Cumulus expects all lines to be data lines.
  • Be aware that different devices use different line terminators, so ensure that after editing a file, the line terminator is correct for the device that is running Cumulus:
    • The single character representing line feed (in most encodings, LF is binary equivalent of a decimal 10) is used for both UNIX and Linux devices (including Raspberry Pi Operating System)
    • The single character representing carriage return (in most encodings, CR is binary equivalent of a decimal 13) is used for Apple operating systems (like Mac)
    • The two character sequence first CR then LF is used to terminate lines in all Windows operating systems (part of Microsoft's determination to be different)
    • Problems with terminating characters are normally intercepted by operating system, before the contents of a line reaches any software like Cumulus, but if partial editing or merging has produced a file with mixed line terminators, there is a high possibility this will stop any software understanding the resulting file, so be careful if you edit the file on a different device to that running Cumulus.
    • Finally, if you are going to use a script (such as JavaScript or PHP) to attempt to read a Cumulus file, that script might only recognise a different line terminator to that your device operating system recognises (most likely with processing on a windows device, the script will treat one of the terminating characters (CR) as part of the adjacent field's text, and only treat the LF as a line terminator).