Cumulus Wiki

Calculate Missing Values: Difference between revisions

Page Discussion

Calculate Missing Values (view source)

Revision as of 09:01, 22 August 2022

8,662 bytes added ,  22 August 2022
→‎CreateMissing.exe: Rewritten for 3.20.0 release
m (→‎CreateMissing.exe: That sentence still read wrongly!)
(→‎CreateMissing.exe: Rewritten for 3.20.0 release)
==CreateMissing.exe==
 
Mark Crossley has written a utility that can add any of the following fields to your [[Standard_log_files#List of fields in the file|MMMYYlog.txt]] lines:
This utility is only for those who have already installed Cumulus MX:
* [[Wind chill|Wind Chill]]
* It uses some files that are in the MX installation package
* [[Apparent_temperature|Australian Apparent Temperature]]
* It uses some files that are created when MX is run
* [[Feels Like|Feels Like temperature]]
* [[Heat index|North American Heat Index]]
* [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
* [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
 
The primary purpose of the utility is however to create a new ''dayfile.txt'' populating each field as summarised in table below.
 
''Please note the developer does not fully describe his utility at [https://github.com/cumulusmx/CreateMissing/blob/master/README.md his github page] so the author of this Wiki update cannot guarantee the detailed documentation here is correct''
 
This utility was written by Mark Crossley specifically to insert any missing data:
* It renames any existing [[dayfile.txt]] to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]
** The utility will not run if a file called '''dayfile.txt.sav''' already exists in data folder, the utility is intended to be used just once
* If any of the following fields are not populated in [[Standard_log_files#List of fields in the file|MMMYYlog.txt]] lines, the utility can populate them as it works through that file to work out daily extremes (see calculations in table below).
** [[Wind chill|Wind Chill]]
** [[Apparent_temperature|Australian Apparent Temperature]]
** [[Feels Like|Feels Like temperature]]
** [[Heat index|North American Heat Index]]
** [[Temperature_(and_humidity)_measurement#Dry_and_Wet_Bulb|Wet Bulb temperature]]
** [[Temperature_(and_humidity)_measurement#How_Cumulus_software_handles_Temperature_and_Humidity|Dew Point]]
* The utility creates a new ''dayfile.txt'' populating each field as summarised in table below
 
===Requirements for using Create Missing utility===
 
This utility is only for those who have already installed Cumulus MX:
===Obtaining the Create Missing Utility===
* It uses some [[Software#By_Mark_Crossley|.dll files]] that are in the MX installation package referenced beside the [https://github.com/cumulusmx/CreateMissing/releases release download]
* Each "CreateMissing.exe" release relates to the fields in [[dayfile.txt]] at a particular "CumulusMX.exe" release (see [[Software#Create_Missing]] page)
* It uses some information in [[Cumulus.ini]] a file created when MX is run
 
Although the developer works on "CreateMissing.exe" independently of "CumulusMX.exe" code, so they can get out of step, normally the latest release of the former works with the latest release of the latter.
There is a download link on [[Software#Create_Missing]] page, unzip to reveal the components in the package, and install all of those in same folder as CumulusMX.exe.
 
In a Microsoft Windows environment, Create Missing uses the .NET software that is normally already available.
If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[MX_on_Linux#chmod]])
 
In a UNIX-derived operating system (e.g. a computer running Linux or Raspberry Pi operating system), there is a need to [[Preparing_your_Linux_computer_for_MX#Checking_if_mono-complete_is_in_Source_List|install MONO]] to run either "CreateMissing.exe" or "CumulusMX.exe".
Please note that "Create Missing Version 1.1.0" can only be used with "Cumulus MX release 3.12.0" and above. It makes use of components not included in earlier MX releases.
 
Earlier "Create Missing" versions can be found at https://github.com/cumulusmx/CreateMissing, and these will work with earlier MX releases, but be aware that Create Missing version 1.0.2 had been tested and was found to have a bug.
 
===Running the Create Missing Utility===
 
===Obtaining the Create Missing Utility===
The utility program can be run while MX is left running (except at rollover time when MX writes to dayfile.txt, this includes any time while MX is doing "catch-up" and therefore can do rollovers for past days).
 
There is a download link on [[Software#Create_Missing]] page for the latest release of the utility. The MX 3.20.0 release zip also includes a copy of the utility; it was not included in release packages for earlier MX releases, and I can't predict whether it will be included in any subsequent MX release.
 
All releases are available at [https://github.com/cumulusmx/CreateMissing/releases Github release download], but be aware that Create Missing version 1.0.2 definitely had a bug as mentioned by developer in the forum, and the forum reports issues found by some users over handling the first day of any month with some other releases. Each release works only with specific "CumulusMX.exe" releases as described (with links to details) in previous subsection.
 
# Run whatever package your computer uses to extract/unzip packages.
# The package should list 2 or 3 components found in the zip file
# Extract those components to a new folder in your download area
# Copy/install the "CreateMissing.exe" and "CreateMissing.exe.config" into same folder as CumulusMX.exe.
# If there is a "Updates.txt" included, you will need to rename that file before copying as it conflicts with a file of that same name issued with "CumulusMX.exe" and "ExportToMySQL.exe" release downloads.
 
If you are installing it into a UNIX environment (e.g. a computer running Linux or Raspberry Pi operating system), the '''CreateMissing.exe''' file may need to be given execute access (see [[Preparing_your_Linux_computer_for_MX#chmod]]).
 
===Preparing to run the Create Missing Utility===
To run CumulusMX.exe needs .NET or MONO to be installed, and because MX is already running, CreateMissing.exe can be run, as it too requires .NET or MONO.
 
# Open up the MX interface in a browser, and navigate to ''Settings menu -> Station Settings -> General Settings -> Advanced Options -> Records Began Date''
#* The date that is shown there is the date where "Create Missing" will start by default, so if you have MMMYYlog.txt log files with an earlier date, edit the date here, keeping to same format
#*# Ensure any new date you enter there uses exactly the same format as the date that was there
#* Click '''Save Settings''' button if you have made a change to this date
#*# Click '''Save Settings''' button if you have made a change to this date
# Close your browser, and (if using an interactive screen for your computer) open up your file manager, or (if using a terminal session for access to your computer) navigate to your CumulusMX folder
# Navigate to your [[Data_folder|data sub-folder]]
# If there is a file there called '''dayfile.txt.sav''', rename that file to '''dayfile.txt.sav.bak''' (or any other name that does not already exist)
# Now change directory back up to parent folder <code>cd ..</code>, i.e the folder containing "CreateMissing.exe"
#* On Linux operating systems, you need execute rights on that file (prefix with '''sudo''' if you don't have rights) type <code>mono CreateMissing.exe</code>.
#* On Microsoft Windows operating systems, type <code>CreateMissing</code> in a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon.
 
Optionally, you may wish to take a backup of the existing contents of the "data" directory onto a separate storage device, because the utility edits files in there, and if something goes wrong mid-edit (e.g. power cut, storage device failure) you could lose your valuable data.
Remember, as MX only reads the dayfile.txt as MX starts up, any changes this utility makes will not be picked up by MX until MX is stopped and restarted.
 
 
===Running the Create Missing Utility===
 
The utility can be run while "CumulusMX.exe" is running (see below) either interactively or as a service, or while MX is stopped.
 
'''On Microsoft Windows operating systems''':
# First change the path to your Cumulus MX root folder
# Start a command window, Powershell window, or Terminal window (whichever is available when you right click in the folder or on the "Start" icon)
# Now type <code>CreateMissing</code> (or type "CreateMissing.exe", both will work)
 
 
'''On Unix-derived operating systems (such as Linux)''':
# First change your command line path to your Cumulus MX root folder (cd CHOSEN PATH/CumulusMX)
# Now type <code>sudo mono CreateMissing.exe</code>
#* If the user you are using already has execute access it is possible to leave out 'sudo' (the present writer has not tested this)
#* If mono is already running, it has been suggested it might be possible to leave out 'mono' (the present writer has not tested this)
 
 
''It is important to understand'' that when you start MX interactively, or as a service, it reads the contents of '''dayfile.txt''' into an internal array (held in random access memory - RAM).
* The only other time that 'CumulusMX.exe'' accesses the '''dayfile.txt''' is at rollover when it copies what it generates internally for the day just ended into the file
* You should never run '''CreateMissing.exe''' near to your rollover time
* At any other time, if you run '''CreateMissing.exe''' while ''CumulusMX.exe'' is running, any updates to the file are ''not'' seen by MX until that software is restarted.
* Release 3.20.0 (beta build 3199 onwards) adds a new "Utils" menu with a new option to refresh the internally held values without restarting MX.
* The internally held values, not the contents of '''dayfile.txt''' itself, are displayed when you use any of the [[Correcting_Extremes|extreme record editors]]
 
 
===How the Create Missing Utility works===
 
The utility program will output to any terminal session open and also (with further detail) to a file saved in [[MXdiags_folder#How the utility reports progress|MXdiags sub-folderdirectory]].
 
This utility program looks in [[Cumulus.ini]] for:
# If there is a file there called '''dayfile.txt.sav''', the utility aborts
# If there is a file there called '''dayfile.txt''', the utility renames that to ''dayfile.txt.sav''
# The utility creates an empty file, naming it '''dayfile.txt'''
# The utility continues by opening the MMMYYlog.txt file for the earliest date (see above) it is going to process
#* See table below for how contents of this file are read/updated and used to create lines in the new '''dayfile.txt''' file
# Each subsequent MMMYYlog.txt file is processed in turn
 
===How the utility reports progress===
 
The sequence followed by the utility is:
Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month):
# The utility will not run if a file called '''dayfile.txt.sav''' already exists in [[data folder]]
<pre>
# The utility reads [[Cumulus.ini]] and displays the 'record start date' it finds
2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
# It asks if this date is correct
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
# If you answer with anything other than 'Y' or 'y', the utility aborts
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
# If you answer with 'Y' or 'y', the utility continues
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
# It creates an internal array to hold the equivalent of '''dayfile.txt''' in random access memory - RAM
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
# If there is an existing [[dayfile.txt]], its contents are used to populate that two-dimensional array
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
# It renames any existing 'dayfile.txt' to '''dayfile.txt.sav''' in your [[Data_folder|data sub-folder]]
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
# It then works through the MMMYYlog.txt files that have lines for dates after that 'record start date'
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
# For each of these [[Standard log files]] there is another sequence:
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
## The inner sequence involves opening the file (the log in MXDiags records when each file is opened, files may be opened more than once)
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
## The dates being processed are shown in the terminal window; as each output is followed by just a "Line Feed" character the lines will overwrite each other in a Microsoft Windows environment (which expects "Carriage Return" and "Line Feed" in sequence to terminate a line), will be on successive lines in a Linux environment (where the normal line terminator is "Line Feed"), and will be on one long line in a Mac operating system (where the normal line terminator is "Carriage Return")
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
## If a line read in the source file does not include a particular [[#Derived_value|derived value]] (see list [[#CreateMissing.exe|above]]), then the utility will calculate that required value from the [[#Source_value|spot source field values]] in the same line, and update that line
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
## The exact calculation done for each item in dayfile.txt is listed in the table [[#How the utility creates a dayfile.txt line]] below, but here are the basic principles
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
## For solar data, it examines lines in the file with times between two successive midnights
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
##* If the array already holds a "sun hours" figure for the date quoted in those lines, the utility skips to next midnight to midnight period
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
##* If the array is missing a "sun hours" figure for the date quoted in those lines, the utility stores the sun hours recorded in last entry of that period in the part of the internal array with matching date
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
## From release 1.3.0, CreateMissing.exe will also calculate the maximum rainfall in a 24-hour period; to do this it looks at rain counter for the line in the source file it is currently processing, and attempts to find rain counter in another line nearest to 24 hours later (if the time interval between the lines is not exactly 24 hours then the difference figure reported will be for whatever period is available) looking up the rain counter there; it will track the maximum difference ending on a date that date determines where in the array the figure is stored; if the difference in rain counter yields a negative figure it is ignored
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
## For other daily data, the lines examined will start at rollover time and continue until just before the next rollover time (with allowance for any DST change)
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
##* If the daily data is an extreme, then the highest or lowest (source or derived) value seen in that range of lines is added to the array if an entry does not already exist
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
##* If the daily data is cumulative (over a day or over a season), then if the internal array does not already have a value, that from the last value in the range of lines is stored
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
##* If the daily data is not directly related to fields held in the source file (e.g. cumulative chill hours), then it will be calculated from what is available (for chill hours, if the recorded temperature in line being examined is below threshold, then the time interval passed since the previous entry is added to an internally held count; note this MX calculation is slightly different to Steve Loft approach that worked on average of temperature in a particular line and the temperature in previous line being below threshold)
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
## When all lines in a particular file have been examined, that file is closed (the log in MXDiags records when each file is closed, files may be closed more than once)
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
## The next file (chronologically) is opened (the log in MXDiags records when each file is opened, files may be opened more than once) and the inner sequence continues
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
# When all the source files have been processed, the utility continues
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
# The utility creates a new file, naming it '''dayfile.txt'''
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
# The utility copies what is stored in its internal array to the new file
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
# To prevent the terminal screen closing, the utility ends with a "Press any key to continue" prompt
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt
</pre>
 
 
# If there is an increment, sum value from step 1
# At end of loop, store final value of sum after (except on first day of month specified as Start of Chill hours season) adding it to value in previous day
|-
| Maximum rainfall in a 24 hour period ending on particular day
| colspan="4" | Rainfall Counter
| Two log file lines are examined, approximately 24 hours apart, the rainfall counter value in earlier of the two lines is subtracted from that in later of the two lines. If the difference is negative it is ignored. If the difference is zero, or positive, then it is inserted into the Rain 24 hours for the relevant Meteorological day date for the time in the later line, unless it is lower than any number already there
|-
| Time of maximum rainfall in a 24 hour period
| colspan="4" | Hour-Minute
| Stores the time in later of the two log file lines used in the previous field
|}
 
===After running Create Missing===
 
If MX was running while Create Missing was run, then MX will remain unaware that "dayfile.txt" has been updated, and that can cause a problem at rollover when MX appends a new line to the file.
 
If you are running a MX release up to 3.19.3, then you should first stop and then restart MX, so it loads the contents of the new file into its internal array (see [[#Running the Create Missing Utility]]. From release 3.20.0 (beta build 3199 onwards) released as build 3202, there is a '''Utils''' menu in the interface, and you should select ''Reload Dayfile'' menu item and on the web page subsequently displayed click the ''Reload Dayfile'' button.
 
Create Missing changes the content of your [[Standard log files|MMMYYlog.txt files]] and your [[Amending dayfile|dayfile.txt]], so you may then wish to work through all the [[Correcting_Extremes|extreme record editors]] and decide if you want to update any entries.
 
 
===How the utility reports progress===
 
Here is a short section of typical output (from version 1.0.2 that had a bug and never processed 1st day of month) in a log that was stored in [[MXdiags folder|MXdiags]]:
<pre>
2021-06-08 19:35:44.108 Loading log file - data/Jul20log.txt
2021-06-08 19:35:44.191 01/07/2020 : No monthly data was found, not updating this record
2021-06-08 19:35:44.688 Date: 02/07/2020 : Adding missing data
2021-06-08 19:35:44.705 Date: 03/07/2020 : Adding missing data
2021-06-08 19:35:44.719 Date: 04/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 05/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 06/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 07/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 08/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 09/07/2020 : Entry is OK
2021-06-08 19:35:44.720 Date: 10/07/2020 : Entry is OK
2021-06-08 19:35:44.721 Date: 11/07/2020 : Adding missing data
2021-06-08 19:35:44.777 Date: 12/07/2020 : Adding missing data
2021-06-08 19:35:44.791 Date: 13/07/2020 : Adding missing data
2021-06-08 19:35:44.805 Date: 14/07/2020 : Adding missing data
2021-06-08 19:35:44.819 Date: 15/07/2020 : Adding missing data
2021-06-08 19:35:44.834 Date: 16/07/2020 : Adding missing data
2021-06-08 19:35:44.848 Date: 17/07/2020 : Adding missing data
2021-06-08 19:35:44.863 Date: 18/07/2020 : Adding missing data
2021-06-08 19:35:44.877 Date: 19/07/2020 : Adding missing data
2021-06-08 19:35:44.892 Date: 20/07/2020 : Adding missing data
2021-06-08 19:35:44.905 Date: 21/07/2020 : Adding missing data
2021-06-08 19:35:44.919 Date: 22/07/2020 : Adding missing data
2021-06-08 19:35:44.933 Date: 23/07/2020 : Adding missing data
2021-06-08 19:35:44.948 Date: 24/07/2020 : Adding missing data
2021-06-08 19:35:44.962 Date: 25/07/2020 : Adding missing data
2021-06-08 19:35:44.977 Date: 26/07/2020 : Adding missing data
2021-06-08 19:35:44.992 Date: 27/07/2020 : Adding missing data
2021-06-08 19:35:45.006 Date: 28/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 29/07/2020 : Entry is OK
2021-06-08 19:35:45.006 Date: 30/07/2020 : Entry is OK
2021-06-08 19:35:45.141 Date: 31/07/2020 : Adding missing data
2021-06-08 19:35:45.156 Finished processing log file - data/Jul20log.txt
2021-06-08 19:35:45.156 Loading log file - data/Aug20log.txt
</pre>
 
==Using a PHP script on your web server==
Sfws
5,838

edits

Retrieved from "https://www.cumuluswiki.org/a/Special:MobileDiff/10646"